A method for identifying driver risk perception state based on heterogeneous modal knowledge distillation

Abstract

The driver risk perception state identification method based on heterogeneous modal knowledge distillation belongs to the field of data analysis and processing technology. It addresses the lack of a technical solution that simultaneously meets the following requirements: First, it can utilize driver physiological responses to inversely constrain the risk label generation process; second, it can effectively transfer "external scenario risk knowledge" to "non-contact facial / physiological modalities"; third, it can achieve rapid individualized adaptation using a small number of high-confidence samples when facing new drivers who have not participated in training. The method includes the following steps: multi-source data acquisition and time synchronization, physiological event detection and retrospective annotation, multi-evidence fusion for risk-induced SCR event determination, risk perception sample construction, teacher network construction and training, student network construction and distillation training, and weakly supervised individualized fine-tuning for unseen drivers.

Description

Technical Field

[0001] This invention belongs to the field of data analysis and processing technology, specifically relating to a method for identifying driver risk perception state based on heterogeneous modal knowledge distillation. Background Technology

[0002] With the rapid development of intelligent connected vehicles, driver assistance systems, and human-machine co-driving technologies, how to perceive a driver's awareness of external traffic risks in real time and accurately has become a crucial fundamental issue for improving driving safety, enhancing human-machine collaboration strategies, and achieving personalized assisted control. Existing research and industrial solutions primarily focus on detecting driver states such as fatigue, distraction, emotional states, and road rage. Inputs typically come from facial video, eye movement data, heart rate data, lane-keeping behavior, steering wheel operations, and vehicle kinematic parameters.

[0003] However, a driver's "risk perception state" is not entirely the same as traditional fatigue or distraction states. Risk perception state emphasizes the coupling process between a driver's subjective perception of the degree of external danger, internal psychological reactions, and external behavioral / physiological changes after a specific traffic stimulus occurs. Objective traffic indicators such as vehicle distance, collision time, or headway alone can often only describe whether a potential hazard exists in a scenario, but are insufficient to characterize the differences in perception among different drivers of the same scenario. Furthermore, relying solely on facial features, rPPG, or other single physiological explicit information is easily affected by individual baseline differences, lighting conditions, posture changes, and scene noise, leading to coarse labeling and insufficient generalization.

[0004] To address the aforementioned issues, several similar solutions have emerged in the prior art. For example, Chinese invention patent application CN110781828A, "A Fatigue State Detection Method Based on Micro-expressions," discloses an individual-oriented approach to micro-expression fatigue detection; Chinese invention patent application CN117892178A, "A Scene Extraction Method Based on Intelligent Vehicle Traffic Situation Risk Recognition," discloses a method for extracting and classifying traffic risk scenes.

[0005] While existing technologies have provided valuable insights into multimodal driver state monitoring, non-contact physiological parameter extraction, individualized detection, and risk scenario extraction, a complete solution that can simultaneously meet the following requirements is still lacking: First, it should be able to use the driver's physiological response to inversely constrain the risk label generation process; second, it should be able to effectively transfer "external scenario risk knowledge" to "non-contact facial / physiological modalities"; and third, it should be able to achieve rapid individualized adaptation with a small number of high-confidence samples when facing new drivers who have not participated in training. Summary of the Invention

[0006] To address the current lack of a complete solution that simultaneously meets the aforementioned requirements, this invention provides a method for identifying driver risk perception states based on heterogeneous modal knowledge distillation, specifically: S1. Multi-source data acquisition and time synchronization: Collect multi-source heterogeneous data on "driver-vehicle-traffic" during the driving process and complete time alignment; S2. Physiological Event Detection and Retrospective Labeling: Identify skin conductance response (SCR) events during driving by using driver EDA data of skin conductance activity (EDA) and construct physiological trigger windows and traffic baseline windows around each SCR event. S3. Determination of risk-induced SCR events through multi-evidence fusion: Extract relevant state quantities of driving events from the physiological induced window and the traffic baseline window respectively, construct a multi-evidence fusion scoring function, and screen out effective risk-induced SCR events; S4. Risk perception sample construction: positive samples are constructed with the screened effective risk-induced SCR events as the center, and negative samples are constructed with the time periods far away from all positive samples. S5. Teacher Network Construction and Training: Construct a teacher network to learn discriminative knowledge that maps external scene stimuli to the driver's risk perception state. The teacher network sample consists of vehicle motion state data, traffic vehicle state data, and seat pressure distribution data; train the teacher network. S6. Student Network Construction and Distillation Training: A student network is constructed to identify the driver's risk perception state from non-contact modalities. The student network samples consist of driver facial movement and head posture feature data, heart rate detection data, and seat pressure distribution data. A multi-objective loss function including knowledge distillation loss is used to train the student network. S7. Individualized Fine-Tuning for Weakly Supervised Drivers Without Experience: For new drivers who have not participated in training, their continuous driving data is collected, and a sample sequence is constructed according to the same time window as the training phase. The samples that pass the quality screening are input into the frozen teacher network to obtain the teacher network's prediction value of the driver's risk perception state and the corresponding probability of risk perception state for each sample. The parameters of the student network trained at the group level are used as initialization to fine-tune the parameters of the student network. After fine-tuning, the driver's facial movements and head posture feature data, heart rate detection data, and seat pressure distribution data are input into the actual inference phase to output the driver's risk perception state identification result.

[0007] Furthermore, in step S1: Multi-source heterogeneous data includes at least: vehicle motion state data, traffic vehicle state data, driver skin conductance data (EDA), driver facial video data, facial movement and head posture feature data extracted from facial videos, heart rate detection data, and seat pressure distribution data.

[0008] Furthermore, in step S2: The EDA signal is preprocessed and decomposed to identify skin conductance response (SCR) events during driving. Extract the start time of each SCR event. and peak time EDA preprocessing includes denoising, baseline separation, normalization, and SCR peak detection; A physiological trigger window and a traffic baseline window are constructed around each SCR event. The physiological trigger window is denoted as . ,in The traffic baseline window is used to cover the latency time for external stimuli to elicit a physiological response. ,in Use the background as a reference for the window length; The physiological trigger window is used to cover the dynamic process of "external traffic stimulus - driver risk perception - physiological response initiation"; the traffic baseline window is used to provide the local background state at an earlier time in the same driving context, and is used to measure the intensity of change of state quantities within the physiological trigger window relative to the background conditions.

[0009] Furthermore, in step S3: The driving event-related state quantities include at least: the presence of the preceding vehicle, the time to collision (TTC), the headway (THW), the longitudinal braking intensity of the vehicle, the yaw rate of the vehicle, and the longitudinal acceleration of the vehicle. The construction of the multi-evidence fusion scoring function is specifically as follows: for the first... There are 1 SCR events, and their event scores are denoted as follows: ,in For the first Weight of each piece of evidence The result or normalized intensity value is the result of whether the corresponding judgment indicator is valid; the judgment indicator includes: the preceding vehicle has a gating condition, TTC is less than the first risk threshold, THW is less than the second risk threshold, the longitudinal braking intensity of the vehicle exceeds the baseline adaptive threshold, the yaw rate of the vehicle exceeds the lateral operation threshold, and the longitudinal acceleration of the vehicle exceeds the sudden change threshold. when Greater than or equal to the preset judgment threshold If the event is deemed to be a valid risk event induced by changes in traffic conditions, it will be classified as such; otherwise, it will be considered a non-risk-induced SCR or a low-relevance event.

[0010] Furthermore, step S4 specifically includes: For the One valid SCR event, around its start time. Constructing a positive sample window , Indicates the duration of forward backtracking. Indicates the duration of the backward continuation; When constructing negative samples, first extend all positive sample windows forward and backward by a safety interval along the time axis. Then, in the remaining continuous time period, select intervals with a length greater than or equal to the preset minimum length, and use a fixed window length. and sliding step size Slide the image to generate low-risk cognitive state samples.

[0011] Furthermore, in step S5: The teacher network consists of a three-channel temporal convolutional encoder and a classifier head; the three inputs correspond to the vehicle motion state sequence, the traffic vehicle state sequence, and the seat pressure distribution sequence, respectively. Each temporal convolutional encoder includes: an input batch normalization layer, multiple one-dimensional convolutional layers, activation layers, normalization layers, random deactivation layers, and adaptive global average pooling layers; The three encoded embedding vectors are concatenated to form the teacher fusion feature, which is then output by the fully connected classification head to show the teacher network's predicted value of the driver's risk perception state and the corresponding probability of the risk perception state. During online training, teachers use a binary cross-entropy loss function for supervised training, combined with category-weighted random sampling.

[0012] Furthermore, in step S6: The student network consists of a three-channel temporal convolutional encoder and a classifier head; the three inputs are facial motion and head pose feature sequences, heart rate detection data sequences, and seat pressure distribution sequences, respectively. Each temporal convolutional encoder includes: an input batch normalization layer, multiple one-dimensional convolutional layers, activation layers, normalization layers, random deactivation layers, and adaptive global average pooling layers; The three encoded embedding vectors are concatenated to form the student fusion feature, which is then output by the fully connected classification head to show the student network's predicted value of the driver's risk perception state and the corresponding probability of the risk perception state. The student network is trained using a multi-objective loss function, with the total loss function being: ,in, For binary cross-entropy loss based on real labels, For knowledge distillation loss, For rPPG reconstruction loss; , and These are the coefficients corresponding to each type of loss.

[0013] Furthermore, in step S7: During fine-tuning, the fine-tuning loss is the consistency loss between the student network output and the teacher network soft label, and a pseudo-label supervision loss is superimposed. The student network is locally updated using several high-confidence individual samples to obtain an individualized student network.

[0014] Another aspect of the present invention provides a driver risk perception state identification system based on heterogeneous modal knowledge distillation, the system being used to implement the method described above, the system comprising: The data acquisition and synchronization module is used to collect multi-source heterogeneous data on "drivers, vehicles, and traffic" and complete time alignment. The physiological event detection module is used to detect skin conductance response (SCR) events and extract the event start time, peak time, and amplitude. A dual-time-window construction module is used to construct physiological trigger windows and traffic baseline windows; The multi-evidence fusion judgment module is used to determine the risk inducement of SCR events based on the state variables within the window; The sample construction module is used to generate positive samples, negative samples, and aligned samples of teacher and student networks. The teacher network training module is used to train teacher networks based on scenario-side modalities. The student network training module is used to perform joint training based on real labels, distillation loss, and non-contact pulse wave rPPG reconstruction loss; The individualized fine-tuning module is used to fine-tune the student network without drivers based on high-confidence pseudo-labels of high and low quantiles; The online reasoning module is used to output the identification results of whether the driver's cognitive state is high-risk or low-risk.

[0015] The beneficial effects of the method described in this invention are as follows: By constructing risk labels using EDA / SCR as physiological anchors, the training samples are made closer to the driver's real subjective risk perception process. Compared with schemes that rely solely on traffic kinematic thresholds or manual coarse labels, the label effectiveness is higher.

[0016] Introducing a traffic baseline window to model the local background of the current operating conditions can adaptively evaluate the intensity of state variables such as braking, reducing the problem of insufficient generalization of fixed thresholds in different drivers and scenarios.

[0017] By employing teacher-student heterogeneous modal knowledge distillation, the ability to identify external risks in a scene is transferred to lightweight and easily deployable modalities such as face / rPPG / body posture, so that high recognition performance can be obtained without continuously relying on full scene-side sensors during online operation.

[0018] Adding rPPG reconstruction auxiliary constraints to the student network helps maintain the temporal structural expressiveness of the rPPG encoder, reduces feature degradation, and improves the stability and generalization ability of the model.

[0019] By using high-confidence pseudo-labels at high and low quantiles and fine-tuning with a small number of samples, we can achieve rapid individualized adaptation for new drivers, which can significantly reduce the performance degradation problem when directly transferring from a group model. Attached Figure Description

[0020] Figure 1 This is a flowchart of the method described in an embodiment of the present invention; Figure 2 This is a schematic diagram of the dual time window and sample construction based on SCR backtracking in an embodiment of the present invention; Figure 3 This is a schematic diagram of the teacher-student network configuration and individualized fine-tuning framework in an embodiment of the present invention. Detailed Implementation

[0021] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0022] Example 1 This embodiment provides a method for identifying driver risk perception state based on heterogeneous modal knowledge distillation. First, it introduces the shortcomings of existing technologies in this field: (1) There is insufficient coupling between the label construction method and the driver's actual risk perception. Existing solutions often directly give risk labels based on collision time, headway, or subjective manual labeling, which makes it difficult to reflect the individual differences in the driver's subjective risk perception process, and there is a deviation between the label and the actual perception state.

[0023] (2) When using only facial videos or driving behavior for state recognition, there is a lack of causal constraints from external traffic stimuli. As a result, changes caused by individual baseline differences, lighting changes, or occasional actions are easily misjudged as changes in risk perception, leading to insufficient robustness of the model.

[0024] (3) Multimodal fusion mostly stays at the level of direct feature splicing and has not yet formed a cross-modal knowledge transfer mechanism between "scenario risk knowledge - physiological explicit behavior", making it difficult to make full use of the risk discrimination ability learned by the teacher model.

[0025] (4) Existing driver status models are usually built based on the average pattern of the group and lack a rapid individualized adjustment mechanism for drivers who have not been seen. When deployed to new users, the performance degrades significantly.

[0026] To address the aforementioned shortcomings, the present invention aims to provide a method for identifying driver risk perception states based on physiological retrospective annotation and heterogeneous modal knowledge distillation. This method constructs a risk event screening mechanism anchored by EDA / SCR to generate training labels that more closely reflect the driver's subjective risk perception. By establishing teacher and student networks, it transfers scenario-based risk judgment knowledge to non-contact modalities such as facial videos, heart rate data, and body posture. Furthermore, it utilizes high- and low-quantile high-confidence pseudo-labels to perform weakly supervised individualized fine-tuning for unseen drivers, thereby improving identification accuracy, generalization ability, and practical deployment applicability.

[0027] like Figure 1 As shown in the figure, the overall process from multi-source data acquisition and time synchronization, SCR event detection and physiological retrospection, dual time window construction, multi-evidence fusion judgment, sample construction, to outputting the driver's risk perception status results.

[0028] The method is specifically as follows: I. Multi-source data acquisition and time synchronization Step S1: Collect multi-source heterogeneous data on "driver-vehicle-traffic" during the driving process and complete time alignment. The data includes at least: vehicle motion state data, traffic vehicle state data, driver's electrodermal activity (EDA) data, driver's facial video data, facial movement and head posture feature data extracted from the facial video, heart rate detection data, and seat pressure distribution data.

[0029] The heart rate data is obtained by remote photoplethysmography (rPPG), and will be referred to as rPPG data below.

[0030] The vehicle motion status data may include vehicle speed, longitudinal acceleration, lateral acceleration, yaw rate, steering wheel angle, brake pedal opening, accelerator pedal opening, etc.; traffic vehicle status data may include relative distance, relative speed, relative acceleration, presence of the vehicle in front, lane relationship of the target vehicle, etc.; facial movement and head posture features may include facial motion unit intensity, motion unit activation probability, three-dimensional head posture angle, and gaze direction features; rPPG data is a pulse wave time series extracted based on subtle color changes in the region of interest of the face; seat pressure data is the pressure distribution characteristics corresponding to changes in the driver's sitting posture and center of gravity.

[0031] Preferably, a unified timeline or synchronous timestamp mechanism is used to align the data of each modality, and missing values, infinite values ​​and abnormal timestamps are cleaned to ensure that the input from the teacher's end and the input from the student's end strictly correspond in the same time window.

[0032] II. Physiological Event Detection and Backtracking Annotation Based on EDA / SCR Step S2: Preprocess and decompose the EDA signal to identify Skin Conductance Response (SCR) events during driving. For the first... Extract the start time of each SCR event. and peak time EDA preprocessing may include steps such as denoising, baseline separation, normalization, and SCR peak detection.

[0033] Step S3: Construct a physiological trigger window and a traffic baseline window around each SCR event. Preferably, the physiological trigger window is denoted as... ,in The latency time used to cover the physiological response induced by external stimuli is preferably between 1 and 4 seconds, more preferably 4 seconds; the traffic baseline window is denoted as... ,in The background reference window length is preferably set to 2 to 6 seconds.

[0034] The physiological trigger window is used to cover the short-term dynamic process of "external traffic stimulus - driver risk perception - physiological response activation"; the traffic baseline window is used to provide local background states at an earlier time within the same driving context, in order to measure the intensity of changes in state quantities within the risk trigger window relative to the background conditions. (Appendix) Figure 2 The above-mentioned dual time window construction method is given. The figure shows the relationship between the traffic baseline window, physiological cause window and positive sample window constructed based on the SCR event start point, as well as the method of generating negative samples by sliding in the interval far from the positive sample.

[0035] III. Determination of Risk-Induced SCR Events Through Multi-Evidence Fusion Step S4: In the physiological trigger window respectively and traffic baseline window The system extracts relevant state variables for driving events and performs event-level scoring. These state variables include at least: the presence of the preceding vehicle, the time to collision (TTC), the time distance to the front of the vehicle (THW), the longitudinal braking intensity of the vehicle, the yaw rate of the vehicle, the longitudinal acceleration of the vehicle, and, if necessary, the lateral displacement or the rate of change of steering wheel angle.

[0036] Preferably, the presence status of the preceding vehicle is determined by a binary result of whether a target vehicle exists within a preset distance range ahead in the same lane; TTC and THW are taken as the minimum values ​​within the physiological induced window; yaw rate is taken as the maximum absolute value; longitudinal acceleration is taken as the maximum absolute value; and the longitudinal braking intensity of the vehicle is adaptively thresholded based on the distribution within the traffic baseline window. For example, the high or low quantile value of the braking acceleration distribution within the traffic baseline window can be taken as the dynamic threshold under the current operating conditions to avoid the fixed threshold from failing in different drivers or different scenarios.

[0037] Step S5: Construct a multi-evidence fusion scoring function. For the first... Each SCR event can be recorded as an event score. ,in For the first Weight of each piece of evidence This refers to the result or normalized intensity value of whether the corresponding judgment indicator is met. The judgment indicator may include: the preceding vehicle has a gating condition, TTC is less than the first risk threshold, THW is less than the second risk threshold, the longitudinal braking intensity of the vehicle exceeds the baseline adaptive threshold, the yaw rate of the vehicle exceeds the lateral operation threshold, and the longitudinal acceleration of the vehicle exceeds the sudden change threshold, etc.

[0038] when Greater than or equal to the preset judgment threshold If the SCR event is deemed to be a valid risk event induced by changes in traffic conditions, it is considered a non-risk-induced SCR or a low-correlation event. This step allows for the screening of SCR events with clear traffic-triggered significance from physiological fluctuations, reducing interference from purely physiological noise or non-driving factors.

[0039] IV. Construction of Risk Perception Samples Step S6: Construct positive samples centered on the screened effective risk-induced SCR events, and construct negative samples from time periods far removed from all positive samples. For the first... One valid SCR event, around its start time. Constructing a positive sample window Preferably, It lasts from 5 to 8 seconds. The window length is 2 to 4 seconds, more preferably 10 seconds for positive samples; This represents the forward retracement duration relative to the start time of the SCR event, used to cover the external traffic stimuli that triggered the driver's risk perception and the process of perception formation. This indicates the backward duration relative to the start time of the SCR event, used to preserve physiological and behavioral responses related to the state of risk perception.

[0040] When constructing negative samples, first extend all positive sample windows forward and backward by a safety interval along the time axis. To avoid contamination between positive and negative samples; then, in the remaining continuous time period, select intervals with a length greater than or equal to the preset minimum length, and use a fixed window length. and sliding step size Slide the image to generate low-risk cognitive state samples. Preferably, Take 10 seconds. Take 5 seconds.

[0041] For any given time window, the teacher's sample consists of the vehicle's motion state, the traffic vehicle's state, and the seat pressure distribution; the student's sample consists of facial movements and head posture features, non-contact pulse wave rPPG, and the seat pressure distribution. Both sets of samples share the same risk perception label and are strictly aligned within the time window.

[0042] V. Teacher Network Construction and Training Step S7: Construct a teacher network. The teacher network is used to learn discriminative knowledge that maps external scene stimuli to the driver's risk perception state. Preferably, the teacher network includes a three-channel temporal convolutional encoder and a classifier head; the three inputs correspond to the vehicle motion state sequence, the traffic vehicle state sequence, and the seat pressure distribution sequence, respectively.

[0043] Each temporal convolutional encoder may include: an input batch normalization layer, multiple one-dimensional convolutional layers, an activation layer, a normalization layer, a random deactivation layer, and an adaptive global average pooling layer. Preferably, the three convolutional blocks sequentially employ a one-dimensional convolutional structure with kernel widths of 7, 7, and 3 to accommodate feature extraction from both short-term abrupt change patterns and medium- to long-term dynamic patterns. The embedded vectors from the three encoded paths are concatenated to form the teacher fusion feature, which is then output via a fully connected classification head to show the teacher network's prediction of the driver's risk perception state. and the corresponding probability of risk perception state ,in This is the Sigmoid function.

[0044] When teachers conduct online training, they can use the binary cross-entropy loss function for supervised training, and preferably combine it with class-weighted random sampling to mitigate the imbalance problem of high- and low-risk samples.

[0045] VI. Student Network Construction and Distillation Training Step S8: Construct a student network. The student network is used to identify the driver's risk perception state from a more easily deployable non-contact modality. Preferably, the student network includes a three-channel temporal convolutional encoder and a classifier head; the three inputs are facial action and head pose feature sequences, rPPG data sequences, and seat pressure distribution sequences, respectively.

[0046] Each temporal convolutional encoder includes: an input batch normalization layer, multiple one-dimensional convolutional layers, activation layers, normalization layers, random deactivation layers, and adaptive global average pooling layers; The embedded vectors after the three encodings are concatenated to form the student fusion feature, which is then output by a fully connected classification head to show the student network's predicted value of the driver's risk perception state. and the corresponding probability of risk perception state ; Considering the differences in information density across different modalities, the output dimensions of each encoder in the student network can differ. For example, the facial motion and head pose feature channels can be encoded with higher dimensions, while the rPPG and seat pressure channels can be encoded with relatively lower dimensions to suppress overfitting of low-information-density modalities while maintaining expressive power.

[0047] During the training phase, in addition to outputting the main classification result, the student network can also output the rPPG reconstruction result. Preferably, a projection head can also be set to map the student's blending features to a public space that is comparable to the teacher's blending features.

[0048] Step S9: Train the student network using a multi-objective loss function. Preferably, the total loss function is: ; in, The binary cross-entropy loss is based on the real labels; For knowledge distillation loss; For rPPG reconstruction loss, , and These are the coefficients corresponding to each type of loss.

[0049] The knowledge distillation loss is preferably expressed as the temperature-scaled Kullback-Leibler divergence, which can be specifically written as: ,in For the temperature parameter, a value of 2 to 5 is preferred, and a value of 4 is more preferred. Kullback-Leibler divergence is calculated to measure the difference between the soft probability distributions of the student network and the teacher network. Indicates the student network in terms of temperature parameters The soft probability distribution obtained under the action, Indicates the teacher network in terms of temperature parameters The soft probability distribution obtained under the influence of the network. Preferably, if the student network's prediction value for the driver's risk perception state is... The predicted value output by the teacher network is The corresponding soft probability distributions are as follows: , By introducing temperature parameters This can make the soft probability distribution of the teacher network and the student network smoother, thereby fully preserving the teacher network's expression of the similarity of risk perception state categories and prediction uncertainty, and transmitting the above information to the student network.

[0050] The rPPG reconstruction loss preferably adopts mean squared error loss, which is used to constrain the rPPG encoder to retain the original temporal structure information and reduce feature degradation that occurs when relying solely on the classification target.

[0051] VII. Individualized Fine-tuning for Weak Supervision of Drivers Not Seen Step S10: For new drivers who have not participated in training, collect their continuous driving data and construct a sample sequence according to the same time window as the training phase. First, perform quality screening on the samples to remove invalid samples with large missing values, structural outliers, unstable facial features, severe rPPG distortion, or abnormal missing seat pressure.

[0052] Step S11: Input the samples that have passed the quality screening into the frozen teacher network to obtain the teacher logit corresponding to each sample. and risk probability Considering that the overall risk distribution of different drivers may be different, it is preferable to use the teacher's logit to perform relative ranking within the driver and construct pseudo-labels by taking samples from the high and low quantiles.

[0053] Preferably, the 95th percentile of the teacher logit distribution is calculated. and 5% quantile ; will satisfy The samples were assigned a high-risk false label 1, which will satisfy the requirements. The samples with the highest confidence are assigned the low-risk pseudo-label 0, while the remaining samples are not included in the fine-tuning. This results in a set of high-confidence pseudo-labels.

[0054] Step S12: Using the pre-trained student network parameters at the group level as initialization, fine-tune only all or some of the parameters of the student network. Preferably, the fine-tuning loss uses the consistency loss between student output and teacher soft labels, or pseudo-label supervision loss can be added. The student network is locally updated using a small number of high-confidence individual samples, allowing the model to learn the driver's stable facial expression patterns, rPPG fluctuation characteristics, and body posture change patterns, thus obtaining an individualized student network.

[0055] After fine-tuning, only non-contact or low-contact data such as facial movements and head posture features, rPPG, and seat pressure need to be input during the actual inference phase to output the driver's risk perception state. The teacher network is only used during the training or pseudo-label construction phase and does not need to run continuously in long-term in-vehicle online inference. Figure 3The diagram shows the teacher-student network configuration and individualized fine-tuning framework. The teacher network is trained based on the vehicle's motion state, traffic vehicle state, and seat pressure, while the student network is trained based on facial movements and head posture, rPPG, and seat pressure. The model transfer and adaptation are achieved through knowledge distillation and individualized fine-tuning.

[0056] Furthermore, for some of the above steps, the following alternative solutions are provided for reference: In the selection of physiological anchor points, in addition to EDA / SCR, physiological responses that can reflect short-term psychological arousal, such as electrocardiogram, pulse wave, eye movement and pupil changes, and skin temperature changes, can also be used as event backtracking anchor points, or multiple physiological signals can be combined to construct a trigger window.

[0057] In terms of distillation mechanisms, in addition to logit distillation, further methods such as feature-aligned distillation, attention distillation, contrastive distillation, and consistency regularization distillation can be adopted.

[0058] In the individualized fine-tuning stage, in addition to the 95th and 5th percentile thresholds, pseudo-label samples can also be selected based on the entropy, uncertainty, double-threshold confidence interval, clustering results, or distribution adaptive rules of the teacher output distribution.

[0059] In terms of deployment structure, teacher networks and student networks can be deployed on vehicle-mounted computing platforms, roadside computing platforms, cloud servers, or edge computing nodes, or they can be implemented using a combination of offline training, online inference, or online incremental updates.

[0060] Example 2 This embodiment further defines Embodiment 1, and uses specific application examples to illustrate Embodiment 1.

[0061] In a driving simulation environment or a real vehicle data acquisition environment, the driver's vehicle motion state, the surrounding traffic vehicle state, EDA, facial video, and seat pressure data are acquired. The EDA signals are preprocessed and SCR events are detected. The first 4 seconds before the start of each SCR are taken as the physiological trigger window, and the first 8 seconds to the first 4 seconds are taken as the traffic baseline window. Within the physiological trigger window, the minimum TTC, minimum THW, absolute value of maximum yaw rate, absolute value of maximum longitudinal acceleration, and braking intensity determined based on the baseline window quantile are calculated. When the event score meets the threshold, the SCR event is identified as a valid risk event.

[0062] Subsequently, positive samples are constructed from 7 seconds before to 3 seconds after the SCR start point; negative samples are generated in the remaining interval after expanding the safety boundary, with a length of 10 seconds and a step size of 5 seconds. The teacher network is trained using the vehicle's motion state, traffic vehicle state, and seat pressure; the student network is trained using facial movements and head pose features, rPPG, and seat pressure, where the student network is optimized using a joint objective of real label supervision loss, distillation loss, and rPPG reconstruction loss.

[0063] For a new driver, their driving data is first segmented into time windows of the same length as during training, and samples with high failure rates in facial landmark tracking, insufficient rPPG effectiveness, or large-scale loss of seat pressure are filtered out. The remaining samples are then input into the teacher network, with the top 5% of the teacher network's output logit values ​​used as high-risk pseudo-labels and the bottom 5% as low-risk pseudo-labels. The student network is then fine-tuned in a few rounds. After individualized fine-tuning, the driver's risk perception state can be identified solely based on facial features, rPPG, and seat pressure.

[0064] Example 2 This embodiment, based on the method given in Embodiment 1, presents a driver risk perception state identification system based on heterogeneous modal knowledge distillation, used to implement the above method.

[0065] The system includes: The data acquisition and synchronization module is used to collect multi-source heterogeneous data on "drivers, vehicles, and traffic" and complete time alignment. The physiological event detection module is used to detect skin conductance response (SCR) events and extract the event start time, peak time, and amplitude. A dual-time-window construction module is used to construct physiological trigger windows and traffic baseline windows; The multi-evidence fusion judgment module is used to determine the risk inducement of SCR events based on the state variables within the window; The sample construction module is used to generate positive samples, negative samples, and aligned samples of teacher and student networks. The teacher network training module is used to train teacher networks based on scenario-side modalities. The student network training module is used to perform joint training based on real labels, distillation loss, and non-contact pulse wave rPPG reconstruction loss; The individualized fine-tuning module is used to fine-tune the student network without drivers based on high-confidence pseudo-labels of high and low quantiles; The online reasoning module is used to output the identification results of whether the driver's cognitive state is high-risk or low-risk.

Claims

1. A method for identifying driver risk perception state based on heterogeneous modal knowledge distillation, characterized in that, The method includes the following steps: S1. Multi-source data acquisition and time synchronization: Collect multi-source heterogeneous data on "driver-vehicle-traffic" during the driving process and complete time alignment; S2. Physiological Event Detection and Retrospective Labeling: Identify skin conductance response (SCR) events during driving by using driver EDA data of skin conductance activity (EDA) and construct physiological trigger windows and traffic baseline windows around each SCR event. S3. Determination of risk-induced SCR events through multi-evidence fusion: Extract relevant state quantities of driving events from the physiological induced window and the traffic baseline window respectively, construct a multi-evidence fusion scoring function, and screen out effective risk-induced SCR events; S4. Risk perception sample construction: positive samples are constructed with the screened effective risk-induced SCR events as the center, and negative samples are constructed with the time periods far away from all positive samples. S5. Teacher Network Construction and Training: Construct a teacher network to learn discriminative knowledge that maps external scene stimuli to the driver's risk perception state. The teacher network sample consists of vehicle motion state data, traffic vehicle state data, and seat pressure distribution data; train the teacher network. S6. Student Network Construction and Distillation Training: A student network is constructed to identify the driver's risk perception state from non-contact modalities. The student network samples consist of driver facial movement and head posture feature data, heart rate detection data, and seat pressure distribution data. A multi-objective loss function including knowledge distillation loss is used to train the student network. S7. Individualized fine-tuning for weak supervision of unseen drivers: For new drivers who have not participated in training, their continuous driving data is collected, and a sample sequence is constructed according to the same time window as the training phase; the samples that pass the quality screening are input into the frozen teacher network to obtain the teacher network's prediction value of the driver's risk perception state and the corresponding probability of risk perception state for each sample. Using the pre-trained student network parameters as initialization, the parameters of the student network are fine-tuned. After fine-tuning, the driver's facial movements and head posture feature data, heart rate detection data, and seat pressure distribution data are input into the actual inference stage, and the risk perception state identification result of the driver is output.

2. The method for identifying driver risk perception state based on heterogeneous modal knowledge distillation according to claim 1, characterized in that, In step S1: Multi-source heterogeneous data includes at least: vehicle motion state data, traffic vehicle state data, driver skin conductance data (EDA), driver facial video data, facial movement and head posture feature data extracted from facial videos, heart rate detection data, and seat pressure distribution data.

3. The method for identifying driver risk perception state based on heterogeneous modal knowledge distillation according to claim 2, characterized in that, In step S2: The EDA signal is preprocessed and decomposed to identify skin conductance response (SCR) events during driving. Extract the start time of each SCR event. and peak time EDA preprocessing includes denoising, baseline separation, normalization, and SCR peak detection; A physiological trigger window and a traffic baseline window are constructed around each SCR event. The physiological trigger window is denoted as . ,in The traffic baseline window is used to cover the latency time for external stimuli to elicit a physiological response. ,in Use the background as a reference for the window length; The physiological trigger window is used to cover the dynamic process of "external traffic stimulus - driver risk perception - physiological response initiation"; the traffic baseline window is used to provide the local background state at an earlier time in the same driving context, and is used to measure the intensity of change of state quantities within the physiological trigger window relative to the background conditions.

4. The method for identifying driver risk perception state based on heterogeneous modal knowledge distillation according to claim 3, characterized in that, In step S3: The driving event-related state quantities include at least: the presence of the preceding vehicle, the time to collision (TTC), the headway (THW), the longitudinal braking intensity of the vehicle, the yaw rate of the vehicle, and the longitudinal acceleration of the vehicle. The construction of the multi-evidence fusion scoring function is specifically as follows: for the first... There are 1 SCR event, and its event score is denoted as . ,in For the first Weight of each piece of evidence The result or normalized intensity value is the result of whether the corresponding judgment indicator is valid; the judgment indicator includes: the preceding vehicle has a gating condition, TTC is less than the first risk threshold, THW is less than the second risk threshold, the longitudinal braking intensity of the vehicle exceeds the baseline adaptive threshold, the yaw rate of the vehicle exceeds the lateral operation threshold, and the longitudinal acceleration of the vehicle exceeds the sudden change threshold. when Greater than or equal to the preset judgment threshold If the event is deemed to be a valid risk event induced by changes in traffic conditions, it will be classified as such; otherwise, it will be considered a non-risk-induced SCR or a low-relevance event.

5. The method for identifying driver risk perception state based on heterogeneous modal knowledge distillation according to claim 4, characterized in that, Step S4 is as follows: For the One valid SCR event, around its start time. Constructing a positive sample window , Indicates the duration of forward backtracking. Indicates the duration of backward continuation; When constructing negative samples, first extend all positive sample windows forward and backward by a safety interval along the time axis. Then, in the remaining continuous time period, select intervals with a length greater than or equal to the preset minimum length, and use a fixed window length. and sliding step size Slide the image to generate low-risk cognitive state samples.

6. The method for identifying driver risk perception state based on heterogeneous modal knowledge distillation according to claim 5, characterized in that, In step S5: The teacher network consists of a three-way temporal convolutional encoder and a classification head; The three inputs correspond to the vehicle motion state sequence, the traffic vehicle state sequence, and the seat pressure distribution sequence, respectively. Each temporal convolutional encoder includes: an input batch normalization layer, multiple one-dimensional convolutional layers, activation layers, normalization layers, random deactivation layers, and adaptive global average pooling layers; The three encoded embedding vectors are concatenated to form the teacher fusion feature, which is then output by the fully connected classification head to show the teacher network's predicted value of the driver's risk perception state and the corresponding probability of the risk perception state. During online training, teachers use a binary cross-entropy loss function for supervised training, combined with category-weighted random sampling.

7. The method for identifying driver risk perception state based on heterogeneous modal knowledge distillation according to claim 6, characterized in that, In step S6: The student network consists of a three-channel temporal convolutional encoder and a classifier head; the three inputs are facial motion and head pose feature sequences, heart rate detection data sequences, and seat pressure distribution sequences, respectively. Each temporal convolutional encoder includes: an input batch normalization layer, multiple one-dimensional convolutional layers, activation layers, normalization layers, random deactivation layers, and adaptive global average pooling layers; The three encoded embedding vectors are concatenated to form the student fusion feature, which is then output by the fully connected classification head to show the student network's predicted value of the driver's risk perception state and the corresponding probability of the risk perception state. The student network is trained using a multi-objective loss function, with the total loss function being: ,in, For binary cross-entropy loss based on real labels, For knowledge distillation loss, For rPPG reconstruction loss; , and These are the coefficients corresponding to each type of loss.

8. The method for identifying driver risk perception state based on heterogeneous modal knowledge distillation according to claim 7, characterized in that, In step S7: When fine-tuning, the fine-tuning loss adopts the consistency loss between the student network output and the teacher network soft label, and superimposed with the pseudo-label supervision loss. The student network is locally updated using several high-confidence individual samples, thereby obtaining an individualized student network.

9. A driver risk cognitive state identification system based on heterogeneous modal knowledge distillation, characterized in that, The system is used to implement the method as described in any one of claims 1-8, the system comprising: The data acquisition and synchronization module is used to collect multi-source heterogeneous data on "drivers, vehicles, and traffic" and complete time alignment. The physiological event detection module is used to detect skin conductance response (SCR) events and extract the event start time, peak time, and amplitude. A dual-time-window construction module is used to construct physiological trigger windows and traffic baseline windows; The multi-evidence fusion judgment module is used to determine the risk inducement of SCR events based on the state variables within the window; The sample construction module is used to generate positive samples, negative samples, and aligned samples of teacher and student networks. The teacher network training module is used to train teacher networks based on scenario-side modalities. The student network training module is used to perform joint training based on real labels, distillation loss, and non-contact pulse wave rPPG reconstruction loss; The individualized fine-tuning module is used to fine-tune the student network without drivers based on high-confidence pseudo-labels of high and low quantiles; The online reasoning module is used to output the identification results of the driver's risk perception state.

Images