Driver cognitive load prediction method, system, device and storage medium

Abstract

The present application relates to the technical field of intelligent cockpit driver state monitoring and vehicle-mounted machine learning risk early warning, in particular to a driving cognitive load prediction method, which divides the physiological response groups of driving cognitive load through a questionnaire classification model, matches a special group timing model, synchronously collects the stable driving baseline and multi-modal physiological characteristics of the driver and completes individualized normalization, follows the causal timing rules, only uses historical data to construct a rolling sequence input model to obtain single-sample load determination, aggregates effective samples to calculate stable high cognitive load prediction probability within a set output period, outputs vehicle-mounted intervention control instructions according to the probability, adapts to the differentiated physiological responses of the driver, avoids timing reasoning data leakage, suppresses the prediction jitter caused by sensor noise, and can stably realize real-time monitoring of the high cognitive load of the vehicle-mounted driver and linkage of cockpit active safety intervention, and adapts to the real-time risk control landing scene of the intelligent cockpit.

Description

Technical Field

[0001] This invention relates to the field of intelligent cockpit driver state monitoring and in-vehicle machine learning risk warning technology, and in particular to a driver cognitive load prediction method, system, device and storage medium. Background Technology

[0002] Driver cognitive load refers to the total amount of information a driver processes while driving, monitoring driver assistance systems, operating in-vehicle interactive devices, and performing takeover tasks. Cognitive load increases rapidly when road conditions are complex, navigation information is dense, temporary takeover commands are issued, or the driver is in poor physical or mental condition. Prolonged exposure to high cognitive load weakens the driver's ability to allocate attention, perceive environmental situations, and maintain vehicle operational stability, significantly prolongs the driver's reaction time, and significantly increases the probability of misoperation and road traffic accidents.

[0003] Existing driver cognitive load identification technologies generally employ a unified machine learning model, training a single classification model based on ECG, heart rate variability (HRV), eye movement and blinking, skin conductance, and vehicle dynamics signals, and outputting cognitive load levels window by window. This approach achieves high accuracy under simulation testing conditions with the same driver and randomly divided time-series samples. However, its generalization ability significantly decreases under strict cross-driver testing scenarios. The core shortcomings can be categorized into four points: 1. Significant differences in individual physiological responses among drivers: Drivers with different personalities, emotional stability, and stress coping traits exhibit completely opposite trends in autonomic nervous system and eye movement responses when faced with the same cognitive load task; some drivers blink more frequently under high cognitive load, while others blink less frequently under high cognitive load, and a unified model cannot adapt to the bidirectional physiological changes. 2. Online real-time inference suffers from time-series data leakage: Traditional offline training schemes introduce future time window data during inference, which violates the constraints of real-time causal inference where vehicle data streams arrive sequentially and can only use historical data. The accuracy of offline simulation cannot be reproduced in real vehicle deployment scenarios. 3. Single-point model output is susceptible to noise interference: Sensor frame loss, sparse blink events, instantaneous ECG fluctuations, and in-vehicle light interference can cause frequent jumps in single-sample model judgments. Directly outputting instantaneous judgment results will trigger high-frequency false alarms in the cabin, seriously affecting the driving experience. 4. Lack of a complete in-vehicle deployment loop: Existing technologies only output cognitive load level values ​​and do not provide standardized probability output logic, making it impossible to directly link cockpit interaction, assisted driving, and cloud risk recording modules, resulting in high difficulty in engineering deployment and adaptation.

[0004] In summary, the existing solution cannot achieve automatic grouping and adaptation of physiological responses to differentiated physiological patterns through driver questionnaires, lacks personalized normalization methods for drivers' own baselines, does not adopt a pure historical causal temporal reasoning architecture, and lacks an interval probability aggregation smoothing and anti-shaking mechanism. This results in poor cross-driver generalization, distortion of real-vehicle reasoning, frequent false alerts, and the inability to form a standardized in-vehicle intervention closed loop.

[0005] Therefore, the industry urgently needs a driver cognitive load prediction solution that can distinguish individual differences in driver physiological response, meet the constraints of in-vehicle causal temporal reasoning, suppress single-point noise jitter, and fully adapt to closed-loop safety intervention in intelligent cockpits. Summary of the Invention

[0006] The purpose of this invention is to overcome the problems of the prior art and provide a method, system, device, and storage medium for predicting driver cognitive load. This addresses the shortcomings of existing driver cognitive load prediction technologies, such as difficulty in adapting to individual physiological differences among drivers through questionnaire segmentation, poor generalization of general models, susceptibility to temporal data leakage during online inference, and susceptibility to sensor noise interference leading to false alarms. Furthermore, traditional solutions can only output simple load assessment results, failing to form standardized probability outputs and hierarchical in-vehicle intervention loops, making them unsuitable for real-time safety monitoring requirements in smart cockpits and unable to effectively mitigate driving risks caused by high driver cognitive load.

[0007] The above objectives are achieved through the following technical solutions: A method for predicting driver cognitive load includes the following steps: S1. Obtain the driver's questionnaire data and generate a questionnaire feature vector based on the questionnaire data; S2. Input the questionnaire feature vector into a pre-trained questionnaire classification model and output the cognitive load physiological response family corresponding to the driver; the cognitive load physiological response family is obtained based on the changing trend of the driver's blink frequency relative to his own stable driving baseline under different cognitive load levels; S3. Match the corresponding clustered cognitive load time series model based on the cognitive load physiological response family; S4. Continuously collect the driver's smooth driving baseline feature stream and multimodal online feature stream, calculate the subject's intra-subject normalized statistics based on the smooth driving baseline feature stream, and perform personalized normalization preprocessing on the multimodal online feature stream based on the smooth driving baseline feature stream to obtain the preprocessed online feature stream. S5. Construct a fixed-length rolling time series sequence for the online feature stream after normalization preprocessing, input the rolling time series sequence into the matched clustered cognitive load time series model, and output the single-sample cognitive load judgment result. S6. Within a preset output interval, statistically analyze the judgment results of all valid single samples, calculate the ratio of the number of samples judged as having high cognitive load to the total number of valid samples, and calculate the stable high cognitive load prediction probability; generate vehicle control commands based on the stable high cognitive load prediction probability; In this step, all valid single samples refer to binary judgment results that meet the baseline readiness condition, the timing buffer duration constraint, the window data quality threshold, and the model successfully completes inference within the preset output interval.

[0008] As a further optimization of this method, the questionnaire classification model is trained in the following way: collecting questionnaire data and eye movement data under multi-gradient cognitive load tasks from historical drivers; calculating the change in blink frequency of each historical driver relative to their own driving baseline under different cognitive load levels; distinguishing multiple physiological response types based on the trend of the change with the increase of cognitive load level, and merging them into two cognitive load physiological response families; training the questionnaire classification model with the questionnaire feature vector of historical drivers as input and the physiological response families as supervision labels.

[0009] As a further optimization of this method, the multimodal online feature stream includes eye movement behavior features and electrophysiological features; the eye movement behavior features include the eye aspect ratio time series and blink frequency calculated based on eye key points; the electrophysiological features include mean heart rate and heart rate variability index; eye movement behavior features and electrophysiological features of the same time period are synchronously aligned and fused to form a multidimensional combined feature window.

[0010] As a further optimization of this method, the preprocessing includes: in step S5, strictly following the causal temporal constraints, window alignment and rolling temporal sequence construction are completed using historical arrived temporal data; the clustered cognitive load temporal model includes parallel eye-tracking flow encoders and physiological flow encoders, each encoder containing a one-dimensional convolutional layer, an activation layer, a recurrent neural network layer and a temporal attention layer.

[0011] As a further optimization of this method, the clustered cognitive load time series model is trained by partitioning the driver dataset. Historical drivers are stratified according to the cognitive load physiological response family. All time series samples of the same historical driver are assigned to one of the three sets: training set, validation set, or test set. Drivers corresponding to different datasets do not overlap at all. The online inference process uses the current time and the time series data that have been reached before.

[0012] As a further optimization of this method, based on the single-sample cognitive load determination result output in step S5, the frequency of high cognitive load determination within a preset output interval is counted, and the stable high cognitive load prediction probability is calculated based on the proportion of effective samples within the interval. The stable high cognitive load prediction probability is compared with a preset threshold. When the determination logic corresponding to the probability value being greater than or equal to the threshold or the moving average threshold of the continuous output interval is met, an in-vehicle intervention action is executed. The in-vehicle intervention action includes at least one of reducing the information push frequency, outputting rest reminders, and adjusting the assisted driving monitoring level.

[0013] This solution also provides a driver cognitive load prediction system for implementing the above method, including: The questionnaire feature generation module is used to obtain the driver questionnaire selection data, and generate a standardized questionnaire feature vector after quantification. The questionnaire typing module is used to acquire the driver's questionnaire data and generate questionnaire feature vectors. The questionnaire feature vectors are then input into a pre-trained questionnaire typing model, which outputs a family of physiological responses to cognitive load. The model routing module is used to match a specific and adapted clustered cognitive load time series model based on the cognitive load physiological response family. The online feature processing module is used to receive the driver's multimodal online feature stream, collect the smooth driving baseline feature stream, and complete the subject-specific personalized normalization preprocessing of the online feature stream based on the baseline feature stream; The temporal reasoning module is used to construct a rolling temporal sequence using only historical temporal data, input the preprocessed multimodal online feature stream into the matched clustered cognitive load temporal model, and output the single-sample cognitive load judgment result. The probability aggregation module is used to aggregate all valid single-sample judgment results within a preset output interval and calculate the stable high cognitive load prediction probability. The control command generation module is used to generate vehicle control commands based on the predicted probability of high cognitive load.

[0014] As a further optimization of this system, the system also includes a feature acquisition terminal, an application execution terminal, and a cloud storage module; the feature acquisition terminal includes at least one of an in-vehicle camera, an electrocardiogram sensor, and a wearable device, used to collect the driver's eye movement behavior characteristics, electrocardiogram physiological characteristics, and smooth driving baseline data; the application execution terminal receives in-vehicle control commands generated based on interval aggregation probability, executes corresponding in-vehicle intervention actions in layers, and synchronously uploads driving risk events to the cloud storage module.

[0015] This solution provides an in-vehicle electronic device, including a processor, a memory, and a program or instructions stored in the memory and executable on the processor. When the program or instructions are executed by the processor, they implement the steps of the driver cognitive load prediction method described above.

[0016] This solution also provides a computer-readable storage medium storing a program or instructions that, when executed by a processor, implement the steps of the driver cognitive load prediction method described above.

[0017] The driver cognitive load prediction method, system, device, and storage medium provided by this invention divides the physiological response family of cognitive load according to the blink change trend of drivers under different loads and matches the exclusive cluster time series model with the questionnaire classification model. This achieves adaptation to the differentiated physiological response patterns of different drivers and improves the cross-driver recognition generalization ability. By collecting the driver's stable driving baseline to complete the personalized normalization preprocessing within the subject, it eliminates the deviation of the driver's individual basic physiological indicators and reduces the interference of individual differences on the recognition results. By using only historical time series data to construct rolling sequences to perform causal inference, it eliminates time series data leakage and ensures that the inference effect in the real vehicle is consistent with the offline simulation. By aggregating effective samples within the interval to calculate the high cognitive load prediction probability and regenerating the vehicle control command, it suppresses the single-point judgment jitter caused by sensor noise and reduces high-frequency false alarms in the cockpit. It completely builds a closed loop of vehicle safety risk control with active intervention for physiological load prediction and is suitable for various intelligent cockpit real-time monitoring scenarios. Attached Figure Description

[0018] Figure 1 This is a flowchart of the driver cognitive load prediction method described in this invention; Figure 2 This is a schematic diagram of the questionnaire typing and blink response family construction process in the driver cognitive load prediction method of the present invention; Figure 3 This is a schematic diagram of the offline data processing and feature alignment process in the driver cognitive load prediction method of the present invention; Figure 4 This is a schematic diagram of the dual-stream time-series cognitive load model structure in the driver cognitive load prediction method of the present invention; Figure 5 This is a schematic diagram of the strict cross-driver training and testing process in the driver cognitive load prediction method of the present invention; Figure 6 This is a schematic diagram of the online state management and rolling inference process in the driver cognitive load prediction method of the present invention; Figure 7 This is a schematic diagram illustrating the calculation of the probability of high cognitive load based on the output interval in the driver cognitive load prediction method described in this invention. Detailed Implementation

[0019] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. The described embodiments are merely some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] A method for predicting driver cognitive load consists of two core modules: an offline model training phase and an in-vehicle online real-time inference phase. It is equipped with a prediction system, an in-vehicle electronic device, and a computer-readable storage medium to execute the method.

[0021] The offline phase includes the following steps: driver physiological response family classification based on blink load variation trends, questionnaire typology model training, multimodal feature standardization extraction, and hierarchical cross-driver training of the clustered dual-stream time series model: Step 1: Historical Subject Data Collection and Calculation of Blink Frequency Changes Complete experimental data were collected from multiple historical drivers. The experiment included four scenarios: a stable autonomous driving baseline task, a low-gradient cognitive load task, a medium-gradient cognitive load task, and a high-gradient cognitive load task. Eye movement (EAR) time-series data for each driver were recorded simultaneously throughout the experiment. The change in blink frequency relative to the driver's own driving baseline at each cognitive load level was calculated using the following formula: , In the formula: For the first A famous historical driver Changes in blink frequency under graded cognitive load; For the driver Average blink rate at the load level; The average blink frequency under baseline, stable driving conditions for the driver; These represent three levels of cognitive load tasks: low, medium, and high.

[0022] Step 2: Classify and merge the four types of physiological responses into two major response families. in accordance with Based on the changing trend of cognitive load levels, four types of physiological responses are identified: Type 1 (Key=1): The change in blink frequency increases from low load to medium load, and the change in blink frequency decreases from medium load to high load; Type 2 (Key=2): The change in blink frequency decreases from low load to medium load, and the change in blink frequency increases from medium load to high load; Type 3 (Key=3): Cognitive load continues to increase, and the change in blink frequency increases synchronously and continuously; Type 4 (Key=4): Cognitive load continues to increase, while the change in blink frequency continues to decrease synchronously; Merging and grouping rules: Key1 and Key3 are merged into a high-load blink rising response family; Key2 and Key4 are merged into a high-load blink falling response family.

[0023] Step 3: Questionnaire typing model construction and training Standardized questionnaire data were collected from each historical driver. The questionnaire items covered at least one category: personality, temperament, emotional stability, conscientiousness, stress coping, task persistence, risk attitude, driving behavior, frustration, aggressive driving tendency, danger sensitivity, death sensitivity, general trust tendency, and social avoidance tendency. All questionnaire items were quantified and concatenated to generate a driver-specific questionnaire feature vector. ; Questionnaire feature vector The questionnaire-based classification model is trained using the driver's response family as input and the driver's response family as supervised label. The model outputs a probability vector of the driver belonging to each response family: , In the formula: Let G be the probability that the i-th driver belongs to the g-th response family, and G be the total number of response families. In this scheme, G=2.

[0024] Step 4: Standardized extraction and temporal alignment of eye-tracking and electrocardiogram features The raw eye-tracking data is an 8Hz EAR time series. The basic formula for calculating the eye aspect ratio (EAR) is as follows: , In the formula, Indicates the key points at the inner corner of the left eye. Indicates the key points at the outer corner of the right eye. Indicates key points of the left upper eyelid. Indicates key points of the lower eyelid of the left eye. Indicates key points of the upper eyelid of the right eye. This indicates the key point on the lower eyelid of the right eye, and ||·|| represents the Euclidean distance.

[0025] A 10-second window and a 5-second sliding step size are uniformly used to extract EAR timing sequences. Window data is retained when the proportion of effective EAR frames within the window is ≥80%. Six types of eye-tracking features are calculated window by window, with complete calculation formulas provided: 1) Window EAR mean: In the formula, Indicates the first EAR mean within a window; Indicates the first Valid within a window Frame rate; Indicates the first The first window 1 valid frame value; 2) Minimum EAR for the window: In the formula, Indicates the first Minimum EAR value within a window; and Same meaning as above; 3) Window EAR standard deviation: In the formula, Indicates the first EAR standard deviation within a window; , and Same meaning as above; 4) Window blinking frequency: In the formula, Indicates the first Blinking frequency within a window; Indicates the first The duration of each window; Indicates the first The first window Is the frame a blink frame? Indicates the first Number of valid frames within a window; This indicates an indicator function that takes the value 1 if the condition is true and 0 otherwise. 5) Average blink interval between adjacent windows: In the formula, Indicates the first Average blink interval within a window; Indicates the first Number of blink start points within a window; Indicates the first The first window The starting frame number of a blink; Indicates the eye-tracking sampling frame rate; 6) Number of consecutive segments with low EAR in the window: In the formula, Indicates the first The number of consecutive low EAR segments within a window; the first term is used to count the segments that are already in the low EAR state in the first frame of the window, and the summation term is used to count the starting point of segments that enter the low EAR state from the non-low EAR state within the window; , , and Same meaning as above; Driver-specific low EAR threshold: In the formula, Indicates the first Low EAR threshold for famous drivers; Represents the 20th quantile function; Indicates the first Driver Baseline segment The EAR value of the frame.

[0026] ECG data is extracted synchronously by the upstream preprocessing module, using a 10-second window and a 5-second step size to output two types of physiological features: ECG_Rate_Mean (average heart rate within the window) and HRV_RMSSD (root mean square difference in heart rate variability). The system uses the window end timestamp as the unique alignment identifier to stitch together the 6-dimensional eye movement features and 2-dimensional ECG features within the same time period. When the number of missing frames in a single window is ≤5, linear interpolation is used to fill in the missing frames; if the number of missing frames exceeds 5, the data in that window is directly discarded.

[0027] Step 5: Rigorous cross-driver dataset partitioning and offline training of clustered time series models All historical driver sets are stratified according to response families, with the rising response family and falling response family driver datasets being independent of each other; a driver-level dataset splitting rule is adopted, where all time-series samples of the same historical driver are only allowed to be assigned to one of the training set, validation set, or test set, and the drivers in the three datasets are completely mutually exclusive and have no overlap. Two sets of dual-stream time series models were trained independently, one for rising response families and the other for falling response families. During training, feature standardization parameters, questionnaire item selection rules, model hyperparameters, high and low load classification thresholds, and early stop strategies were fitted using only the training and validation sets of driver data. The test set was used only for evaluating the generalization performance of unfamiliar drivers. The model evaluation metrics included accuracy, macroF1, weighted F1, and confusion matrix.

[0028] The complete internal architecture of the clustered dual-stream time series model is as follows: The single-set cognitive load binary classification time series model is a dual-stream parallel encoder architecture. The model input is a 20-window × 8-dimensional fused time series feature sequence. The two encoders independently process 6-dimensional eye-tracking EAR features and 2-dimensional electrocardiogram physiological features. The layer order of the single encoder is: one-dimensional convolution Conv1D (16 channels, kernel size 3), ReLU activation layer, LSTM recurrent neural network layer (16 hidden units), and temporal attention layer.

[0029] 1. Complete calculation process for time attention Single timing step Attention scoring formula: ; Output the timing hidden state for the LSTM layer. These are the trainable attention parameters for the model; Normalized attention weights: L=20 represents the total number of windows contained in a single time series sample; Single-modal global timing representation: ; 2. Multimodal feature fusion and sample classification output Global temporal representation of eye-tracking modality Global temporal representation of physiological modalities Learnable modal weights Weighted splicing and fusion: , Fusion features The vector is fed into a fully connected ReLU feature extraction layer for dimensionality reduction, and finally fed into a softmax binary classification layer to output the class probability vector. The formula for determining sample-level cognitive load is as follows: , In the formula, Indicates the sample-level predicted category; This represents the category selection operation that maximizes the category probability. Indicates the candidate category; Indicates a low cognitive load category; Indicates a high cognitive load category; Indicate category The corresponding predicted probability.

[0030] If the questionnaire typing model outputs a multi-response family probability distribution When using weighted hybrid expert routing, the formula for calculating the overall probability of high load at the sample level is as follows: , In the formula, This represents the sample-level high load probability obtained by fusing response family probabilities; Indicates the number of response families; This indicates that the target driver belongs to the first category. The probability of a family of responses; Indicates the first Each response family corresponds to a clustering model for the input sequence. High load probability in the output; This represents the normalized rolling time series feature sequence.

[0031] The entire online inference process strictly adheres to causal temporal constraints, using only the current moment and previously received historical time-series data. It utilizes a driver's stable driving baseline for personalized normalization, constructs a fixed-length rolling time-series sequence for inference, and smoothly outputs stable load prediction results through interval sample probability aggregation. It prohibits the use of future data beyond the current moment. The complete steps are as follows: S101. The vehicle-mounted terminal collects questionnaire selection data from the target driver and generates a standardized questionnaire feature vector after quantitative processing. S102. Input the questionnaire feature vector into the offline trained questionnaire classification model, and output the probability distribution of the driver's response family or multiple response families. S103. The model routing module matches the corresponding clustered dual-stream time series model. If the model outputs a response family probability distribution, three matching methods can be selected: maximum a posteriori hard route, Top-K sparse route, and probability-weighted hybrid expert route. S104. The vehicle-mounted sensors continuously transmit an online feature stream in 5-second data block format, while simultaneously continuously acquiring baseline feature streams of driving behavior; the online feature streams include all eye-tracking and ECG standardized window features; S105. Personalized normalization within subjects: Before online inference, a smooth driving baseline feature stream of no less than 60 seconds must be collected, and the specific normalized statistics of the two types of features must be calculated separately. Normalization formula for eye movement behavior features: In the formula The baseline mean. Baseline standard deviation It is a minimal constant that can be divided into zero; Normalization formula for electrophysiological characteristics of the heart: In the formula is the baseline median, and IQR is the baseline interquartile range; If the baseline acquisition time is insufficient, baseline features are missing, or non-finite values ​​appear after normalization, the system outputs a baseline not ready or low confidence state and skips this round of model inference.

[0032] S106, Constraints for Constructing Rolling Time Series: Calculate the shortest effective inference data duration: 10s for a single window, 5s for the sliding step between adjacent windows, and a fixed 20 consecutive windows per scrolling sequence. The online rolling buffer continuously stores time series windows. If the total duration of the time series in the buffer is less than 105s, it will not be sent to the cluster model for inference. All windows are constructed only from historically arrived data, and no future time series data are involved throughout the process.

[0033] S107. The normalized 20-segment window time series is fed into the matched clustered dual-stream time series model, and the output is the binary judgment result of high and low cognitive load of single sequence samples. S108, Interval Probability Aggregation Calculation: The preset output interval can be selected as 30s / 60s / 120s / 2 minutes. All valid inference samples within the output interval are statistically analyzed. The formula for calculating the prediction probability under high cognitive load is as follows: , In the formula, This represents the total number of samples identified as having a high load within the output interval. The total number of valid samples is calculated by removing those with baseline anomalies, missing data, insufficient buffer time, and failed format validation; all types of abnormal data are excluded from the denominator. .

[0034] S109, Vehicle-mounted closed-loop control execution: The calculated Compared with the preset static threshold and the moving average threshold of the continuous output interval, when the triggering condition is met, the vehicle control operation is executed: reduce the frequency of vehicle information push, delay complex human-machine interaction pop-ups, provide voice / seat tactile rest reminders, improve the monitoring level of assisted driving, and simultaneously upload this high-load risk event to the cloud for recording and storage; after execution, the interval statistical count is cleared, and the next output interval cyclic detection is started to form a complete closed loop.

[0035] This solution provides a driver cognitive load prediction system, which includes a three-layer architecture: feature acquisition terminal, vehicle computing terminal, and application execution terminal; Feature acquisition hardware: vehicle-mounted forward-looking eye-tracking camera, steering wheel / seat ECG sensor, wearable device gateway, smart cockpit SDK; responsible for collecting questionnaire text, eye-tracking frame sequences, ECG physiological signals, and smooth driving baseline data. The onboard computing terminal has a built-in load prediction device and includes seven functional modules: questionnaire feature generation module, questionnaire classification module, model routing module, online feature processing module, time series inference module, probability aggregation module, and control command generation module. Among them, the questionnaire feature generation module receives driver questionnaire data and quantifies it to generate standardized questionnaire feature vectors; the control command generation module receives the high cognitive load prediction probability output by the probability aggregation module, generates corresponding onboard control commands, and completes all normalization, time series modeling, and probability calculation logic.

[0036] Application execution hardware includes: in-vehicle central control display, voice audio system, seat vibration and haptic unit, driver assistance controller, and cloud-based driving risk storage platform; it receives in-vehicle control commands generated based on interval aggregation probability, executes corresponding levels of in-vehicle safety intervention actions in a hierarchical manner, and synchronously uploads driving risk events corresponding to high cognitive load to the cloud-based driving risk storage platform for record keeping.

[0037] The system of this invention also includes a feature acquisition terminal, an in-vehicle computing terminal, and an application execution terminal. The questionnaire feature generation module is used in conjunction with the questionnaire typing module, and the probability aggregation module outputs a stable, high-load prediction probability to the control command generation module. The control command generation module then outputs in-vehicle control commands to the application execution terminal.

[0038] This solution provides an in-vehicle electronic device, the hardware carrier of which is an in-vehicle cockpit domain controller, with a built-in multi-core processor and in-vehicle solid-state memory; the memory stores the complete computer execution program, and the program runs automatically in the background after the vehicle is powered on, collecting sensor data in real time, performing timing inference, and outputting in-vehicle control commands.

[0039] This solution also provides a computer-readable storage medium, which can be a USB flash drive, vehicle solid-state storage, server hard drive, or ROM. The medium stores complete program code. When the code is loaded and run by the processor, it completely reproduces the entire cognitive load prediction method process of this invention. The medium supports independent mass production, disassembly, and replacement.

[0040] In one embodiment, such as Figure 1 As shown, this embodiment provides a method for predicting driver cognitive load, including historical database construction, questionnaire typing, online feature processing, model inference, probability aggregation, and in-vehicle closed-loop application. The specific steps are as follows: S101. Obtain the questionnaire data of the target driver and generate a questionnaire feature vector based on the questionnaire data; S102. Input the questionnaire feature vector into a pre-trained questionnaire classification model and output the cognitive load physiological response type, response family, or response family probability distribution of the target driver. S103. Based on the cognitive load physiological response type, response family or response family probability distribution, determine the clustered cognitive load binary classification time series model corresponding to the target driver; S104. Receive the baseline feature stream and online feature stream of the target driver's driving behavior, wherein the online feature stream includes eye movement behavior features and electrophysiological features; S105. Calculate the intra-subject normalized statistic of the target driver based on the baseline feature stream of the driving behavior, and use the normalized statistic to normalize the online feature stream; for example... Figure 5 As shown; S106. Under the condition of using only the data that has arrived at the current time and before the current time, perform window alignment and rolling sequence construction on the normalized online feature stream; S107. Input the rolling sequence into the grouped cognitive load binary classification time series model and output the sample-level low cognitive load or high cognitive load determination. S108. Within a preset output interval, the frequency of effective samples that are determined to be high cognitive load is statistically analyzed, and the frequency of effective samples is used as the high cognitive load prediction probability of the preset output interval. S109. Generate vehicle application control results based on the high cognitive load prediction probability, and enter the next preset output interval after outputting the vehicle application control results.

[0041] In one embodiment, such as Figure 2 As shown in the example, this embodiment illustrates the complete process of offline driver physiological response family segmentation and questionnaire typing model training. Physiological response family segmentation refers to the process of clustering drivers with similar physiological response patterns into two main categories—either increased or decreased blinking frequency under high cognitive load levels—by analyzing the trends in blinking frequency relative to their stable driving baseline under different cognitive load levels during the offline training phase.

[0042] During the historical database construction phase, baseline driving behavior tasks and low, medium, and high cognitive load tasks were collected for each historical driver. The system calculated the change in blink frequency for each task relative to the individual's own driving behavior baseline.

[0043] , In the formula, Indicates the first A number of historical drivers at the cognitive load level The change in blink frequency; Indicates the first A number of historical drivers at the cognitive load level Average blink frequency; Indicates the first Historical drivers at the baseline of driving behavior ( Average blink frequency at baseline; This indicates a low, medium, or high cognitive load task.

[0044] Based on the trend of change, the first, second, third and fourth types are determined, and the first and third types are merged into a high-load blink rising response family, and the second and fourth types are merged into a high-load blink falling response family.

[0045] , In the formula, Indicates the first Feature vectors of questionnaires from drivers; The parameter is Questionnaire typing model; Represents the family probability vector of the response; This indicates that the i-th driver belongs to the i-th driver. The probability of a family of responses; Indicates the number of response families.

[0046] In one embodiment, such as Figure 3 As shown, this embodiment fully illustrates the entire calculation logic for EAR eye-tracking feature extraction, alignment, and standardization, as well as ECG feature extraction. The eye-tracking data is derived from the EAR time series, and the ECG features are derived from the heart rate and HRV features output by the upstream ECG processing module.

[0047] The basic formula for calculating the eye aspect ratio (EAR) is as follows: , In the formula, Indicates the aspect ratio of the eye; Indicates the key points at the inner corner of the left eye. Indicates the key points at the outer corner of the right eye. Indicates key points of the left upper eyelid. Indicates key points of the lower eyelid of the left eye. Indicates key points of the upper eyelid of the right eye. Indicates key points on the lower eyelid of the right eye; This represents the Euclidean distance between two key points.

[0048] The driver-specific low EAR threshold is calculated as follows: , In the formula, Indicates the first EAR mean within a window; Indicates the first Valid within a window Frame rate; Indicates the first The first window 1 valid frame value.

[0049] The minimum EAR for the window is as follows: , In the formula, Indicates the first Minimum EAR value within a window; and The meaning is the same as above.

[0050] The standard deviation of the window EAR is as follows: , In the formula, Indicates the first EAR standard deviation within a window; , and The meaning is the same as above.

[0051] The blinking frequency of the window is as follows: , In the formula, Indicates the first Blinking frequency within a window; Indicates the first The duration of each window; Indicates the first The first window Is the frame a blink frame? Indicates the first Number of valid frames within a window; This indicates an indicator function that takes the value 1 if the condition is true, and 0 otherwise.

[0052] The average blink interval between adjacent windows is as follows: , In the formula, Indicates the first Average blink interval within a window; Indicates the first Number of blinks within a window; Indicates the first The first window The starting frame number of a blink; This indicates the eye-tracking sampling frame rate.

[0053] Driver-specific low EAR threshold: , In the formula, Indicates the first Low EAR threshold for famous drivers; Represents the 20th quantile function; Indicates the first Driver Baseline segment The EAR value of the frame.

[0054] The formula for the number of consecutive segments at the low EAR of the window is as follows: In the formula, Indicates the first The number of consecutive low EAR segments within a window; the first term is used to count the segments that are already in the low EAR state in the first frame of the window, and the summation term is used to count the starting point of segments that enter the low EAR state from the non-low EAR state within the window; , , and The meaning is the same as above.

[0055] In one embodiment, when deploying online, the system first receives at least 60 seconds of Drive baseline data from the target driver, and then calculates the normalized parameters.

[0056] The formula for normalizing eye movement behavior features is as follows: , In the formula, Indicates the first The first eye movement feature in the first The normalized values ​​for each window; Represents the original value; express Baseline mean; Indicates the baseline standard deviation of Drive; This represents the smallest positive number that should not be divided by zero.

[0057] The normalization formula for electrophysiological characteristics of the heart is as follows: , In the formula, Indicates the first The first electrophysiological characteristic in the first The normalized values ​​for each window; Represents the original value; express Baseline median; express Baseline interquartile range; The meaning is the same as above.

[0058] The system only uses data from the current time and data that has arrived before the current time to generate windows, rolling sequences, normalized statistics, and sample-level decisions; it does not use future data.

[0059] In one embodiment, such as Figure 4 As shown, the shortest inference time in this embodiment is calculated using the following formula: , In the formula, This indicates the minimum online data duration required to trigger an online inference; Indicates the length of a single feature window; Indicates the step size between adjacent windows; This indicates the number of windows contained in the sequence. In a preferred embodiment, =10s =5s、 =20, therefore =105s.

[0060] The binary classification temporal model for categorized cognitive load includes an eye-tracking flow encoder and a physiological flow encoder. Each flow encoder may include a one-dimensional convolutional layer, a ReLU activation layer, an LSTM layer, and a temporal attention layer.

[0061] In this embodiment, the complete computational chain for time attention is as follows: , In the formula, Indicates the first Attention score for each time step; This indicates that the recurrent neural network is in the first... The hidden state output at each time step; , and This represents the trainable attention parameters.

[0062] In this embodiment, the normalized weights are as follows: , In the formula, Indicates the first Attention weights for each time step; Indicates the first Attention score for each time step; Indicates the first Attention score for each time step; Indicates the sequence length.

[0063] In this embodiment, the single-mode timing is as follows: , In the formula, Modal timing representation; Indicates the first Attention weights for each time step; Indicates the first The hidden state at each time step; Indicates the sequence length.

[0064] In this embodiment, the multimodal fusion and classification output are as follows: The weighted splicing formula is as follows: , In the formula, This indicates a fusion representation; Indicates feature concatenation operation; Represents eye-tracking modal weights; Represents physiological modality weights; Representation of eye-tracking flow temporal sequence; This represents the timing representation of physiological flow.

[0065] , In the formula, Represents a category probability vector; This represents the normalized exponential function; and Indicates the classification layer parameters; This indicates a fusion representation.

[0066] The class probabilities output by the softmax classification layer are used to determine the sample as follows: , In the formula, Indicates the sample-level predicted category; This represents the category selection operation that maximizes the category probability. Indicates the candidate category; Indicates a low cognitive load category; Indicates a high cognitive load category; Indicate category The corresponding predicted probability.

[0067] In one embodiment, such as Figure 6 and 7 As shown, the basic calculation formula for the prediction probability of high cognitive load in this embodiment is as follows: , In the formula, This indicates the probability of predicting high cognitive load within a preset output interval; This represents the number of valid samples identified as having high cognitive load within the preset output interval. This represents the total number of samples that complete model inference and generate valid judgments within the preset output interval.

[0068] , In the formula, This represents the sample-level high load probability obtained by fusing response family probabilities; Indicates the number of response families; This indicates that the target driver belongs to the first category. The probability of a family of responses; Indicates the first Each response family corresponds to a clustering model for the input sequence. High load probability in the output; This represents the normalized rolling time series feature sequence.

[0069] The system will The system compares the data with preset thresholds, continuous output interval thresholds, or moving average thresholds. When the trigger condition is met, the system executes in-vehicle display alerts, voice alerts, tactile alerts, interactive noise reduction, adjustments to assisted driving control strategies, or recording of driving risks. After execution, the system enters the next detection cycle, thus forming a closed loop from method reasoning and application actions to data collection in the next cycle.

[0070] In one embodiment, the apparatus of the present invention may include a questionnaire feature generation module, a questionnaire typing module, a model routing module, an online feature processing module, a temporal reasoning module, a probability aggregation module, and a control command generation module. Each module may be implemented by software programs, hardware circuits, or a combination of software and hardware.

[0071] The system of this invention may include a feature acquisition terminal, an in-vehicle computing terminal, and an application execution terminal. The feature acquisition terminal may include an in-vehicle camera, an eye-tracking algorithm module, a steering wheel ECG sensor, a seat ECG sensor, a wearable device gateway, or a smart cockpit SDK. The in-vehicle computing terminal executes the above method and generates a high cognitive load prediction probability. The application execution terminal executes display reminders, voice reminders, haptic reminders, interactive noise reduction, assisted driving control strategy adjustments, or driving risk recording based on the predicted probability.

[0072] The above description is merely illustrative of the embodiments of the present invention and is not intended to limit the present invention. For those skilled in the art, any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for predicting driver cognitive load, characterized in that, include: S1. Obtain the driver's questionnaire data and generate a questionnaire feature vector based on the questionnaire data; S2. Input the questionnaire feature vector into a pre-trained questionnaire classification model and output the cognitive load physiological response family corresponding to the driver; the cognitive load physiological response family is obtained based on the changing trend of the driver's blink frequency relative to his own stable driving baseline under different cognitive load levels; S3. Match the corresponding clustered cognitive load time series model based on the cognitive load physiological response family; S4. Continuously collect the driver's smooth driving baseline feature stream and multimodal online feature stream, calculate the subject's intra-subject normalized statistics based on the smooth driving baseline feature stream, and perform personalized normalization preprocessing on the multimodal online feature stream based on the smooth driving baseline feature stream to obtain the preprocessed online feature stream. S5. Construct a fixed-length rolling time series sequence for the online feature stream after normalization preprocessing, input the rolling time series sequence into the matched clustered cognitive load time series model, and output the single-sample cognitive load judgment result. S6. Within a preset output interval, count the judgment results of all valid single samples, count the ratio of the number of samples judged as high cognitive load to the total number of valid samples, and calculate the stable high cognitive load prediction probability; generate vehicle control commands based on the stable high cognitive load prediction probability.

2. The driver cognitive load prediction method according to claim 1, characterized in that, The questionnaire classification model was trained by collecting questionnaire data from historical drivers and eye-tracking data under multi-gradient cognitive load tasks. The change in blink frequency of each historical driver relative to their driving baseline under different cognitive load levels is calculated; based on the trend of the change with the increase of cognitive load level, multiple physiological response types are distinguished and merged into two cognitive load physiological response families; the questionnaire classification model is trained with the questionnaire feature vector of the historical drivers as input and the physiological response families as supervision labels.

3. The driver cognitive load prediction method according to claim 1, characterized in that, The multimodal online feature stream includes eye-movement behavior features and electrophysiological features; The eye movement behavior features include the eye aspect ratio time series calculated based on key eye points and blink frequency; the electrophysiological features include mean heart rate and heart rate variability index. Simultaneously align eye movement behavior features and electrophysiological features from the same time period, and fuse them to form a multi-dimensional combined feature window.

4. The driver cognitive load prediction method according to claim 1, characterized in that, The preprocessing includes: in step S5, strictly following causal temporal constraints, window alignment and rolling temporal sequence construction are completed using historical arrived temporal data; the clustered cognitive load temporal model includes parallel eye-tracking flow encoders and physiological flow encoders, each encoder containing a one-dimensional convolutional layer, activation layer, recurrent neural network layer and temporal attention layer.

5. The driver cognitive load prediction method according to claim 1, characterized in that, The clustered cognitive load time series model is trained by partitioning the driver dataset. Historical drivers are stratified according to their cognitive load physiological response families. All time series samples of the same historical driver are assigned to one of the three sets: training set, validation set, or test set. Drivers from different datasets do not overlap at all.

6. The driver cognitive load prediction method according to claim 1, characterized in that, Based on the single-sample cognitive load determination result output in step S5, the frequency of high cognitive load determination within the preset output interval is statistically analyzed, and the probability of stable high cognitive load prediction is calculated based on the proportion of effective samples within the interval. The predicted probability of stable high cognitive load is compared with a preset threshold. When the probability value is greater than or equal to the threshold or the judgment logic corresponding to the continuous output interval moving average threshold is met, an in-vehicle intervention action is executed. The in-vehicle intervention action includes at least one of reducing the information push frequency, outputting rest reminders, and adjusting the assisted driving monitoring level.

7. A driver cognitive load prediction system, characterized in that, include: The questionnaire feature generation module is used to obtain the driver questionnaire selection data, and generate a standardized questionnaire feature vector after quantification. The questionnaire typing module is used to acquire the driver's questionnaire data and generate questionnaire feature vectors. The questionnaire feature vectors are then input into a pre-trained questionnaire typing model, which outputs a family of physiological responses to cognitive load. The model routing module is used to match a specific and adapted clustered cognitive load time series model based on the cognitive load physiological response family. The online feature processing module is used to receive the driver's multimodal online feature stream, collect the smooth driving baseline feature stream, and complete the subject-specific personalized normalization preprocessing of the online feature stream based on the baseline feature stream; The temporal reasoning module is used to construct a rolling temporal sequence using only historical temporal data, input the preprocessed multimodal online feature stream into the matched clustered cognitive load temporal model, and output the single-sample cognitive load judgment result. The probability aggregation module is used to aggregate all valid single-sample judgment results within a preset output interval and calculate the stable high cognitive load prediction probability. The control command generation module is used to generate vehicle control commands based on the predicted probability of high cognitive load.

8. The driver cognitive load prediction system according to claim 7, characterized in that, The system also includes a feature acquisition terminal, an application execution terminal, and a cloud storage module; the feature acquisition terminal includes at least one of an in-vehicle camera, an electrocardiogram sensor, and a wearable device, used to collect the driver's eye movement behavior characteristics, electrocardiographic characteristics, and smooth driving baseline data; The application execution terminal receives vehicle control commands generated based on interval aggregation probability, executes corresponding vehicle intervention actions in layers, and synchronously uploads driving risk events to the cloud storage module.

9. A vehicle-mounted electronic device, characterized in that, It includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the driver cognitive load prediction method as described in any one of claims 1-6.

10. A computer-readable storage medium, characterized in that, The storage medium stores a program or instructions that, when executed by a processor, implement the steps of the driver cognitive load prediction method as described in any one of claims 1-6.

Images