A fatigue adaptive monitoring method and system based on a psychological expected value control theory

Abstract

This invention relates to the field of intelligent transportation technology, specifically a fatigue adaptive monitoring method and system based on the psychological expectation-value control theory. Addressing the pain points of existing technologies—insufficient cross-subject generalization ability, high false negative rates in early and mid-stage fatigue of highly compensated drivers, and high computational overhead making in-vehicle edge deployment difficult—this invention pre-identifies drivers' psychological traits, classifying individuals according to cognitive characteristics; simultaneously collects non-invasive multimodal signals and performs lightweight temporal preprocessing; constructs a multimodal feature independent encoding and deep fusion network; combines a trait-aware hard routing mechanism, employing a two-stage training method of global pre-training plus trait-specific fine-tuning to ensure model accuracy and generalization; and finally outputs a tiered, graded intervention strategy based on fatigue level, achieving high-precision, low-computing-power real-time fatigue monitoring. This system can be stably deployed on in-vehicle edge devices and is suitable for use in mass-produced passenger vehicle cabins.

Description

Technical Field

[0001] This invention relates to the field of intelligent transportation technology, and in particular to a fatigue adaptation monitoring method and system based on the psychological expectation value control theory. Background Technology

[0002] With the rapid popularization of intelligent connected vehicles and advanced driver assistance systems, real-time monitoring of driver fatigue has become a core technology for ensuring road safety. Current driver fatigue detection is mainly developing in two directions: multimodal fusion and cross-subject generalization. However, existing technical solutions generally suffer from the industry pain point of failing to balance detection accuracy, individual adaptability, and feasibility of in-vehicle deployment.

[0003] One existing technology is a fatigue driving detection system based on multimodal global fusion. This system extracts image features using ResNet18 and temporal signal features using LSTM, optimizes feature weights by combining channel and spatial attention, and then performs multimodal fusion via Hadamard product before feeding the results into a globally unified fully connected classifier. While this approach achieves a certain level of detection accuracy on a single dataset, it suffers from two major drawbacks: First, it has poor cross-subject generalization ability. It uses a global single decision model, assuming that the fatigue evolution of all drivers follows the same data distribution, ignoring the essential differences in individual fatigue mechanisms and signal manifestations in cognitive psychology. This easily confuses fatigue features with the inherent baseline characteristics of drivers, resulting in severe under-detection of early and mid-stage fatigue in resilient drivers with high compensatory abilities, and extremely low recall rate for moderate fatigue. Second, it is difficult to deploy at the vehicle edge. The system introduces complex attention calculations at each level, and forward inference requires full activation of all network branches and weight matrices. The number of model parameters and floating-point operations increases exponentially, and the latency of a single inference operation far exceeds the computing power and low energy consumption requirements of vehicle edge computing chips.

[0004] The second existing technology is a deep learning method for cross-subject fatigue detection based on multi-scale parallel convolution of EEG channels. This approach uses multi-scale parallel convolution to forcibly extract EEG signal features to mitigate individual differences. However, its shortcomings are also prominent: First, it has extremely poor vehicle-mountability, as the solution heavily relies on multi-channel EEG electrode caps, which is a highly invasive acquisition method and cannot be practically used in mass-produced passenger vehicle cabins; second, it does not eliminate individual biases from the cognitive root, but only filters individual differences through mathematical convolution, completely ignoring the fatigue active regulation mechanism revealed by the expected value control theory, and still cannot solve the problem of the fatigue characteristics of highly compensated drivers being masked; finally, it suffers from severe computational redundancy, as multi-scale parallel convolution requires full activation of multiple branches, generating a large amount of redundant computation, which is difficult to meet the high-frequency real-time processing requirements of automotive-grade applications.

[0005] In summary, existing fatigue detection technologies generally suffer from a series of problems, such as insufficient generalization ability across subjects, high false negative rate of early and mid-term fatigue in highly compensated drivers, huge computational overhead of high-precision models that cannot be deployed in lightweight vehicles, and excessive invasiveness of some solutions that makes them difficult to mass-produce. They cannot simultaneously meet the comprehensive requirements of accurate identification, individual adaptation, low-power real-time inference, and vehicle safety deployment. Summary of the Invention

[0006] The purpose of this invention is to overcome the problems of the prior art and provide a fatigue adaptability monitoring method and system based on the psychological expectation value control theory. This method addresses the problems of poor generalization ability across subjects and high false negative rate for early and mid-term fatigue in drivers with high compensatory ability in existing fatigue detection methods. It also overcomes the shortcomings of high-precision models, such as high computational cost and high inference latency, which make them difficult to deploy in a lightweight and real-time manner at the vehicle edge, thereby improving detection adaptability and feasibility for vehicle deployment.

[0007] The above objectives are achieved through the following technical solutions: A fatigue adaptation monitoring method based on the psychological expectancy-value control theory includes the following steps: The test driver acquires multimodal physiological signals, where multimodal refers to data from two different modalities: physiological signals and behavioral signals. Specifically, the data includes heart rate (HR) physiological signals, as well as behavioral signals such as facial images, the Big Five personality disorder questionnaire, and the somnolence scale, and combines them with the trait labels of the target driver determined based on the expectancy-value control theory. The multimodal physiological and behavioral signals are input into a pre-trained multimodal feature encoding network, and multimodal fusion features are extracted and fused. The multimodal fusion features are input into an expert hybrid network consisting of four expert classifiers of different categories. Based on the trait labels, the dedicated expert classifiers corresponding to the trait labels are activated through a trait-aware hard routing mechanism. The activated dedicated expert classifiers output the current driving fatigue level.

[0008] As a further optimization of this method, the determination of driver trait labels based on expected value control theory specifically includes: Obtain the Big Five personality test data and the sleepiness scale (SSS) attribute data of the driver, and perform index scoring, reverse processing and Z-score standardization on the above two types of data; Based on the expected value control theory, the data after Z-score standardization is mapped into a two-dimensional feature space coordinate system of revenue and cost. The two-dimensional feature space coordinates are clustered using a potential profile analysis algorithm. After the driver's standardized data is clustered, it is classified into four trait groups and the corresponding trait labels are output. The four trait groups include: high cost & low profit, high cost & high profit, low cost & low profit, and low cost & high profit.

[0009] As a further optimization of this method, the psychological assessment data includes the conscientiousness and neuroticism dimensions of the simplified version of the Chinese Big Five Personality Questionnaire and the Stanford Somnolence Scale score; the driving attribute data includes driving experience, and the driving experience is reverse-scored.

[0010] As a further optimization of this method, the multimodal physiological and behavioral signals include video image stream signals acquired by vehicle-mounted cameras, eye aspect ratio time series calculated based on eye key points, and heart rate time series acquired by ECG or heart rate sensors; the multimodal feature coding network includes a face (Face) visual stream coding branch, a face (Face) temporal coding branch, and a physiological (HR) temporal coding branch. Each branch is independently coded and then the features are spliced ​​and fused to obtain the multimodal fused features.

[0011] As a further optimization of this method, the trait-aware hard routing mechanism is as follows: a one-hot routing vector is generated based on the trait label, and only one of the four classification experts in the corresponding dimension is activated. The computation channels of the other expert classifiers are completely truncated, and only the forward computation of a single expert branch is performed during inference.

[0012] A fatigue adaptation monitoring system based on the psychological expectancy-value control theory includes: The driver trait pre-identification module is used to acquire drivers' psychological assessment and driving attribute data, construct two-dimensional features based on expected value control theory, and output trait labels through latent profile analysis. The multimodal data synchronous acquisition module is used to synchronously acquire video visual streams and ECG / heart rate signals and complete global timestamp alignment. The data preprocessing module is used to perform time downsampling, face detection, eye aspect ratio calculation, and heart rate calculation on the acquired signals to generate standardized multimodal temporal tensors. The fatigue adaptation monitoring model module based on the psychological expectation value control theory has a built-in multimodal feature encoding network, a trait-aware hard routing controller, and multiple trait-specific expert classifiers, which are used to activate the corresponding expert classification based on the trait label and output the fatigue level. The output module is used to trigger corresponding tiered intervention strategies based on fatigue levels.

[0013] As a further optimization of this system, the multimodal data synchronization acquisition module includes a hardware sensor layer, a raw signal acquisition layer, a time synchronization layer, and an output terminal. It timestamps the video stream and ECG signal respectively and completes absolute time alignment based on the system master clock.

[0014] As a further optimization of this system, the data preprocessing module uses a fixed non-overlapping sliding window of 20 seconds, adopts stage-aware dynamic frame sampling for the visual stream, uses Otsu's method for adaptive threshold calibration of the eye aspect ratio, and performs filtering and noise reduction and instantaneous heart rate calculation on the electrocardiogram signal.

[0015] As a further optimization of this system, the fatigue adaptation monitoring model module based on the psychological expectation value control theory adopts a two-stage training method of global pre-training plus trait-specific fine-tuning: in the global pre-training stage, the general feature encoder and global classifier are trained using full-trait samples; in the trait-specific fine-tuning stage, the encoder weights are frozen, and each expert classifier is fine-tuned only using corresponding trait samples.

[0016] This invention provides a fatigue adaptation monitoring method and system based on the psychological expectancy-value control theory. By introducing this theory, it achieves pre-discrimination of driver traits, effectively improving cross-subject generalization ability, significantly reducing the early-to-mid-stage fatigue false negative rate in drivers with high compensatory ability, and greatly improving overall detection accuracy and moderate fatigue recall rate. It employs a trait-aware hard routing mechanism to ensure that inference activates only a single expert branch, keeping computational overhead and model parameter count within the vehicle edge deployment range, guaranteeing low-latency real-time operation. Simultaneously, it utilizes non-intrusive multimodal data acquisition adapted to mass-produced cockpits, and its tiered warning strategy balances driving safety and passenger experience, possessing strong engineering feasibility and broad scenario expansion capabilities. Compared with existing technologies, the specific beneficial effects are as follows: 1. Significantly improved detection accuracy: Overall accuracy rate of 94.15%, an improvement of 12.93 percentage points from the baseline; moderate fatigue recall rate of 94.68%, an improvement of 28.51 percentage points; and accuracy rate for resilient drivers of 83.78%, an improvement of 5.40 percentage points, fundamentally solving the problem of missed detection of highly compensated drivers.

[0017] 2. Excellent edge deployment capability: The hard router only activates a single branch, the number of parameters increases by only 0.02% compared to the baseline, the 20-second window inference latency is only 1.23ms, and the computing power overhead is basically the same as the single-branch model, which can be directly deployed on the vehicle NPU.

[0018] 3. Highly adaptable to engineering applications: It adopts non-invasive data acquisition using cameras and ECG / heart rate, making it compatible with mass-produced cabins; adaptive thresholds eliminate individual differences; and tiered intervention avoids excessive disturbance, balancing safety and driving experience.

[0019] 4. High generalization and scalability: Based on EVC theory, it models individual cognitive differences and supports extension to scenarios such as emotion recognition, attention monitoring, and workload assessment. It also supports equivalent replacement of various sensors, algorithms, and clustering methods. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the overall architecture of a fatigue adaptability monitoring system based on the psychological expectation-value control theory described in this invention. Figure 2 This is an internal flowchart of the driver trait pre-identification module in a fatigue adaptation monitoring system based on the psychological expectation-value control theory described in this invention. Figure 3 This is an internal architecture diagram of the multimodal data synchronous acquisition module in a fatigue adaptation monitoring system based on the psychological expectation-value control theory described in this invention. Figure 4 This is an internal flowchart of the data preprocessing module in a fatigue adaptability monitoring system based on the psychological expectation-value control theory described in this invention. Figure 5 This is an internal flowchart of the fatigue adaptation monitoring model module based on the psychological expectation-value control theory in the fatigue adaptation monitoring system described in this invention. Figure 6 This is an internal flowchart of the output module in a fatigue adaptability monitoring system based on the psychological expectation-value control theory described in this invention. Figure 7 This is a schematic diagram of the two-stage training process of the model in the fatigue adaptability monitoring method based on the psychological expectation value control theory described in this invention.

[0021] Illustration markings: 101-Driver trait pre-identification module, 102-Multimodal data synchronous acquisition module, 103-Data preprocessing module, 104-Fatigue adaptability monitoring model module based on psychological expectation value control theory, 105-Output module. Detailed Implementation

[0022] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. The described embodiments are merely some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] This solution provides a fatigue adaptation monitoring method and system based on the psychological expectancy-value control theory. Utilizing a deep learning model, it accurately detects the fatigue state of drivers with different traits (resilient, diligent, passive, and vulnerable) in simulated driving tasks. The specific implementation process of the proposed system and method is explained in detail below. Those skilled in the art should understand that the application scenarios described are merely illustrative, and the technical solution of this invention is also applicable to other tasks that require distinguishing different cognitive or physiological states from multimodal data containing significant individual differences, such as emotion recognition, attention monitoring, and workload assessment. The signals used are not limited to video, eye tracking, and heart rate; they can be extended to other modalities such as electroencephalography (EEG), surface electroreception (GSR), steering wheel angle, and vehicle dynamics signals.

[0024] like Figure 1 As shown, this solution provides a fatigue adaptation monitoring system based on the psychological expectancy-value control theory. The system mainly includes five core modules: a driver trait pre-identification module 101, a multimodal data synchronous acquisition module 102, a data preprocessing module 103, a fatigue adaptation monitoring model module based on the psychological expectancy-value control theory 104, and an output module 105. Details are as follows: The driver trait pre-identification module 101 is responsible for reading the driver's psychological assessment and driving attribute data from the vehicle database or cloud compliance database, performing benefit-cost two-dimensional feature modeling and potential profile analysis, and outputting individual trait category labels (such as resilient, hardworking, etc.).

[0025] The multimodal data synchronous acquisition module 102 is responsible for synchronously acquiring the driver's original video stream and electrocardiogram signal through the vehicle's pre-installed hardware, and for timestamp marking and absolute time alignment through a globally unified clock.

[0026] The data preprocessing module 103 is connected after the acquisition module and is responsible for preprocessing the raw data to generate standardized multimodal time series tensors (such as visual image streams, eye aspect ratio time series and heart rate time series signals within a 20-second time window).

[0027] The fatigue adaptability monitoring model module 104, based on the psychological expectation value control theory, is the core hub of the system. It contains a multimodal feature independent encoding network and an expert routing decision mechanism. After extracting and fusing multimodal features from the input time-series signal tensor, the module uses a hard routing mechanism to accurately activate the corresponding trait-specific expert classifier based on the trait labels output by the driver trait pre-identification module 101, and outputs the current fatigue level judgment result.

[0028] The output module 105 then triggers the corresponding tiered intervention strategy based on the determined fatigue level.

[0029] like Figure 2 As shown, in this embodiment, the driver trait pre-identification module 101 aims to transform objective psychological data and driving attribute data into trait labels that can be used for downstream model routing based on the Expected Value Control (EVC) theory. Its specific execution steps are as follows: First, the module is triggered upon vehicle power-on and driver identification, receiving a multi-dimensional set of observation variables from the driver being tested. The objective psychological data referred to in this invention refers to the quantitative psychological trait scores obtained by the driver through standardized psychological scale assessments; the multi-dimensional set of observation variables refers to the set of input variables composed of both objective psychological data and driving attribute data, specifically including: the conscientiousness dimension score and neuroticism dimension score from the simplified version of the Chinese Big Five Personality Questionnaire (CBF-PI-15), the baseline score of the Stanford Somnolence Scale (SSS), and the number of years of valid driving experience.

[0030] Indicator Scoring and Reverse Processing: The scale data were positively scored. The conscientiousness dimension included 3 items (items 6, 8, and 15, reflecting self-discipline and intrinsic identification with safe driving), and the neuroticism dimension included 3 items (items 7, 11, and 12, reflecting emotional stability and tendency to worry), both using a 1-5 level Likert scale. The SSS scale used one 7-level positively scored item. Simultaneously, the valid driving years were reverse-scored using the following formula: , The higher the value, the lower the degree of automation of driving skills and the higher the basic cognitive cost of maintaining alertness.

[0031] Dimensional and Scale Alignment: To eliminate dimensional differences among various indicators, the Z-score method was used to standardize all processed observation indicators. The calculation formula is as follows: , in Let j be the standard score of the i-th driver for the j-th indicator. This is the original score. The mean of the sample. This represents the within-sample standard deviation. After standardization, all indicators have a mean of 0 and a standard deviation of 1, allowing for direct cross-dimensional calculation. This outputs the standardized set of observed variables after scaling.

[0032] Cognitive Theory Spatial Mapping: This is the core step of this module. Based on Expected Value Control (EVC) theory, a two-dimensional feature model of benefits and costs is constructed. Specifically, the standardized conscientiousness feature is mapped to "expected benefit features" (i.e., total score of expected benefit dimension = Z_conscientiousness); the mean of neuroticism, SSS score, and reverse driving experience is mapped to "subjective cost features," as shown in the following formula: This step transforms the original four-dimensional physiological and psychological data into a two-dimensional coordinate system that conforms to cognitive psychology.

[0033] Latent Profile Analysis Clustering: Based on the two-dimensional feature space, a Gaussian Mixture Model (GMM) is used to perform Latent Profile Analysis (LPA). Specifically, the scikit-learn library in Python is used, with the number of latent classes K=4. The model parameters are iteratively updated using the Expectation-Maximization (EM) algorithm, with a maximum of 1000 iterations and a convergence threshold of 1e-3. The full covariance matrix (covariance_type='full') is used for calculation until the log-likelihood converges.

[0034] Hard classification and label assignment: Based on the position of each cluster center in the (total revenue score, total cost score) coordinate system, calculate the posterior probability of each driver belonging to each potential category, and hard assign the individual to the most suitable category based on the maximum posterior probability. Specific determination rules are as follows: Category A (Resilient): High returns (total return score > 0) + Low costs (total cost score < 0), trait label z = A; Category B (Effort-Driven): High Returns (Total Returns Score > 0) + High Costs (Total Costs Score > 0), Trait Tag z = B; Category C (Passive): Low returns (total return score < 0) + Low costs (total cost score < 0), trait label z = C; Type D (Vulnerable): Low returns (total return score < 0) + High costs (total cost score > 0), trait label z = D.

[0035] Finally, the module outputs a unique individual trait category label z ∈ {A,B,C,D} and sends it to the downstream 102 acquisition module and 104 expert routing network.

[0036] like Figure 3 As shown, in this embodiment, the multimodal data synchronization acquisition module 102 adopts a top-down hierarchical processing architecture, aiming to solve the technical pain points of inconsistent sampling frequencies and physical delays in heterogeneous hardware, thereby providing a raw data stream with a strict time alignment benchmark for downstream feature preprocessing. Its specific execution flow is as follows: The first layer is the hardware sensor layer: This layer is the data source for the entire system and is equipped with two parallel physical sensors. One side is an in-vehicle RGB camera fixed inside the vehicle to capture the driver's facial visual state; the other side is an electrocardiogram sensor (such as electrocardiogram electrodes integrated into the steering wheel or smart wearable devices) to capture the driver's physiological electrical signals.

[0037] The second layer is the raw signal acquisition layer: it connects to the hardware sensor and is equipped with two independent continuous acquisition channels. It performs "continuous acquisition of raw video stream" and "continuous acquisition of raw ECG signal" respectively to obtain continuous physical signals with different native sampling rates (e.g., 30fps for video and high frequency for ECG).

[0038] The third layer is the time synchronization layer: this is the core technology of this acquisition module. To eliminate clock drift between the two independent acquisition channels, the module incorporates a "globally unified time and timestamp alignment" mechanism. Specifically, the system uses the underlying "system master clock" as the absolute time reference, performing "video frame timestamp marking" on the incoming visual stream and "ECG timestamp marking" on the physiological stream. Subsequently, through the underlying signal alignment mechanism, the two heterogeneous signals with absolute timestamps are merged into a unified timeline pipeline.

[0039] The fourth layer is the output end: After strict time synchronization, this module outputs "raw video stream data" and "raw electrocardiogram signal" with aligned timestamps to the downstream data preprocessing module 103 for subsequent time downsampling and time segment slicing operations.

[0040] As an alternative, the aforementioned physiological signal acquisition can be replaced by PPG from smart bracelets / watches or non-contact millimeter-wave radar to acquire heart rate / respiratory rate; visual acquisition can retain the vehicle-mounted RGB camera solution.

[0041] like Figure 4 As shown, in this embodiment, the data preprocessing module 103 is connected after the multimodal data synchronous acquisition module 102. It is responsible for transforming the original multimodal data stream with absolute timestamp alignment into a standardized multimodal temporal tensor required by the downstream deep learning network through spatial dimensionality reduction, temporal resampling, and feature engineering. The system uses a fixed non-overlapping sliding window of T=20 seconds and executes the following three fine-tuning pipelines in parallel: Visual flow preprocessing pipeline: First, the input raw video clip is downsampled to 4Hz to initially obtain a continuous sequence containing 80 frames. To effectively control the parameter quantity and computational cost of subsequent spatiotemporal networks (such as 3D-CNN or LSTM) and make them perfectly adaptable to in-vehicle edge deployment, this invention introduces a stage-aware dynamic frame sampling strategy. Specifically, the system divides the 80-frame sequence into 10 consecutive time segments (i.e., each segment contains 8 frames) in the time dimension and performs differentiated sampling according to the system's current operating stage: During the model training phase, a segmented random sampling strategy is adopted, that is, one frame is randomly sampled within each segment using a random number generator (such as random.randint). This mechanism, as a built-in temporal data augmentation method, greatly increases the spatiotemporal diversity of the input data and improves the model's robustness to action temporal shifts.

[0042] During the model testing and online inference phases, a deterministic center sampling strategy is adopted, which involves selecting the most central frame on the timeline for each segment. This mechanism effectively filters out random disturbances, ensuring the absolute determinism of a single forward inference and the repeatability of the detection results.

[0043] Subsequently, the system sequentially invokes a lightweight YOLOv8n model, such as YOLOv8n, to perform face bounding box detection, face alignment based on facial landmarks (correcting head pose deviation to a frontal angle), and precise cropping of facial regions of interest (ROIs) on the 10 extracted keyframes. The cropped images are uniformly scaled to 160×160 pixels and then subjected to single-channel grayscale conversion and pixel value normalization. The final output is a normalized visual feature tensor with dimensions of 10 × 1 × 160 × 160. The standardized visual feature tensor referred to in this invention refers to visual input data that has undergone face detection, alignment, ROI cropping, size scaling, grayscale conversion, and pixel value normalization, and can be directly used as the input of the visual flow spatiotemporal coding branch in a multimodal feature coding network.

[0044] An eye-tracking behavior preprocessing pipeline based on adaptive thresholds: This pipeline aims to extract high-precision eyelid closure features using lightweight engineering algorithms. Unlike traditional approaches that directly process full-frame raw video streams, this pipeline uses precisely cropped and aligned facial Regions of Interest (ROI) images from the visual stream preprocessing step as the direct input source. This preprocessing design based on locally cropped images not only completely eliminates interference from complex in-vehicle lighting and cabin backgrounds but also significantly reduces the computational cost of subsequent feature point detection and dramatically improves spatial accuracy.

[0045] Specifically, the system converts the acquired facial ROI image sequence using OpenCV and then inputs it into the MediaPipe Face Mesh AI engine. This engine infers the 3D geometric structure in real time within the local facial region, accurately locating the key point coordinates of each eye contour: namely, the horizontal axis endpoints P1 and P4 formed by the inner and outer corners of the left and right eyes; the vertical axis endpoints P2 and P6 of the upper and lower eyelids near the inner side; and the vertical axis endpoints P3 and P5 of the upper and lower eyelids near the outer side.

[0046] Next, the system calls the NumPy linear algebra library to calculate the Euclidean distance between the above coordinate points, and strictly follows the classical algorithm logic to calculate the eye aspect ratio (EAR) of a single frame, as shown in the following formula: , The numerator of this formula represents the vertical opening of the eye, while the denominator represents the horizontal length. When the driver's eyes are open normally, this value is at a relatively high baseline (approximately 0.25 to 0.35); when blinking or closing the eyes due to fatigue occurs, the vertical distance of the numerator rapidly approaches 0, causing a sharp downward jump in the EAR value.

[0047] To eliminate fixed threshold errors caused by individual driver physiological structures (such as eye size and palpebral fissure width), the system introduces Otsu's Method to perform automatic threshold calibration. By analyzing the EAR data distribution characteristics of the subject within the initial time window, the script adaptively resolves the optimal segmentation threshold between the "open eye state" and the "closed eye state" for that specific driver, achieving personalized dynamic adjustment of discrimination sensitivity.

[0048] Finally, the system rigorously resamples the calculated discrete EAR sequence to 4Hz using a time interpolation algorithm. This generates a one-dimensional eye-tracking feature sequence vector containing 80 consecutive data points within a fixed 20-second window. .

[0049] Physiological signal preprocessing pipeline: The system performs filtering and noise reduction (such as eliminating baseline drift and high-frequency motion artifacts) on the raw electrocardiogram (ECG) signal input from the multimodal data synchronous acquisition module 102, and performs instantaneous heart rate calculation. The feature extraction frequency is set to 1Hz, and 20 heart rate data points are precisely extracted within a 20-second window to generate a one-dimensional physiological feature sequence vector. .

[0050] Finally, this module packages and concatenates the three heterogeneous tensors into standardized multimodal time-series data. It is directly used as the input to the downstream fatigue adaptation monitoring model module 104 based on the psychological expectation value control theory.

[0051] As alternatives, the face and keypoint detection mentioned above can be replaced with RetinaFace, OpenPose, or Dlib libraries; eye features can be replaced with PERCLOS; and visual encoding networks can be replaced with structures such as 3D-CNN, ViT, and TimeSformer.

[0052] like Figure 5 and 6As shown, in this embodiment, the fatigue adaptation monitoring model module 104 based on the psychological expectation value control theory serves as the "decision brain" of the system, receiving standardized multimodal temporal tensors from the data preprocessing module 103. and driver trait category labels from driver trait pre-recognition module 101 It outputs the fatigue level determination result for the current time window through feature-independent encoding, cross-modal fusion, and feature-aware hard routing mechanism. The specific implementation steps are as follows: Independent multimodal feature encoding: To fully capture the spatiotemporal dynamics of heterogeneous signals, the system invokes a pre-trained multimodal feature encoding network, performing feature extraction through three independent deep learning branches: Visual stream spatiotemporal coding branch (a dedicated coding module responsible for processing facial visual signals): receives a sequence of grayscale images with dimensions of 10 × 1 × 160 × 160. The architecture employs a combined "3-layer 2D-CNN + single-layer LSTM". All three convolutional layers use 3 × 3 convolutional kernels, coupled with max pooling layers and the ReLU activation function, to extract static spatial features of the face frame by frame. The spatial feature sequence is then input into a single-layer LSTM network with a hidden layer dimension of 256 to model the temporal dynamic evolution of 10 keyframes. Finally, after temporal mean pooling, a 256-dimensional visual feature vector is output. .

[0053] One-dimensional temporal coding branch for eye movement behavior (a dedicated coding module responsible for processing eye movement physiological signals): receives EAR sequences with a dimension of 80. A two-layer 1D-CNN structure is adopted: the first layer has a kernel size of 5, 32 output channels, and ReLU activation; the second layer has a kernel size of 5, 64 output channels, and ReLU activation. After global max pooling, a 64-dimensional eye-tracking feature vector is output. .

[0054] One-dimensional temporal coding branch for physiological characteristics (a dedicated coding module responsible for processing heart rate physiological signals): receives heart rate sequences with a dimension of 20. A two-layer 1D-CNN structure is adopted: the first layer has a kernel size of 3, output channels of 32, and ReLU activation; the second layer has a kernel size of 3, output channels of 64, and ReLU activation. After global max pooling, a 64-dimensional heart rate feature vector is output. .

[0055] Cross-modal feature fusion: Unlike simple probability weighting at the decision level, deep fusion is performed at the feature level. The system concatenates the feature vectors extracted independently from the three paths mentioned above: This generates a global fusion feature vector with a dimension of 384. This vector fully encompasses the coordinated information of visual expression, eye movement rhythm, and the autonomic nervous system.

[0056] Trait-aware hard routing control: This step is the core mechanism of this invention. Traditional hybrid expert networks (MoE) often use softmax soft routing, calculating and weighting all expert branches, resulting in huge computational overhead. This system innovatively introduces a hard routing mechanism based on psychological traits. The system internally constructs four structurally identical but parameter-independent expert classifiers (Expert A, B, C, D), corresponding to four types of driving groups: resilient, hardworking, passive, and vulnerable, respectively. The hard routing controller reads the trait labels passed from the front-end 101 module. Generate a 4-dimensional one-hot routing vector. The generation rules are as follows: Where, k∈{A,B,C,D}, These are the trait categories assigned to drivers using the LPA model.

[0057] During the forward propagation of inference, the system only incorporates fused features. Import the corresponding In the expert classifier with a value of 1, the computational pathways of the other three dormant expert branches are completely truncated. This mechanism allows the model to maintain high precision while requiring only the computational cost of a single inference iteration, equivalent to that of a single-branch network, thus greatly satisfying the lightweight requirements of automotive-grade edge computing devices.

[0058] Dedicated expert inference and probabilistic output: The activated trait-specific expert classifier (essentially a single-layer MLP, with an input layer dimension of 384, a hidden layer dimension of 128, and an output layer dimension of 3) fuses features. A nonlinear mapping is performed. Then, the posterior probability distribution of the features at the three fatigue levels (mild, moderate, and severe) is output using the Softmax function. The formula is as follows: , Due to the characteristics of hard routing, the above formula only calculates the activation terms. The system ultimately extracts the category corresponding to the maximum value in this probability distribution as the driving fatigue level determination result for the current time window, and then passes it to the downstream output module 105.

[0059] As alternative solutions, physical-level model switching (storing multiple independent models, loading the corresponding model and unloading the rest of the models according to the feature labels) or Top-K sparse routing are used in this field. These solutions all utilize feature labels to achieve computational power truncation, which is an equivalent replacement for the hard routing concept of this invention.

[0060] like Figure 6 As shown, in this embodiment, the output module 105 is connected to the fatigue adaptability monitoring model module of the expected value control theory 104, and is responsible for converting the soft probability results output by the model into automotive-grade physical classification intervention strategies. Its specific execution logic is as follows: First, the module receives the posterior probability distribution of fatigue level output from the upstream trait-specific expert classifier, which is a vector containing the probabilities of three fatigue levels, as shown in the following formula:

[0061] Subsequently, the system enters the decision logic node based on the strategy triggered by the highest probability.

[0062] The system extracts the maximum value from the probability distribution through the Argmax operation, uses the fatigue level corresponding to the maximum posterior probability as the final state determination for the current time window, and activates one of the following three hierarchical strategy branches accordingly: Maintain the status quo branch (corresponding to no / mild fatigue): when When the probability of mild fatigue reaches its maximum, the system determines that the driver is in good condition or in a very early stage of fatigue that can be compensated for by the driver himself. To avoid excessively disturbing the driver, the system implements a status quo strategy, recording the status only in the background log or providing very slight visual indication signals on the instrument panel.

[0063] Moderate warning branch: When When the (probability of moderate fatigue) reaches its maximum value, a moderate warning strategy is triggered. At this point, the system determines that fatigue has substantially affected driving cognition and will invoke multimodal reminder methods through the in-vehicle human-machine interface (HMI), such as combining in-vehicle voice warnings with tactile feedback (such as high-frequency vibration of the steering wheel or seat belt pretensioning), to forcibly increase the driver's level of alertness.

[0064] High-alert branch (corresponding to severe fatigue): When When the probability of high fatigue reaches its maximum value, a high-alert strategy is triggered. If the system determines that the driver has entered a high-risk microsleep or severely lethargic state, it will immediately trigger the highest level emergency alarm sound and forcibly link the in-vehicle advanced driver assistance system (ADAS) to execute vehicle-level control commands, such as activating hazard lights, automatically intervening in braking and deceleration, or even performing a safe pullover to prevent potential serious traffic accidents.

[0065] Finally, the activated single strategy flows to the merged early warning decision and trigger intervention stage. The system encapsulates the abstract decision logic into standard vehicle CAN bus control messages and sends them to each physical actuator, ultimately completing the system closed loop for driver fatigue level warning output.

[0066] Because the four driving groups categorized based on Expected Value Control (EVC) theory inherently exhibit imbalances in data distribution, and collecting large-scale fatigue data for a single individual is extremely difficult, this invention innovatively proposes a two-stage training strategy of "global pre-training + trait-specific fine-tuning," such as... Figure 7 As shown in the figure, this strategy not only ensures the model's ability to generalize and extract basic fatigue features across traits, but also enables the expert classifier to accurately adapt to the trait group.

[0067] Phase 1: Global Pre-training The goal of this stage is to train a general multimodal feature encoding network and a global classifier head, learning a general fatigue representation across traits. The specific process is as follows: Training input: Full pooling of all samples of all trait categories (A / B / C / D) in the training set, totaling 56 drivers (70 people in the total dataset minus 14 people in the test set).

[0068] Network structure: A complete multimodal feature encoding network (3 branches) + a global classification head. The global classification head has the same structure as the expert classifier (384→128→3) and is only used in the pre-training stage.

[0069] Optimizer: Adam optimizer, with an initial learning rate of 3×10^-4.

[0070] Loss function: Weighted cross-entropy loss. in, For batch size, The weight of category c, For indicator functions, For the true label of sample i, Predict the probability that sample i belongs to class c for the model. Weights The calculation formula is: ,in The total number of samples in the training set. The number of samples in category c.

[0071] To address the imbalance in fatigue level categories.

[0072] Training rounds: 50 rounds, batch size is fixed at 32, training is stopped early when the accuracy on the validation set does not improve for 5 consecutive rounds.

[0073] Output: Saves the pre-trained converged feature encoder weights and global classifier head weights.

[0074] Phase 2: Personalized fine-tuning of the four trait expert models The goal of this stage is to adapt each expert classifier to the fatigue characteristics of the corresponding trait group. The specific process is as follows: Weight initialization: All four expert classifiers are initialized with the global classifier head weights pre-trained in Stage 1 to ensure consistent initial performance baseline.

[0075] Training rules: Freeze all weights of the feature encoding network and fine-tune only the parameters of the expert classifiers; each expert classifier is tuned using only samples of the corresponding trait class in the training set (A-class experts use only A-class samples, and so on).

[0076] Hyperparameter settings: Adam optimizer, initial learning rate fixed at 1×10^-4 (reduced from the pre-training stage to prevent damage to pre-trained features), 30 training epochs, batch size fixed at 16.

[0077] Loss function: Weighted cross-entropy loss. in, For batch size, The weight of category c, For indicator functions, For the true label of sample i, Predict the probability that sample i belongs to class c for the model. Weights The calculation formula is: ,in The total number of samples in the training set. The number of samples in category c.

[0078] To address the imbalance in fatigue level categories.

[0079] For small sample groups with specific characteristics, such as Class A, an early stopping strategy is adopted during fine-tuning (training is stopped if there is no improvement in the validation set loss for 5 consecutive rounds) to avoid model overfitting.

[0080] Output: The four trait-specific expert classifier weights, after fine-tuning, together with the pre-trained encoder, form the complete inference model of this invention.

[0081] To verify the technical effectiveness of this invention, a comparative experiment was conducted on the HRWS (Hybrid Real-World Sets) multimodal driving fatigue dataset, which was collected from our own experiments. This dataset collected physiological data, facial data, and psychological assessments from 80 people driving under different levels of fatigue. The baseline model uses a multimodal coding network that is completely consistent with that of this invention, only replacing the hard routing expert hybrid network with a globally unified classification head, while keeping the other hyperparameters and training environment the same.

[0082] The experiment was conducted using an Intel 5th generation Xeon Scalable processor (Emerald Rapids architecture), an H100 graphics card, and the PyTorch 2.8.0 framework.

[0083] The above-described product can perform the methods provided in the embodiments of this application, and has the corresponding functional modules and beneficial effects for performing the methods. Technical details not described in detail in this embodiment can be found in the methods provided in the embodiments of this application.

[0084] Non-volatile computer-readable storage media may include a stored program area and a stored data area, wherein the stored program area may store an operating system and an application program required for at least one function; the stored data area may store data created based on the use of the device, etc. Furthermore, the non-volatile computer-readable storage medium may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the non-volatile computer-readable storage medium may optionally include memory remotely located relative to the processor, and these remote memories may be connected to the device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0085] This invention also provides an electronic device comprising: at least one processor and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the steps of the fatigue adaptation monitoring method and system based on the psychological expectation value control theory of any embodiment of this invention.

[0086] The electronic devices described in this application exist in various forms, including but not limited to: (1) Mobile communication devices: These devices are characterized by their mobile communication capabilities and primarily aim to provide voice and data communication. These terminals include smartphones, multimedia phones, feature phones, and low-end phones.

[0087] (2) Ultra-mobile personal computer devices: These devices fall under the category of personal computers, possessing computing and processing capabilities, and generally also have mobile internet access features. These terminals include PDAs, MIDs, and UMPCs, such as tablet computers.

[0088] (3) Portable entertainment devices: These devices can display and play multimedia content. This category includes audio and video players, handheld game consoles, e-book readers, as well as smart toys and portable car navigation devices.

[0089] (4) Vehicle computing and edge devices: These devices are integrated into the vehicle cabin or vehicle electronic control unit (ECU) to collect driver multimodal signals in real time, execute fatigue adaptive monitoring algorithms and link with advanced driver assistance systems (ADAS).

[0090] (5) Other electronic devices with data processing functions.

[0091] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0092] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, vehicle embedded system, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments.

[0093] The above description is merely illustrative of the embodiments of the present invention and is not intended to limit the present invention. For those skilled in the art, any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A fatigue monitoring method based on the psychological expectancy-value control theory, characterized in that, Includes the following steps: The test driver acquires multimodal physiological signals, where multimodal refers to data from two different modalities: physiological signals and behavioral signals. Specifically, the data includes heart rate (HR) physiological signals, as well as behavioral signals such as facial images, the Big Five personality disorder questionnaire, and the somnolence scale, and combines them with the trait labels of the target driver determined based on the expectancy-value control theory. The multimodal physiological and behavioral signals are input into a pre-trained multimodal feature encoding network, and multimodal fusion features are extracted and fused. The multimodal fusion features are input into an expert hybrid network consisting of four expert classifiers of different categories. Based on the trait labels, the dedicated expert classifiers corresponding to the trait labels are activated through a trait-aware hard routing mechanism. The activated dedicated expert classifiers output the current driving fatigue level.

2. The fatigue adaptation monitoring method based on the psychological expectancy-value control theory according to claim 1, characterized in that, The driver trait labels determined based on expected value control theory specifically include: Acquire the Big Five personality test data and the sleepiness scale attribute data of the driver, and perform index scoring, reverse processing and Z-score standardization on the above two types of data. Based on the expected value control theory, the data after Z-score standardization is mapped into a two-dimensional feature space coordinate system of revenue and cost. The two-dimensional feature space coordinates are clustered using a potential profile analysis algorithm. After the driver's standardized data is clustered, it is classified into four trait groups and the corresponding trait labels are output. The four trait groups include: high cost & low profit, high cost & high profit, low cost & low profit, and low cost & high profit.

3. The fatigue adaptation monitoring method based on the psychological expectancy-value control theory according to claim 2, characterized in that, The psychological assessment data includes scores for conscientiousness and neuroticism from the simplified version of the Chinese Big Five Personality Questionnaire and scores from the Stanford Somnolence Scale; the driving attribute data includes driving experience, and the driving experience is scored in reverse.

4. The fatigue adaptation monitoring method based on the psychological expectancy-value control theory according to claim 1, characterized in that, The multimodal physiological and behavioral signals include video image stream signals acquired by vehicle-mounted cameras, eye aspect ratio time series calculated based on eye key points, and heart rate time series acquired by electrocardiogram or heart rate sensors; the multimodal feature coding network includes facial visual stream coding branch, eye movement time series coding branch, and physiological time series coding branch. Each branch is independently coded and then the features are spliced ​​and fused to obtain the multimodal fused features.

5. The fatigue adaptation monitoring method based on the psychological expectancy-value control theory according to claim 1, characterized in that, The trait-aware hard routing mechanism is as follows: a one-hot routing vector is generated based on the trait label, and only one of the four classification experts in the corresponding dimension is activated. The computation channels of the other expert classifiers are completely truncated, and only the forward computation of a single expert branch is performed during inference.

6. A fatigue adaptation monitoring system based on the psychological expectancy-value control theory, characterized in that, include: The driver trait pre-identification module is used to acquire drivers' psychological assessment and driving attribute data, construct two-dimensional features based on expected value control theory, and output trait labels through latent profile analysis. The multimodal data synchronous acquisition module is used to synchronously acquire video visual streams and ECG / heart rate signals and complete global timestamp alignment. The data preprocessing module is used to perform time downsampling, face detection, eye aspect ratio calculation, and heart rate calculation on the acquired signals to generate standardized multimodal temporal tensors. The fatigue adaptation monitoring model module based on the psychological expectation value control theory has a built-in multimodal feature encoding network, a trait-aware hard routing controller, and multiple trait-specific expert classifiers, which are used to activate the corresponding expert classification based on the trait label and output the fatigue level. The output module is used to trigger corresponding tiered intervention strategies based on fatigue levels.

7. A fatigue adaptation monitoring system based on the psychological expectancy-value control theory according to claim 6, characterized in that, The multimodal data synchronization acquisition module includes a hardware sensor layer, a raw signal acquisition layer, a time synchronization layer, and an output terminal. It timestamps the video stream and ECG signal respectively and completes absolute time alignment based on the system master clock.

8. A fatigue adaptation monitoring system based on the psychological expectancy-value control theory according to claim 6, characterized in that, The data preprocessing module uses a fixed non-overlapping sliding window of 20 seconds, employs stage-aware dynamic frame sampling for the visual stream, uses Otsu's method for adaptive threshold calibration of the eye aspect ratio, and performs filtering and noise reduction and instantaneous heart rate calculation on the electrocardiogram signal.

9. A fatigue adaptation monitoring system based on the psychological expectancy-value control theory according to claim 6, characterized in that, The fatigue adaptation monitoring model module based on the psychological expectation value control theory adopts a two-stage training method of global pre-training plus trait-specific fine-tuning: in the global pre-training stage, the general feature encoder and global classifier are trained using full-trait samples; in the trait-specific fine-tuning stage, the encoder weights are frozen, and each expert classifier is fine-tuned only using corresponding trait samples.

Images