Brain fatigue dynamic early warning method and device, computer equipment and storage medium

Abstract

The invention provides a brain fatigue dynamic early warning method and device, computer equipment and a storage medium, and belongs to the field of traffic safety, and the method comprises the steps: obtaining a pupil diameter entropy time sequence, a staring entropy time sequence, time features and a preset task stage label of a driver; preprocessing the features respectively, and splicing preprocessing results to obtain a joint feature matrix; performing time sequence feature extraction on the joint feature matrix to generate shared hidden features; respectively generating distribution parameters of the cognitive load, the fatigue state and the alertness state from the shared hidden features, and further determining state values of the fatigue state and the alertness state from the distribution parameters through re-parametric sampling; and performing early warning according to the state values of the fatigue state and the alertness state in combination with the driving scene. Therefore, through multi-modal fusion, mechanism constraint decoupling and dynamic risk assessment, the brain fatigue state is determined through the pupil state, and the traffic accident rate caused by fatigue in the field of driving safety can be reduced.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of traffic safety, and particularly relates to a brain fatigue dynamic early warning method and device, a computer device and a storage medium. BACKGROUND

[0002] Traffic safety is an important part of building a modern traffic safety prevention and control system, and according to data, more than 20% of traffic accidents are caused by fatigue driving every year.

[0003] The causes of driving brain fatigue are complex and diverse, mainly including fatigue caused by circadian rhythm and sleep debt, and fatigue caused by tasks (active fatigue and passive fatigue). Among them, active fatigue is caused by long-term and high-intensity driving activities. When driving in complex road conditions, the driver needs to maintain high concentration at all times and frequently perform driving operations, which causes excessive consumption of physical and brain resources and causes fatigue. Passive fatigue is caused by monotonous driving tasks and lack of sufficient stimulation, which causes the brain to be in a low-load operation state for a long time, making it difficult for the driver to concentrate, and gradually causing fatigue symptoms such as drowsiness and slow reaction. The four causes of fatigue are different, and the thresholds for early warning and the strategies for intervention are also different. Dynamic monitoring and distinguishing these factors during driving is the key to targeted early warning and reasonable intervention.

[0004] However, existing common technologies cannot distinguish different types of fatigue. Behavioral tests such as PVT (psychomotor vigilance task) and the Mackworth clock experiment rely on peripheral muscle activity, and their results are easily disturbed by muscle factors; the PERCLOS method can quantify the degree of eyelid closure, but skeletal muscle activity can affect its accuracy; electroencephalogram and magnetic resonance devices are difficult to apply to driving scenarios due to their large size and harsh use conditions; and HRV (heart rate variability) is significantly different between individuals and cannot specifically reflect the brain alertness state.

[0005] The locus coeruleus (LC, or simply LC) as the "alertness control center" regulates the global alertness state. The pupil diameter change is highly synchronized with the activity of the locus coeruleus, providing an important window for the study of brain fatigue. Traditional static models cannot analyze the dynamic neural mechanisms such as "exploration-exploitation" strategy switching and locus coeruleus activation. Traditional models usually assume that the effects between factors are linear and constant, but in reality, the interaction between various factors in the cognitive process is complex and variable, showing non-linear and dynamic characteristics. This leads to the fact that traditional models cannot deeply mine the rich neurophysiological information behind the fluctuations of PCI (pupil-locus coeruleus complexity index), limiting the progress of precise modeling and mechanism analysis of cognitive state. Therefore, how to determine the brain fatigue state according to the pupil state is the key to targeted early warning and reasonable intervention. SUMMARY

[0006] In order to solve the above problems, the application provides a brain fatigue dynamic early warning method, device, computer equipment and storage medium.

[0007] In order to achieve the above purpose, the application provides the following technical scheme: A brain fatigue dynamic early warning method, the method comprises: Real-time collection of driver's pupil diameter entropy time sequence and gaze entropy time sequence; determining time characteristics based on current time; obtaining preset task stage label corresponding to current time; Splicing the pupil diameter entropy time sequence, the gaze entropy time sequence, the time characteristics and the preset task stage label to obtain a joint feature matrix; Time sequence feature extraction is performed on the joint feature matrix to generate shared hidden features; distribution parameters of cognitive load, fatigue state and vigilance state are generated from the shared hidden features, and then state values of fatigue state and vigilance state are determined from the distribution parameters through reparameterization sampling; According to the state values of the fatigue state and the vigilance state, the driving scene is combined to perform early warning.

[0008] Optionally, before splicing the pupil diameter entropy time sequence, the gaze entropy time sequence, the time characteristics and the preset task stage label, preprocessing is further performed, including: The pupil diameter entropy time sequence E(t) and the gaze entropy time sequence G(t) are respectively standardized, and the specific formula is: ; ; Wherein, And The sequence mean and the standard deviation of the pupil diameter entropy time sequence are respectively, And The sequence mean and the standard deviation of the gaze entropy time sequence are respectively; The time characteristics h ( t ) are encoded into Fourier bases of 24-hour period And , to generate time encoding characteristics , and the specific formula is: ; Wherein, k The first and second harmonics are respectively, h ( t ) is the time characteristics; The preset task stage label is one-hot encoded to generate a task feature vector , and the specific formula is: ; in, Number of task phases. v i It is the i-th element in the one-hot encoded vector.

[0009] Optionally, the step of concatenating the pupil diameter entropy time series, the gaze entropy time series, the time features, and the preset task stage labels to obtain a joint feature matrix includes: Preprocessed and The joint feature matrix is ​​constructed by concatenating the features along the time dimension, as shown in the following formula: ; in, The joint characteristic matrix at time t has the following shape: T is the time step. The dimension of the input feature. This is the symbol for the correlation coefficient matrix.

[0010] Optionally, temporal feature extraction is performed on the joint feature matrix using a TCN network, wherein the TCN network includes dilated convolutional layers, defined as follows: ; in, The shared hidden features output by the dilated convolutional layer. The input joint feature matrix, L This represents the number of dilated convolution layers.

[0011] Optionally, after determining the state values ​​of fatigue and alertness, the method further includes: Determine the state value of cognitive load; The pupil diameter entropy signal and the gaze entropy signal are reconstructed based on the state values ​​of cognitive load, fatigue state, and alertness state, respectively. The reconstruction formula for the pupil diameter entropy signal is as follows: ; in, To reconstruct the generated pupil diameter entropy signal, These are cognitive load, fatigue state, and alertness state, respectively. The weight matrix for generating the pupil diameter entropy branch. This is the bias term for the pupil diameter entropy generation branch; The reconstruction formula for the gaze entropy signal is: ; in, These represent the state values ​​of cognitive load, fatigue state, and alertness state at time step t, respectively. This represents the feature vector of the task stage corresponding to time step t. Let be the hidden state of the LSTM network at time step t−1; To reconstruct the generated gaze entropy signal, The weight matrix for generating branches of gaze entropy. The bias term for the gaze entropy generation branch; according to and Determine the pupil diameter entropy signal error based on and The gaze entropy signal error is determined, and the fatigue state and alertness state are verified based on the pupil diameter entropy signal error and the gaze entropy signal error. The real-time risk index is then calculated based on the verified fatigue state and alertness state.

[0012] Optionally, before reconstructing the pupil diameter entropy signal and the gaze entropy signal, dynamic prior constraints are applied to cognitive load, fatigue state, and alertness state, including: Cognitive load Perform piecewise stability priors; for fatigue states Perform Brownian motion prior with drift ; state of alertness Perform periodic normal prior Synchronize with the 24-hour biological clock; among which, This is the preset fatigue state drift amount. This represents the standard deviation of the fatigue state fluctuation.

[0013] Optionally, the step of issuing a warning based on the state values ​​of the fatigue state and alertness state combined with the driving scenario is as follows: A risk index is generated based on the state values ​​of fatigue and alertness, using the following formula: ; in, The state value of the fatigue state at time t after verification. The verified alert state value at time t. and These are the variable parameters; Warnings are issued based on the risk index and the risk thresholds corresponding to different driving scenarios.

[0014] A dynamic early warning device for brain fatigue, the device comprising: The acquisition module is used to collect the time series of the driver's pupil diameter entropy and gaze entropy in real time; and to determine the time features based on the current time. The preprocessing module is used for splicing the pupil diameter entropy time sequence, the gaze entropy time sequence, the time feature and the preset task stage label to obtain a joint feature matrix. The determining module is used for performing time sequence feature extraction on the joint feature matrix to generate shared hidden features, generating distribution parameters of cognitive load, fatigue and vigilance state from the shared hidden features, and then determining state values of the fatigue state and the vigilance state from the distribution parameters through reparameterization sampling. The early warning module is used for early warning according to the state values of the fatigue state and the vigilance state in combination with a driving scene.

[0015] A computer readable storage medium, characterized in that the storage medium stores a computer program, and the computer program is executed by a processor to implement the brain fatigue dynamic early warning method.

[0016] A computer device, comprising a memory, a processor and a computer program stored on the memory and executable on the processor, and the processor implements the brain fatigue dynamic early warning method when executing the program.

[0017] The brain fatigue dynamic early warning method provided by the application has the following beneficial effects: The pupil diameter entropy time sequence collected in real time by the application is directly related to the neural activity of the locus coeruleus nucleus, and can capture fatigue signals caused by metabolic product accumulation; the gaze entropy time sequence reflects the stability of eye movement strategy, and indirectly represents attention allocation and cognitive load fluctuation, and the joint feature matrix is generated by feature splicing in combination with time features and preset task stage labels, effectively fusing neural metabolism, behavior patterns and environmental context information, solving the limitation that a single signal is easily disturbed by noise, and significantly enhancing the robustness of fatigue representation; then, deep time sequence dependence of the joint feature matrix is extracted to generate shared hidden features, and distribution parameters of fatigue and vigilance states are generated to further determine state values, realizing determination of brain fatigue state through pupil state, and decoupling of physiological interpretability of the fatigue state; finally, early warning is performed based on the fatigue state and the vigilance state in combination with a driving scene. In this way, the application upgrades the "single-dimensional threshold alarm" of traditional fatigue monitoring to an intelligent prevention and control system of "neural metabolism-behavior-environment collaborative analysis" of brain fatigue state determination according to pupil state through multi-modal fusion, mechanism constraint decoupling and dynamic risk assessment, which can reduce the traffic accident rate caused by fatigue in the field of driving safety, and provides a quantitative basis for personalized health management. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present application and the design scheme thereof, the following will briefly introduce the drawings required by the present embodiment. The drawings in the following description are only part of the embodiments of the present application, and other drawings can also be obtained according to these drawings without creative labor for those skilled in the art.

[0019] Figure 1 A flowchart of a brain fatigue dynamic early warning method provided by the present application according to an exemplary embodiment.

[0020] Figure 2 A model block diagram of an MDD-VAE provided by the present application according to an exemplary embodiment.

[0021] Figure 3 A cross-modal feature fusion schematic diagram provided by the present application according to an exemplary embodiment.

[0022] Figure 4 A driving scene consistency verification flowchart provided by the present application according to an exemplary embodiment.

[0023] Figure 5 An integrated schematic diagram of a closed-loop safety prevention and control system provided by the present application according to an exemplary embodiment.

[0024] Figure 6 A brain fatigue dynamic early warning device block diagram provided by the present application according to an exemplary embodiment. DETAILED DESCRIPTION

[0025] In order to make those skilled in the art better understand the technical solutions of the present application and can be implemented, the present application will be described in detail below in conjunction with the drawings and specific embodiments. The following embodiments are only used to more clearly illustrate the technical solutions of the present application, and cannot limit the protection scope of the present application.

[0026] The present application focuses on the dynamic interaction mechanism of cognitive state by pupil diameter entropy (E(t)) and gaze entropy (G(t)) bimodal signals. Pupil diameter entropy directly represents the neural activity of locus coeruleus norepinephrine system (LC-NE), and its dynamic change is regulated by multiple factors: it reflects the instantaneous cognitive load level (related to task difficulty, individual experience and reward mechanism), presents the time cumulative fatigue effect, and is also affected by the fluctuation of vigilance state caused by circadian rhythm and environmental stimulation. As an attention allocation indicator controlled by eye movement muscles, gaze entropy can indirectly reflect the cognitive load and fatigue degree, but its signal characteristics are significantly affected by the periodic modulation of vigilance level. The core scientific challenge is to decouple the four key potential variables from these two types of time series signals with different physiological sources (autonomic nervous system vs skeletal muscle system) but coupled relationship: ① task-driven cognitive load (regulated by task complexity, experience accumulation and incentive strength); ② monotone increasing cumulative fatigue over time (presented as nonlinear accumulation of neural metabolites); ③ fluctuation caused by exploration / exploitation mode switching; ④ vigilance level jointly affected by biological clock and environmental stimulation (presenting approximately 24-hour periodic fluctuation). The dynamic decoupling of this multi-dimensional state has important theoretical value for precise cognitive state evaluation. According to the decoded state, targeted intervention can be carried out, such as active fatigue caused by high cognitive load, which requires reducing task load, low cognitive load causing decreased vigilance level, which can be improved by remote call communication, and fatigue caused by sleep debt, which requires rest.

[0027] The technical solutions of the embodiments of the present application are described in detail below with reference to the drawings.

[0028] Firstly, the present application provides a brain fatigue dynamic early warning method, as shown in Figure 1 The method comprises the following steps: S101, real-time acquisition of the pupil diameter entropy time series and the gaze entropy time series of the driver; determination of the time characteristics based on the current time; acquisition of the preset task stage label corresponding to the current time.

[0029] In this step, first, an infrared camera or an eye tracker device can be used to capture the driver's eye image at a high frame rate (e.g. 100 Hz), or the pupil position can be accurately located by the pupil center corneal reflection technology (PCCR), and the pupil diameter can be calculated, and the gaze point coordinates can be recorded synchronously to form the original time series data.

[0030] Then the pupil diameter time series is segmented by sliding window, and the sample entropy or approximate entropy of the signal in each window is calculated to obtain the pupil diameter entropy time series, which is used to reflect the nucleus locus activity; based on the gaze point coordinate sequence, the randomness of the visual line distribution is quantified by Markov chain or information entropy model to obtain the gaze entropy time series, which represents the eye movement strategy.

[0031] For the time feature, the time feature h(t) represents the hour value of the current time, which can be read from the clock module in real time. For example, h(t)=9 at 9 o'clock in the morning, h(t)=22 at 22 o'clock at night, or the cumulative hours after the start of each time point of the driving task (for example, when the driving lasts for 3 hours, h(t)=3), which is used to analyze the cumulative effect of fatigue.

[0032] S102, splicing the pupil diameter entropy time series, gaze entropy time series, time feature and preset task stage label to obtain a joint feature matrix.

[0033] In this step, the four features obtained in the preceding steps need to be preprocessed, including the following steps.

[0034] First, the pupil diameter entropy time series E(t) and the gaze entropy time series G(t) are standardized, and the specific formula is: ; ; Among them, is the standardized pupil diameter entropy time series, is the standardized gaze entropy time series, and are the sequence mean and standard deviation of the pupil diameter entropy time series, and are the sequence mean and standard deviation of the gaze entropy time series; Secondly, the time feature is encoded into 24-hour Fourier basis and to generate the time encoding feature , and the specific formula is: ; Among them, k is the first and second harmonic, h ( t) is a time feature representing the current time in hours. This encoding maps discrete hour values into a continuous periodic signal, enhancing the model's ability to model circadian rhythms (e.g., 24-hour fluctuations in alertness levels), is a key input for representing circadian rhythms, and is converted into a model-resolvable periodic signal through Fourier encoding, providing temporal context support for fatigue state decoupling.

[0035] Then the preset task phase label is one-hot encoded to generate a task feature vector , and the specific formula is: ; wherein, is the number of task phases, v i is the i-th element in the one-hot encoding vector.

[0036] Finally, the four feature preprocessing results are spliced, i.e., the preprocessed and are spliced along the time dimension to form a joint feature matrix, and the specific formula is: ; wherein, represents the joint feature matrix at time t, with a shape of , T is the time step, is the dimension of the input feature, is the correlation coefficient matrix symbol.

[0037] S103, time series feature extraction is performed on the joint feature matrix to generate shared hidden features; distribution parameters of cognitive load, fatigue state and alertness state are generated from the shared hidden features, and then state values of fatigue state and alertness state are determined from the distribution parameters through reparameterization sampling.

[0038] In this step, a multi-modal dynamic decoupling variational autoencoder MDD-VAE is constructed, which includes a TCN network and a latent variable branch network; after the joint feature matrix is input into the multi-modal dynamic decoupling variational autoencoder MDD-VAE, the TCN network performs time series feature extraction on the joint feature matrix to generate shared hidden features; the independent multi-layer perceptron MLP in the latent variable branch network generates distribution parameters of cognitive load, fatigue and alertness state from the shared hidden features, and then determines the state values of fatigue state, alertness state and cognitive load according to the distribution parameters.

[0039] In addition, in order to train the MDD-VAE to improve accuracy, and to perform signal denoising and feature enhancement, the MDD-VAE can further include a double-branch decoder, which includes a pupil diameter entropy generation branch and a gaze entropy generation branch, and reconstructs the pupil diameter entropy signal and the gaze entropy signal according to the state values of the cognitive load, the fatigue state and the vigilance state, respectively, through the double-branch decoder; the pupil diameter entropy generation branch is implemented through a dynamic generation mechanism by stacking a TCN layer, and the specific formula is: ; wherein, is the reconstructed pupil diameter entropy signal, respectively, are the cognitive load, the fatigue state and the vigilance state, is a weight matrix of the pupil diameter entropy generation branch, used to map the output of the decoder TCN layer to the reconstructed signal space, is a time encoding feature, and is a bias term of the pupil diameter entropy generation branch, adjusts the baseline of the reconstructed signal.

[0040] The gaze entropy generation branch is an LSTM-based time sequence decoder, and the specific formula is: ; wherein, respectively, are the latent variables at time step t, which specifically correspond to the state values of the cognitive load, the fatigue state and the vigilance state, is a task phase feature vector corresponding to time step t, which is generated by one-hot encoding, is a hidden state of the LSTM network at time step t−1; is the reconstructed gaze entropy signal, is a weight matrix of the gaze entropy generation branch, is a bias term of the gaze entropy generation branch.

[0041] The pupil diameter entropy signal error is determined according to and The gaze entropy signal error is determined according to and The latent variables are verified based on the pupil diameter entropy signal error and the gaze entropy signal error.

[0042] Therefore, in this step, first of all, it is necessary to build a multi-modal dynamic decoupling variational autoencoder MDD-VAE. The model aims to build a multi-modal fusion framework based on a variational autoencoder, decouple the interaction of cognitive load, cumulative fatigue and alertness level through dynamic decoupling, and achieve the following core goals: 1) state decoupling - separate the latent variables with independent representation ability from the mixed signals of pupil diameter entropy (autonomic nervous regulation) and gaze entropy (motor control driven), to ensure that the time sequence characteristics of cognitive load, fatigue and alertness are not confused with each other; 2) physiological interpretability - through KL divergence regularization and dynamic prior constraints (such as the monotonicity of the forced fatigue state and the 24-hour periodicity of the alertness state), the latent variables strictly follow the neurophysiological mechanism; 3) cross-modal complementary enhancement - use the locus coeruleus sensitivity of the pupil signal (high load response) and the eye movement strategy characteristics of the gaze signal (alertness modulation) to design a cross-modal alignment loss function, and improve the joint inference ability for complex cognitive states.

[0043] The model architecture is based on a multi-modal dynamic decoupling variational autoencoder (MDD-VAE), which separates cognitive load, fatigue and alertness states in a structured latent space by jointly encoding pupil diameter entropy and gaze entropy time series, and injects dynamic prior constraints (fatigue monotonicity, alertness periodicity). Finally, through a double-branch decoder, the physiological signals are reconstructed to achieve cross-modal complementary physiological interpretable state decoupling. As shown in Figure 2 Pupil_Entropy, Gaze_Entropy, Time_Features and Task_Labels are the pupil diameter entropy time series, gaze entropy time series, time features and preset task phase labels, respectively. Preprocess is the preprocessing step to generate a joint feature matrix Combined_Features: [E, G:Time, Task]. Then, a shared hidden feature is output by the feature extraction network TCN, and then the distribution parameters of cognitive load, fatigue state and alertness state are generated. Then, the state values of cognitive load, fatigue state and alertness state are generated by resampling, and finally the pupil diameter entropy and gaze entropy time series are reconstructed based on the state values and the reconstruction loss is determined to minimize the loss to complete the training.

[0044] In one embodiment, the MDD-VAE includes the following modules and functions.

[0045] Feature extraction (TCN network). The time convolution network TCN is used as the feature extraction network to model the time dependence, and the dilated convolution layer is defined as:

[0046] ; wherein, is the shared hidden feature output by the dilated convolution layer, L is the number of dilated convolution layers, L = 8, and the dilation factor increases exponentially in each layer The purpose of using dilated convolution is to expand the receptive field, and this method has significant advantages in parameter efficiency, long-distance dependency modeling, and resolution preservation. The output of each layer is Layer normalization (LayerNorm) is applied to the output of each layer.

[0047] Latent variable branch. The latent variable distribution parameters are generated by an independent multi-layer perceptron (MLP):

[0048] ; where, .

[0049] MLP structure: fully connected layer ( ) + GELU activation function.

[0050] Reparameterization sampling. The latent variable is sampled from a Gaussian distribution using the reparameterization trick:

[0051] ; where, is the center value of the latent variable, is the uncertainty measure of the latent variable, is the standard normal noise.

[0052] Dual-branch decoder. The pupil diameter entropy generation branch is a dynamic generation mechanism implemented by stacking TCN layers:

[0053] ; where, is the reconstructed generated pupil diameter entropy signal, are the cognitive load variable, fatigue state, and vigilance state, respectively, is the weight matrix of the pupil diameter entropy generation branch, is the time-encoding feature, is the bias term of the pupil diameter entropy generation branch.

[0054] Latent variables and time features Concatenate along the feature dimension. 4 layers of dilated convolution (dilation factor ), and the output dimension of each layer is The projection layer maps the high-dimensional features of the hidden layer to a one-dimensional real space to generate the final output value.

[0055] Gaze entropy generation branch is an LSTM-based time sequence decoder: ; where, are the latent variables at time step t, corresponding to cognitive load, fatigue state and vigilance state respectively, is the task phase feature vector at time step t, generated by one-hot encoding, is the hidden state of the LSTM network at time step t−1; is the reconstructed gaze entropy signal, is the weight matrix of the gaze entropy generation branch, is the bias term of the gaze entropy generation branch. The LSTM layer is a 2-layer stack with hidden state dimension . The latent variables are conditioned on the task features . The hidden state is initialized by the global average pooling feature of the TCN encoder.

[0056] The pupillary diameter entropy signal error is determined according to and The gaze entropy signal error is determined according to and The latent variables are validated based on the pupillary diameter entropy signal error and the gaze entropy signal error.

[0057] Further, to ensure the physiological meaning of the latent variables, the model approximates the distribution to a preset prior by KL divergence, i.e., before reconstructing the reconstructed pupillary diameter entropy signal and the reconstructed gaze entropy signal, the latent variables are also subject to dynamic prior constraints. For the cognitive load variable a piecewise stationary prior is performed to match the task difficulty label; for the fatigue state a Brownian motion prior with drift is performed to enforce a monotonic increasing trend; for the vigilance state a periodic normal prior is performed to synchronize the 24-hour biological clock. Wherein, is the fatigue state at time step t, is the preset fatigue state drift, is the fluctuation standard deviation of the fatigue state, controlling the stationarity of the increasing trend.

[0058] Finally, the constructed model also needs to be further trained, and the optimization objective of the model consists of three parts: the reconstruction loss ensures the fidelity of signal generation; the KL divergence term forces the distribution of the latent variables to approximate the physiological prior; and the dynamic regularization term introduces domain knowledge constraints. The total loss function is defined as:

[0059] ; wherein is the KL divergence weight coefficient, For the strength parameter of each regular term, For the reconstruction loss function, For the KL divergence term, For the dynamic regular term.

[0060] Reconstruction loss. The reconstruction loss measures the difference between the decoder output and the original signal. Weighted mean squared error is adopted for different characteristics of the pupil and gaze signals:

[0061] ; The default weight is set to , , reflecting the higher signal-to-noise ratio of pupil diameter entropy compared to gaze entropy on locus coeruleus activity.

[0062] KL divergence term. The KL divergence constrains the alignment of the latent variable distribution with the pre-set prior , and different priors are designed for different physiological mechanisms:

[0063] Cognitive load: standard normal prior , the KL calculation is: .

[0064] Fatigue: Brownian motion prior with drift , whose KL term is: .

[0065] Alertness: Fourier periodic prior , the KL form is the same as cognitive load.

[0066] Dynamic regular term. Two regular constraints based on physiological knowledge are introduced:

[0067] Fatigue monotonicity: penalize non-increasing fatigue state, and force metabolic cumulative effect: .

[0068] Alertness cycle synchronization: ensure synchronization with circadian rhythm: .

[0069] Training strategy. KL annealing, linearly increases from 0.1 to 1.0, and the transition is completed in the first 20% of the training period to avoid the suppression of reconstruction learning by the initial KL term. The regular weight , is determined by grid search. The optimizer uses AdamW, the learning rate , and the weight decay to prevent overfitting.

[0070] S104, combining the fatigue state and alertness state with the driving scene to give a warning.

[0071] For example, a real-time risk index can be calculated according to the fatigue state and alertness state in combination with a pre-labeled driving scene label, and a warning can be given based on the classification threshold of the risk index. Alternatively, a risk index is generated according to the state values of the fatigue state and the alertness state, and the formula is:

[0072] ; wherein, is the state value of the fatigue state at time t after verification, is the state value of the alertness state at time t after verification, and are variable parameters, respectively; a warning is given based on the risk index and the risk threshold corresponding to different driving scenes.

[0073] The present application adopts a three-stage method of "data-driven classification-knowledge-guided optimization-closed-loop feedback verification" to construct a mapping rule library, and the specific process is as follows: The fatigue type classification standard is used to divide the potential variable threshold. Based on the statistical distribution (mean ± standard deviation) of the decoupled fatigue factor F(t) and alertness factor A(t), the fatigue state and fatigue type are divided. A dynamic mapping rule library of "cognitive state-intervention strategy" is constructed. Active fatigue (high cognitive load): trigger task degradation (such as ACC follow-up distance extension, voice assistant to take over simple tasks). Passive fatigue (low alertness): activate environmental stimulation (5000K cool light, mint fragrance) and remote communication activation. Sleep debt accumulation: hierarchical rest strategy (15-minute nap → forced service station stop). Dynamic priority mechanism: based on driving scene (highway / urban road) and real-time risk index , adjust the strategy trigger threshold, wherein, is the state value of the fatigue state at time t after verification, is the state value of the alertness state at time t after verification, and are variable parameters, respectively.

[0074] The application also carries out multi-modal data fusion and model explainability verification. First, multi-modal data acquisition and spatio-temporal alignment. The data sources cover multiple aspects. The core signals include pupil diameter sampled at 100 Hz and gaze coordinate points sampled at 100 Hz. The physiological indicators include heart rate variability (HRV) sampled at 1 Hz and EEG (theta / beta power ratio) sampled at 128 Hz. In terms of driving behavior, it involves steering angle variance sampled at 10 Hz and lane deviation frequency. There is also subjective evaluation data, which is obtained by scoring immediately after the task through the NASA-TLX scale. In the preprocessing process, first, the problem of different sampling rates of multiple devices is solved by using dynamic time warping (DTW) method for time alignment. After completing the time alignment, feature extraction is performed, and the extracted features are composed of multiple parts. Secondly, cross-modal feature fusion and enhancement are carried out. As shown in Figure 3 In the application, multiple methods are used to improve explainability verification. First, canonical correlation analysis (CCA) is used to maximize the cross-modal correlation between physiological signals and behavior indicators (lane deviation) and to explore the potential relationship between different modal data.

[0075] Then, adversarial training is carried out through a domain discriminator to constrain the consistency of feature distribution between different driving scenarios (high-speed scenarios and urban scenarios). The loss function of adversarial training is defined as , , which promotes the model to learn more general features in different scenarios. Finally, data enhancement strategy is adopted. On the one hand, time series adversarial generation is used to generate synthetic data of extreme driving scenarios (such as continuous curves with strong light interference) by using Wasserstein GAN, to enrich the diversity of data and enhance the adaptability of the model to complex scenarios. Feature mask strategy is implemented to randomly discard 20% of the gaze data, to simulate the temporary failure of the eye tracker and to carry out robustness training to improve the stability of the model when facing data anomalies.

[0076] Finally, explainability verification methods are carried out, i.e. physiological consistency test. First, subjective-objective alignment: calculate the Pearson correlation coefficient between latent variables and NASA-TLX scores: ,k∈{cognitive load, fatigue, alertness}, where is the standard deviation of NASA-TLX scores NASA k , reflecting the dispersion of subjective scores, is the subjective score obtained by the NASA-TLX scale, which is used to quantify the cognitive load, fatigue or alertness level of the driver. Neural mechanism verification: alertness state is verified by EEG microstate analysis (K-means clustering). Temporal-spatial correlation of alpha wave suppression. Driving scene decoupling visualization: t-SNE projection, dimensionality reduction of latent space to 2D, marking distribution clusters of different driving scenes (night highway, rainy city, etc.); SHAP value analysis: quantifying the contribution of each modality feature to state classification: . Wherein N represents a set of all input features, S represents a subset of N, and does not contain feature i. Second is the driving scene consistency verification, as shown in Figure 4 . Six typical scenes (long straight road, continuous curve, tunnel strong light, etc.) are constructed on the CARLA platform, and 50 subjects aged 26-40 years old with driving age of 3 years or more are recruited to collect cross-scene data; 10 test vehicles equipped with vehicle-mounted systems are used to collect a total of 10,000 kilometers of natural driving data.

[0077] The application adopts a three-stage method of "hierarchical architecture design-real-time optimization-fault tolerance verification" to construct a high-reliability embedded closed-loop safety prevention and control system integration, as shown in Figure 5 , infrared eye tracker or high-precision GNSS is used to collect data, and then the data is transmitted to the processor based on the data transmission protocol (RS-422), and then signal preprocessing FPGA and TCN accelerator IP core are performed, brain fatigue recognition is performed according to QNX Neutrino RTOS in PCle and dynamic priority task scheduling, and feedback is performed according to the recognition result, such as tactile feedback seat, fragrance / light environment control and automatic driving domain controller.

[0078] The specific process of the system design is as follows: in the design of the heterogeneous computing architecture, in terms of hardware platform selection, the main control unit adopts NVIDIA Jetson AGX Orin, which has 64GB RAM and 200 TOPS AI computing power; the real-time coprocessor selects TI TDA4VM, which has dual-core Cortex-R5F and lockstep core redundancy; the communication bus adopts CAN FD (rate up to 8Mbps) and Ethernet AVB (time-sensitive network).

[0079] Adopting the above method, the pupil diameter entropy time sequence collected in real time is directly related to the blue nucleus nerve activity, and can capture the fatigue signal caused by the accumulation of metabolic products;The gaze entropy time sequence reflects the stability of the eye movement strategy, and indirectly represents the attention allocation and cognitive load fluctuation, and by combining the time characteristics and the preset task stage label, a joint feature matrix is generated by feature splicing, effectively fusing the neural metabolism, behavior pattern and environmental context information, solving the limitation that a single signal is easy to be disturbed by noise, and significantly enhancing the robustness of the fatigue representation;Then, the deep layer time sequence dependence of the joint feature matrix is extracted to generate shared hidden features, and by generating the distribution parameters of the fatigue and vigilance states, the state values are determined, realizing the determination of the brain fatigue state through the pupil state, and the physiological interpretability of the fatigue state is decoupled;Finally, based on the fatigue state and the vigilance state, the driving scene is combined for early warning.

[0080] Secondly, the application also provides a brain fatigue dynamic early warning device, as shown in the figure, comprising: Figure 6 The acquisition module 201 is used for acquiring the pupil diameter entropy time sequence and the gaze entropy time sequence of the driver in real time;The time characteristics are determined based on the current time.

[0081] The preprocessing module 202 is used for splicing the pupil diameter entropy time sequence, the gaze entropy time sequence, the time characteristics and the preset task stage label to obtain a joint feature matrix.

[0082] The determination module 203 is used for performing time sequence feature extraction on the joint feature matrix to generate shared hidden features;The distribution parameters of the cognitive load, the fatigue and the vigilance state are generated from the shared hidden features, and then the state values of the fatigue state and the vigilance state are determined from the distribution parameters through reparameterization sampling.

[0083] The early warning module 204 is used for early warning according to the state values of the fatigue state and the vigilance state in combination with the driving scene.

[0084] The application also provides a computer readable storage medium, which stores a computer program, and the computer program can be used to execute the steps of the brain fatigue dynamic early warning method provided above. Figure 1 The application also provides a computer readable storage medium, which stores a computer program, and the computer program can be used to execute the steps of the brain fatigue dynamic early warning method provided above.

[0085] ​The application further provides a computer device, which comprises a processor, an internal bus, a network interface, a memory and a nonvolatile memory at a hardware level, and can further comprise other hardware required by a business. Figure 1 The steps of the brain fatigue dynamic early warning method are provided.

[0086] Those skilled in the art will understand that embodiments of the application can be provided as methods, systems or computer program products. Therefore, the application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Moreover, the application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0087] The application is described with reference to flowcharts and / or block diagrams of the method, device (system) and computer program product according to embodiments of the application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams can be realized by computer program instructions, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device that implements the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 an apparatus that performs the functions specified in one or more blocks or flows.

[0088] These computer program instructions can also be stored in a computer-readable memory that can direct the computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product comprising instruction apparatus, which implements the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 an apparatus that performs the functions specified in one or more blocks or flows.

[0089] These computer program instructions can also be loaded into a computer or other programmable data processing device, so that a series of operation steps are performed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide a process for implementing the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks ​ an apparatus that performs the functions specified in one or more blocks or flows.

[0090] It should be noted that the above detailed description of the specific embodiments of the present application is not intended to limit the present application in any way. Thus, while the present application has been described above with a certain degree of particularity, one skilled in the art will understand that the present application can be carried out by other techniques and modifications are possible without departing from the spirit and scope of the present application. Accordingly, all such modifications are intended to be included within the scope of the present application. No limitation is intended to the scope of the claims based on any embodiment illustrated in the figures.

Claims

1. A brain fatigue dynamic early warning method, characterized in that, The method comprises: real-time collection of a pupil diameter entropy time sequence and a gaze entropy time sequence of a driver; determination of a time feature based on a current time; acquisition of a preset task stage label corresponding to the current time; concatenation of the pupil diameter entropy time sequence, the gaze entropy time sequence, the time feature, and the preset task stage label to obtain a joint feature matrix; time sequence feature extraction on the joint feature matrix to generate shared hidden features; generation of distribution parameters of cognitive load, fatigue state, and alertness state from the shared hidden features, and then determination of state values of the fatigue state and the alertness state from the distribution parameters through reparameterization sampling; warning based on the state values of the fatigue state and the alertness state in combination with a driving scene.

2. The brain fatigue dynamic early warning method according to claim 1, characterized in that, Before the concatenation of the pupil diameter entropy time sequence, the gaze entropy time sequence, the time feature, and the preset task stage label, preprocessing is further performed, including: standardization processing of the pupil diameter entropy time sequence E(t) and the gaze entropy time sequence G(t) respectively, with a specific formula being: ； ； wherein, and are the sequence mean and standard deviation of the pupil diameter entropy time series, respectively, and are the sequence mean and standard deviation of the gaze entropy time series, respectively; Encoding time features h ( t ) Fourier basis of 24 hours period and , generating time encoded features , in particular formula ; wherein k is 1, 2nd harmonic, h t is a time characteristic;​ The preset task stage label is one-hot encoded to generate a task feature vector The specific formula is: ； wherein, is the number of task phases, v i is the i-th element in the one-hot encoding vector.

3. The brain fatigue dynamic early warning method according to claim 2, characterized in that, The concatenation of the pupil diameter entropy time sequence, the gaze entropy time sequence, the time feature, and the preset task stage label to obtain a joint feature matrix includes: The preprocessed and Spliced along the time dimension into a joint feature matrix, the specific formula is: ； wherein, represents a joint feature matrix at time t, shaped as , T is a time step, is a dimension of input features, is a correlation coefficient matrix symbol.

4. The brain fatigue dynamic early warning method according to claim 1, characterized in that, time sequence feature extraction on the joint feature matrix through a TCN network, the TCN network including a cavity convolution layer defined as: ； wherein, shared hidden features output by the dilated convolutional layers, for the input joint feature matrix, L the number of dilated convolutional layers.

5. The brain fatigue dynamic early warning method according to claim 1, characterized in that, After the determination of the state values of the fatigue state and the alertness state, the method further comprises: determination of a state value of cognitive load; reconstruction of a pupil diameter entropy signal and a gaze entropy signal respectively based on the state values of the cognitive load, the fatigue state, and the alertness state; a reconstruction formula of the pupil diameter entropy signal being: ； wherein, is a pupil diameter entropy signal generated for the reconstruction, are cognitive load, fatigue state and vigilance state, respectively, is a weight matrix for the pupil diameter entropy generation branch, is a bias term for the pupil diameter entropy generation branch. a reconstruction formula of the gaze entropy signal being: ； wherein, respectively a state value of cognitive load, fatigue state and vigilance state at time step t, is a task phase feature vector corresponding to time step t, is a hidden state of the LSTM network at time step t−1; is a reconstructed generated gaze entropy signal, is a weight matrix of the gaze entropy generation branch, is a bias term of the gaze entropy generation branch; According to and determining a pupil diameter entropy signal error, determining a gaze entropy signal error based on the pupil diameter entropy signal error and the gaze entropy signal error, validating a fatigue state and a vigilance state based on the pupil diameter entropy signal error and the gaze entropy signal error, and calculating a real-time risk index based on the validated fatigue state and vigilance state. and determining a pupil diameter entropy signal error, determining a gaze entropy signal error based on the pupil diameter entropy signal error and the gaze entropy signal error, validating a fatigue state and a vigilance state based on the pupil diameter entropy signal error and the gaze entropy signal error, and calculating a real-time risk index based on the validated fatigue state and vigilance state.

6. The brain fatigue dynamic early warning method according to claim 5, characterized in that, Before the reconstruction of the pupil diameter entropy signal and the gaze entropy signal, dynamic prior constraints are further performed on the cognitive load, the fatigue state, and the alertness state, including: cognitive load segmented stationary prior with drift Brownian motion prior alertness state periodic normal prior synchronize 24 hour biological clock preset fatigue state drift fatigue state fluctuation standard deviation 7. The brain fatigue dynamic early warning method according to any one of claim 6, characterized in that, The warning based on the state values of the fatigue state and the alertness state in combination with a driving scene is: generation of a risk index based on the state values of the fatigue state and the alertness state, with a formula being: ； wherein, is a state value of the fatigue state at the time t after verification, is a state value of the vigilance state at the time t after verification, and are variable parameters, respectively; warning based on the risk index and risk threshold values corresponding to different driving scenes.

8. A brain fatigue dynamic early warning device, characterized in that, The device comprises: an acquisition module configured to collect a pupil diameter entropy time sequence and a gaze entropy time sequence of a driver in real time; and determine a time feature based on a current time; a preprocessing module configured to concatenate the pupil diameter entropy time sequence, the gaze entropy time sequence, the time feature, and a preset task stage label to obtain a joint feature matrix; a determination module configured to perform time sequence feature extraction on the joint feature matrix to generate shared hidden features; generate distribution parameters of cognitive load, fatigue, and alertness state from the shared hidden features; and then determine state values of the fatigue state and the alertness state from the distribution parameters through reparameterization sampling; a warning module configured to warn based on the state values of the fatigue state and the alertness state in combination with a driving scene.

9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, and the computer program is executed by a processor to implement the method of any one of claims 1-7.

10. A computer device, comprising: Computer program product comprising a memory, a processor and a computer program stored on the memory and loadable into the processor, the processor realizing the method according to any one of the preceding claims 1 to 7 when executing the program.

Images