Driver state progressive determination method and system based on explicit state space model

Abstract

The application discloses a driver state gradual determination method and system based on an explicit state space model, belongs to the technical field of driver state monitoring, and solves the problems of describing continuous states in discrete categories and that adaptive thresholds stay in image feature layers. A space-time interaction representation is extracted from a face image sequence, a three-dimensional state vector of vigilance, fatigue and distraction is maintained, each component is non-negative and the sum is one; the current value is obtained by normalizing the sum of the state transition matrix and the input projection matrix transformation of the previous time value and the space-time interaction representation, respectively. Each of the fatigue and distraction is provided with a determination threshold, the prior threshold of the crowd is taken as a benchmark, and the determination threshold is corrected according to the statistics in the baseline collection period and the sliding window; the sum of the alarm multiples of the predicted standard deviations of the two is compared with the respective determination thresholds, and any one exceeding the determination threshold is alarmed; when neither of the two exceeds the determination threshold, a larger early warning multiple is taken to compare again, and if exceeding the early warning multiple, early warning is performed, otherwise, it is normal. The system manages the prompt state level according to state evolution.

Description

Technical Field

[0001] This invention belongs to the fields of intelligent driving, driver state monitoring and vehicle active safety technology. Specifically, it relates to a method and system for progressively determining driver state based on an explicit state space model. It is mainly applied to in-vehicle safety scenarios that require real-time assessment of driver alertness, fatigue and distraction, such as advanced driver assistance, intelligent cockpit monitoring and commercial vehicle safety supervision. Background Technology

[0002] With the widespread adoption of driver assistance features, driver monitoring systems have evolved from optional auxiliary functions into an essential component of active safety in intelligent vehicles. In assisted driving mode, drivers must regain attention to the road environment and take control of the vehicle within a short period after receiving a takeover request. Whether their fatigue and distraction can be accurately determined in real time directly affects the safety of the takeover process.

[0003] Current driver condition monitoring methods mostly rely on in-vehicle cameras to collect facial videos, and then deep learning models provide the judgment results. This approach categorizes the essentially continuously changing driver state into discrete categories such as alertness, fatigue, or distraction, using discrete labels to drive the classification model. Due to this limitation, the model primarily learns the transitions between categories, failing to adequately depict the gradual accumulation of fatigue over tens of seconds. Detection results are often only given when indicators like the eyelid closure time ratio become significantly abnormal; the output is a category label or category probability, rather than a continuous state quantity with clear physical meaning.

[0004] Existing adaptive thresholding methods mostly remain at the image feature level, adjusting thresholds for indicators such as eyelid aspect ratio and eyelid closure time ratio. The driver fatigue detection method and system disclosed in patent document CN118587689B belongs to this category. It calculates the pupil aspect ratio frame-by-frame during the initial period of the driving journey and takes the average as the adaptive threshold, then compares the average aspect ratio over subsequent time windows with this threshold to determine fatigue. In this type of method, the threshold and the compared quantity are both at the image feature level, failing to utilize the mean, standard deviation, and temporal evolution trend at the state level for personalized decision calibration for different drivers. When facing drivers with significant differences in facial anatomical features, it is difficult to maintain consistent judgment performance.

[0005] Existing state-space models are mostly used as high-dimensional sequence encoding tools, with high hidden state dimensionality, lacking clear meaning of driving behavior, and uninterpretable state transition paths. Traditional Kalman filtering, while providing clear state meaning, relies on pre-defined physical models and fixed state transition matrices, making it difficult to learn state evolution patterns end-to-end from driver video data. Existing systems typically do not continuously track the driver's response after issuing a warning, and cannot upgrade, downgrade, or cancel warnings based on subsequent states. State monitoring and vehicle warnings do not constitute a complete management process.

[0006] Therefore, it is necessary to provide a driver state determination method and system that can output continuous driver state quantities with clear physical meaning, possess uncertainty perception and individual adaptability at the state level, and continuously manage vehicle warning levels based on state evolution. Summary of the Invention

[0007] In view of this, the present invention provides a method and system for progressively determining the driver's state based on an explicit state-space model, in order to solve the four problems mentioned in the background art: characterizing continuous states with discrete categories, adaptive thresholds remaining at the image feature layer, latent states lacking clear meaning of driving behavior, and prompts not constituting a complete management process.

[0008] To achieve the above objectives, this invention provides a method for progressively determining driver state based on an explicit state-space model. It extracts spatiotemporal interaction representations from driver facial image sequences and maintains a three-dimensional state vector composed of alertness, fatigue, and distraction, along with the prediction standard deviations of each of the fatigue and distraction values. All three are non-negative continuous quantities and sum to one, respectively representing the driver's attentional input to driving tasks, attention deficit due to fatigue, and attention shift due to non-driving tasks. Fatigue is observable on the face as an increased proportion of eyelid closure time, distraction is observable on the face as fixed gaze or head deviating from the road direction, and alertness is the difference between one and the sum of the former two.

[0009] The current value of the three-dimensional state vector is determined by adding the previous value obtained through state transition matrix transformation and the spatiotemporal interaction representation obtained through input projection matrix transformation. This current value is obtained by normalizing the state prior or an intermediate result derived from the state prior, ensuring that all components are non-negative and their sum is one. In one implementation, this normalization process involves setting the negative components to zero and then normalizing by the sum of the components, ensuring that alertness, fatigue, and distraction are non-negative and their sum is still one. Fatigue and distraction each have a population prior threshold and a judgment threshold applied to the three-dimensional state vector. The judgment threshold is determined based on the population prior threshold for that component, the statistics of that component during the baseline acquisition period, and the statistics of that component within the current sliding window. Each of fatigue level and distraction level is referred to as the judgment component, and the sum of multiples of each judgment component and its prediction standard deviation is referred to as the judgment quantity. During judgment, the judgment quantity of each judgment component is compared with the judgment threshold of that judgment component. The multiple includes the alarm multiple and the warning multiple which is greater than the alarm multiple. When the multiple is the alarm multiple, if either exceeds its judgment threshold, an alarm is output. When the multiple is the alarm multiple, if neither exceeds the threshold, but when the multiple is the warning multiple, if either exceeds its judgment threshold, a warning is output. When the multiple is the warning multiple, if neither exceeds its judgment threshold, a normal operation is output.

[0010] The normalization process described above, which ensures that all components are non-negative and that the sum of all components is one, can be achieved by setting the negative components to zero and then normalizing by the sum of all components; alternatively, it can be achieved by first mapping each component to a non-negative interval and then normalizing by the sum of all components; or by using other mapping methods that ensure that all three components are non-negative and that the sum of all components is one. The statistical measure on which the above-mentioned threshold is determined is the sample statistical measure of the component's value within the corresponding interval: during the baseline collection period, the mean, median, or other statistical measures reflecting central tendency can be used; during the current sliding window, the mean, standard deviation, or other statistical measures reflecting central tendency and dispersion can be used. As one implementation method, the mean of the component during the baseline collection period, the mean of the component during the current sliding window, and the standard deviation of the component during the current sliding window are taken.

[0011] The above-mentioned judgment outputs of normal, warning, and alarm also correspond to three prompt status levels from low to high: no prompt, warning prompt, and alarm prompt. For the consecutive occurrence of the same output result, if an output result higher than the level corresponding to the current prompt status occurs consecutively for a set number of frames, the prompt status is raised to the level corresponding to that output result. If an output result lower than the level corresponding to the current prompt status occurs consecutively for a set number of frames, the prompt status is lowered or canceled, and the judgment threshold used for cancellation is lower than the judgment threshold used for raising the prompt status by a set hysteresis. If the square of the prediction standard deviation of fatigue level and distraction level, i.e., their total prediction variance, is higher than the uncertainty threshold within a set number of consecutive frames, the prompt status is prohibited from shifting from a high level to a low level, but is not restricted from shifting from a low level to a high level. The above-mentioned prompt status management can be implemented as a component step of the method of this invention, or it can be implemented by the vehicle interaction module described later.

[0012] The three components of the three-dimensional state vector are alertness, fatigue, and distraction, respectively, and are denoted as follows in the following formulas: , and The observable facial features are defined as follows: Fatigue corresponds to the degree of increased eyelid closure time accompanied by yawning, and is observed based on decreased eyelid aspect ratio, increased eyelid closure time, increased duration of continuous eye closure, periodic increase in mouth aspect ratio during yawning, and increased head roll angle due to head tilting to one side; Distraction corresponds to the degree of deviation of the gaze direction or head orientation from the road ahead, and is observed based on the continuous deviation of gaze yaw angle and gaze pitch angle from the road area, and the continuous deviation of head yaw angle and head pitch angle from the vehicle's forward baseline; Alertness corresponds to the driver's effective gaze share of the road ahead when the above two types of deviations are not significant, and its value is directly obtained by summing the three.

[0013] The constraint that the sum of the above three factors is one is based on the theory of limited capacity of attention in cognitive psychology. That is, driving-related attention resources have an upper limit. Attention loss caused by fatigue and attention shift caused by non-driving tasks both come at the cost of a reduction in the driver's effective attention input to driving tasks. Therefore, the three components should not be independently valued, but should be subject to the same total constraint.

[0014] The state prior is derived from the state vector of the previous time step via the state transition matrix. Transformation, and its spatiotemporal interaction representation at the current moment. After input projection matrix The result is obtained by adding the transformed components together: In the formula As a priori state, The value of the three-dimensional state vector at the previous moment.

[0015] The aforementioned state prior is the form of the state equation when random disturbances are neglected. Additionally, the right-hand side of the state equation includes a process noise term, which, together with the state transition term and the input projection term, constitutes the right-hand side of the state equation. This process noise term follows a Gaussian distribution with a mean of zero and a covariance matrix equal to the process noise covariance matrix. It is used to characterize random disturbances not included in the state equation, including sudden changes in illumination, facial occlusion, and state fluctuations caused by individual behavioral habits. Correspondingly, the observation noise in the observation equation described later follows a Gaussian distribution with a mean of zero and a covariance matrix equal to the observation noise covariance matrix.

[0016] The two noise terms mentioned above do not take specific values ​​in the frame-by-frame recursion. Their role is reflected in their respective covariance matrices: the process noise covariance matrix is ​​incorporated into the prior error covariance matrix through the prior error covariance formula described later, and the observation noise covariance matrix is ​​incorporated into the Kalman gain through the Kalman gain formula described later. The two together determine the weight of the state prior and the actual observation in the state correction.

[0017] In continuous recursion, the sum of the three state variables is prone to drift over time. If the state transition matrix only satisfies the condition that the sum of the elements in each row is one, then in the recursive form where the state vectors participate in matrix left multiplication as column vectors, the components of the new state are convex combinations of the original three components. The state variables do not diverge due to recursion, but the sum of the three components does not automatically remain unchanged. Therefore, the state transition matrix... The elements of the matrix are non-negative, the sum of the elements in each row is one, and the diagonal elements of each row are greater than any off-diagonal element of that row; the conservation of the sum of the three components is jointly guaranteed by the non-negative normalization described later and the soft constraint of attention resource conservation during the training phase. As one implementation method, the input projection matrix... The sum of the elements in each column is zero. At this point, the increment introduced by the spatiotemporal interaction representation cancels out each other among the three components, and the sum of the components before normalization remains unchanged.

[0018] The increment introduced by the input projection matrix and the Kalman correction term in the spatiotemporal interaction representation do not guarantee that the sum of the three components remains unchanged. Therefore, after each step of recursion, the negative components in the result are set to zero and normalized according to the sum of each component, so that the three components are non-negative and their sum is always one. During the training phase, the attention resource conservation soft constraint is used to punish the deviation of the sum of the three components from one, so that the model output tends to satisfy the constraint before normalization, and the amount of correction required for normalization is reduced accordingly.

[0019] The state transition matrix is ​​composed of the learnable parameter matrix. The parameterized result is obtained by applying a normalized exponential function with row-wise operation, i.e.: In the formula This represents a row-wise normalized exponential function, which performs a normalized exponential transformation on each row of the learnable parameter matrix, resulting in a matrix with non-negative elements and a sum of one element across all rows.

[0020] The learnable parameter matrix is ​​initialized with the diagonal elements having values ​​greater than the off-diagonal elements. As one implementation, the sum of twice the third-order identity matrix and a third-order square matrix with zero-one elements is taken as its initial value. At this time, the diagonal elements of the learnable parameter matrix are 2.1 and the off-diagonal elements are 0.1. Thus, the state transition matrix obtained at the beginning of training has diagonal elements of approximately 0.79 and off-diagonal elements of approximately 0.11 in each row, with the diagonal elements being significantly greater than any off-diagonal element in that row.

[0021] The normalized exponential function applied row-wise only guarantees that the matrix elements are non-negative and that the sum of the elements in each row is one. It does not guarantee that the diagonal elements will still be greater than the off-diagonal elements in the same row after training convergence. Therefore, an in-row diagonal dominance constraint is set throughout the training process to maintain this. As one implementation method, this constraint is applied by projection: after each parameter update, for each row of the learnable parameter matrix, the maximum value of the off-diagonal elements in that row is taken and the set margin is added. If the diagonal element of that row is less than the sum, it is set to the sum; otherwise, no change is made. When the set margin is 0.5, the ratio of the diagonal element of any row to any off-diagonal element in that row of the state transition matrix is ​​not less than the zeroth power of the natural constant, i.e., approximately 1.65, and the diagonal element is always not less than 0.45. Under the above initial values, the difference between the in-row diagonal elements and off-diagonal elements of the learnable parameter matrix is ​​two, which already satisfies the constraint. The projection does not take effect in the initial stage of training. When the margin is set too large, the state transition matrix approaches the third-order identity matrix, and the response of the state variables to the spatiotemporal interaction becomes slower; when the margin is set too small, the diagonal dominance is insufficient to resist the fluctuations caused by parameter updates. Therefore, its value allows for a balance between the continuity of the state trajectory and the response speed of the state variables.

[0022] The spatiotemporal interaction representation is obtained by fusing features extracted by the spatial encoder and the temporal encoder through cross-attention. The two encoders use independent parameter sets and do not share any network layers, obtained through a training process with orthogonal loss constraints. The orthogonal loss is the square of the Frobenius norm of the cross-covariance matrix of the spatial features and the temporal dynamic features, i.e.: In the formula For frame-by-frame spatial features, For temporal dynamic features, the cross-covariance matrix is ​​calculated across the sample dimension of the training batch. When this loss is minimized, the statistical correlation between the two types of representations decreases, and facial structural information and behavioral dynamic information are concentrated in their respective encoders.

[0023] The spatial encoder is a lightweight convolutional neural network containing three parallel output heads: a facial keypoint regression head, an eye region feature extraction head, and a head pose estimation head. Spatial features are formed by concatenating the keypoint features, eye features, and head pose features output by the three heads. The temporal encoder is a gated temporal convolutional network whose input includes the difference images of adjacent frames and keypoint velocity sequences to explicitly obtain facial motion information.

[0024] During fusion, the spatial features of the current frame are used as the query, and the historical temporal feature matrix composed of temporal dynamic features is used as the key and value. The attention result is obtained through scaled dot product attention, i.e.: In the formula For the first Attention results for each time window This is the historical time series feature matrix within this window. Let be the dimension of the key vector, and let its square root be the scaling factor. , , The learnable projection matrices are, in order, for queries, keys, and values.

[0025] The historical time series feature matrix is ​​derived from multiple time windows of varying lengths, including short, medium, and long time windows. The long time window is longer than the medium time window, and the medium time window is longer than the short time window. Each window independently calculates scaled dot product attention, with the long time window participating in the calculation after downsampling. The attention results for each window are then weighted and fused using adaptive weights output from the gating network to obtain a spatiotemporal interaction representation, namely: In the formula Number of time windows For the first The fusion weights for each window are determined, and the gating network takes the concatenation of the attention results of each window as input. From system startup until each time window is filled, the length of each window is replaced by the actual number of available historical frames. The state output can also be given during the cold start phase.

[0026] As one implementation method, the process of determining the current value of the three-dimensional state vector based on the state prior employs a recursive reasoning in the form of Kalman filtering. First, the state prior is obtained according to the aforementioned state prior formula. Then, the previous value of the posterior error covariance matrix is ​​transformed by the state transition matrix and added to the process noise covariance matrix to obtain the prior error covariance matrix, i.e.: In the formula Let be the prior error covariance matrix. The posterior error covariance matrix is ​​the value taken at the previous time step. Let T be the process noise covariance matrix, with the superscript T indicating matrix transpose.

[0027] The Kalman gain is then determined by the prior error covariance matrix, the observation projection matrix, and the observation noise covariance matrix, i.e.: In the formula For Kalman gain, To observe the projection matrix, To observe the noise covariance matrix, the superscript negative one indicates matrix inversion. The three observed components—gazing yaw angle, gaze pitch angle, and head roll angle—are made to increase monotonically with the increase of their corresponding state variables; the eyelid aspect ratio decreases monotonically with the increase of fatigue, and the mouth aspect ratio increases monotonically with the increase of fatigue. The signs of the corresponding elements in the observation projection matrix are determined accordingly.

[0028] In the formula For actual observations obtained from facial image sequences, To mitigate observation noise, the observation projection matrix maps the three-dimensional state vector into five observations: eyelid aspect ratio, mouth aspect ratio, gaze yaw angle, gaze pitch angle, and head roll angle. The gaze yaw angle, gaze pitch angle, and head roll angle are all taken as the absolute values ​​of their deviations relative to the road center direction or the vehicle's forward baseline. The actual observations are these five observations obtained from the facial image sequence. The difference between the actual and predicted observations is weighted using Kalman gain and then corrected for the state prior to obtain the corrected state, i.e.: The Kalman correction term neither guarantees that the components are non-negative nor that the sum of the three components remains unchanged. Therefore, after setting the negative components in the corrected state to zero and normalizing them according to the sum of their components, we obtain the current value of the three-dimensional state vector, i.e.: In the formula This represents the vector obtained by taking the larger of each element of the corrected state and zero. Let be a three-dimensional column vector with all elements equal to one. After this processing, alertness, fatigue, and distraction are non-negative, and their sum remains one.

[0029] Finally, the prior error covariance matrix is ​​corrected using the Kalman gain and the observation projection matrix to obtain the current value of the posterior error covariance matrix, which is used in the calculation of the prior error covariance matrix at the next time step. In the formula The value of the posterior error covariance matrix at the current time step. It is a third-order identity matrix. The state vector and error covariance matrix are updated at each time step and used in the next time step, and the recursion continues. The initial values ​​of the state transition matrix, input projection matrix, observation projection matrix, process noise covariance matrix, observation noise covariance matrix, and posterior error covariance matrix are all learned end-to-end from the training data, and the facial observable indices corresponding to each component of the three-dimensional state vector are also determined from the training data.

[0030] To provide advance warning before the state variable reaches the alarm condition, a forward prediction is performed on the current value of the three-dimensional state vector. This is first done by linearly extrapolating the difference between adjacent time steps from the spatiotemporal interaction representation, i.e.: In the formula For spatiotemporal interaction representation, starting from the current moment... extrapolated value of step, Take one to integers, The prediction step number is a set positive integer. Then, the current value of the 3D state vector is predicted forward using the power of the state transition matrix, and the accumulated value after transformation by the input projection matrix is ​​included in the forward prediction to obtain the intermediate prediction value, i.e.: In the formula For the first time since the current moment The intermediate quantity of the step prediction Take one to The integer. After setting the negative components in the predicted intermediate quantities to zero and normalizing them according to the sum of their components, we obtain the number of predictions from the current time. The predicted state vector of the step.

[0031] The sum of all elements in the state transition matrix is ​​one. Its power applied to the state vector results in a convex combination of the original three components. No component ever exceeds the largest of the three. When using only the power of the state transition matrix for zero-input prediction, the predicted fatigue value at higher positions monotonically does not increase, precisely when early warning is most needed and the threshold for judgment is not crossed. After incorporating the input term extrapolated from the spatiotemporal interaction representation, the predicted state vector continues to increase with the dynamic changes in facial features, providing a warning before the actual alarm condition is met.

[0032] During the determination, the sum of fatigue and distraction in the predicted state vector and the alarm multiple of the prediction standard deviation of the component at the current time is compared with the determination threshold of the component. If either of them causes the sum to exceed the determination threshold of the component at any step within the prediction steps, and the output at the current time is normal, the output at the current time is changed from normal to warning.

[0033] The predicted standard deviations of fatigue level and distraction level are obtained by taking the square root of the total predicted variance for that component. The total predicted variance is the sum of the inherent noise variance of the data for that component and the model uncertainty variance for that component, i.e.: In the formula Two and three are chosen respectively, corresponding to fatigue level and distraction level; For the first The predictive standard deviation of each component; The variance of the inherent noise of the data represents the variance caused by the imaging noise of the facial image sequence itself, the illumination changes, and the facial occlusion, and is directly predicted by the network output layer. The model uncertainty variance, representing the variance caused by the deviation of the facial image sequence from the training data distribution, is estimated by the variance of the forward propagation results of multiple random discardings of some neurons. In the formula The number of samples to be randomly discarded is indicated by the superscript number in parentheses, and the number with a horizontal line is the mean of the results of those samples.

[0034] During the baseline acquisition period, the decision threshold is taken as the prior threshold of the population for that component, and the mean of that component during this period is recorded as its individual baseline mean. The baseline acquisition period ends when the cumulative frame count reaches the set frame count. After the baseline acquisition period ends, the decision threshold is obtained by weighted summing of three factors: the prior threshold of the population for that component, the difference between the individual baseline mean of that component and the mean of that component within the current sliding window, and the standard deviation of that component within the current sliding window. In the formula This is the threshold for determining this component. This is the prior threshold for the population group in this component. This represents the individual baseline mean for this component. and These are the mean and standard deviation of the component within the current sliding window, respectively. , , The three weights are referred to as the first threshold weight, the second threshold weight, and the third threshold weight, respectively.

[0035] The second term reflects the driver's current state's deviation from their baseline. This term is negative when the mean within the sliding window is higher than the individual baseline, lowering the judgment threshold and improving judgment sensitivity. The third term increases with the standard deviation within the sliding window, correspondingly expanding the judgment margin when the driver is in a state transition period, thus suppressing false prompts caused by single-frame noise.

[0036] When making a judgment, the sum of the multiples of fatigue level, distraction level and their predicted standard deviation is compared with their judgment threshold. The multiples include the warning multiple and the alarm multiple, and the alarm multiple is less than the warning multiple.

[0037] If either of them satisfies, that is: In the formula If the alarm multiplier is specified, then an alarm will be output. When neither of them satisfies the above equation, but either one does, that is: In the formula If the warning multiplier is specified, then a warning will be issued. If neither of these conditions is met, the output is normal. The prediction standard deviation is non-negative and the alarm multiple is less than the warning multiple. When the alarm condition is met, the warning condition must also be met. The unique output level is obtained by judging in the order of alarm first and then warning. There is no gap in judgment between the three levels.

[0038] The spatial encoder, temporal encoder, state transition matrix, and input projection matrix are obtained through a two-stage training process. In the first stage, pseudo-continuous state labels are generated from the dataset containing discrete state labels through temporal smoothing interpolation. Each component is first smoothed using its own Gaussian smoothing kernel of appropriate width. In the formula Take one, two, and three respectively, corresponding to alertness, fatigue, and distraction. For the first Gaussian smoothing kernel for each component; Inter-frame distance; To control the first The smoothness width of the state transition for each component is a parameter, hereinafter referred to as the smoothness width. The smoothness width of the fatigue component corresponds to the physiological time constant of fatigue accumulation, while the smoothness width of the distraction component corresponds to the sudden time scale of a fixation or head-direction deviation from the road event, and the smoothness width of the distraction component is smaller than that of the fatigue component.

[0039] Then, a weighted average is applied to the one-hot encoding of the discrete state annotations within the smoothing window using a Gaussian smoothing kernel, i.e.: In the formula for The pseudo-continuous state label at time 1 The values ​​of each component before cross-component normalization To smooth out half the width of the window, The frame number to be summed. for Discrete state labeling at time step. The first one encoded by its unique heat There are three components. Then, normalization is performed among the three components so that the sum of the components of the pseudo-continuous state label is one, i.e.: In the formula The component index is used for summation. Normalization is performed in two steps: component-wise normalization uses the sum of the Gaussian smoothing kernels of each component within the window as the divisor, so that each component falls between zero and one while retaining its own smoothing width; since the widths of the smoothing kernels used for the three components are different, the sum of the components after component-wise normalization is generally not always one, so cross-component normalization is performed to make the sum of the components always one, consistent with the constraint that the sum of the three three-dimensional state vectors is one.

[0040] For datasets containing only fatigue binary classification labels, discrete state labels take the component corresponding to fatigue level when labeled as fatigued, and the component corresponding to alertness level when labeled as non-fatigue, with the component corresponding to distraction level always being zero. For datasets containing both fatigue and distraction labels, the three components are generated separately in the above manner. Therefore, a large amount of existing discrete label data can be directly used for training continuous state inference models without needing to re-label all video frames.

[0041] The second stage involves joint optimization of the total loss, namely: In the formula For the total loss, , , The weights of each loss term; For state estimation loss, we take the square of the L2 norm of the difference between the current value of the three-dimensional state vector and the pseudo-continuous state label, i.e.: In the formula The value of the three-dimensional state vector at the current moment. This represents the current value of the pseudo-continuous state label. The aforementioned orthogonal loss; For the state transition self-consistent loss, the values ​​of the pseudo-continuous state labels at adjacent time steps are constrained to be consistent with the changes predicted by the state transition matrix and the input projection matrix, i.e.: In the formula This represents the value of the pseudo-continuous state label at the previous moment. This loss applies to the pseudo-continuous state label and constrains whether the state transition matrix can reproduce the label trajectory.

[0042] To account for the uncertainty calibration loss, the prediction standard deviation is made to increase with the increase of the state estimation error, i.e.: In the formula, the subscripts for summation are two and three, corresponding to fatigue level and distraction level, respectively. This term provides a training signal for the inherent noise variance of the data.

[0043] As one implementation method, the total loss also includes a soft constraint term for attention resource conservation. This term is weighted with a fourth weight and included in the total loss. It is the square of the L2 norm of the difference between the sum of alertness, fatigue, and distraction and one. This term does not force the sum of the three components to be strictly equal to one, but instead imposes a penalty proportional to the square of the deviation when it deviates from one. This ensures that the model output tends to satisfy the constraint that the sum of the three components is one before the aforementioned non-negativity and normalization processes. As a result, the amount of correction required for normalization is reduced, and the physical meaning of the state variables is preserved during the training process.

[0044] This invention also provides a driver state progressive determination system based on an explicit state-space model, including an image acquisition and preprocessing module, a spatiotemporal feature extraction module, a progressive state transition module, an uncertainty perception output module, an adaptive decision-making module, and a vehicle interaction module. The composition relationship of each module is as follows: Figure 1 As shown.

[0045] The image acquisition and preprocessing module is used to acquire a sequence of driver facial images. As one implementation, this module also performs face detection, cropping, key point localization, affine correction, and size normalization on each frame to obtain a standardized facial image sequence.

[0046] The spatiotemporal feature extraction module is used to extract spatiotemporal interaction representations. It includes a spatial encoder and a temporal encoder with independent parameters, as well as a fusion unit that calculates the attention results of each time window and performs gating fusion.

[0047] The progressive state transition module is used to maintain a three-dimensional state vector consisting of alertness, fatigue, and distraction based on the spatiotemporal interaction representation. All three are non-negative continuous quantities and their sum is one. The module determines the current value of the three-dimensional state vector according to the aforementioned state prior formula and state correction formula. The elements of the state transition matrix are non-negative, the sum of the elements in each row is one, and the diagonal elements of each row are greater than any off-diagonal element of that row. The module also performs forward prediction according to the aforementioned extrapolation formula and prediction formula to obtain the predicted state vector for each step.

[0048] The uncertainty perception output module outputs the predicted standard deviations for fatigue level and distraction level, respectively. Each predicted standard deviation is obtained by taking the square root of the total predicted variance for that component, which is the sum of the inherent noise variance of the data and the model uncertainty variance of that component. The total predicted variance output by this module is also fed back to the asymptotic state transition module as the observation noise covariance for the next time step to determine the Kalman gain.

[0049] The adaptive decision-making module determines the decision thresholds for the three-dimensional state vector and makes graded decisions. During the baseline acquisition period, this module uses the prior population thresholds for fatigue and distraction as the decision thresholds for each component, and records the mean of the component during this period as its individual baseline mean. After the baseline acquisition period ends, the module determines the decision thresholds for each component according to the aforementioned weighted summation formula. The module outputs normal, warning, or alarm according to the aforementioned graded decision rules, and changes the current output from normal to warning when the predicted state vector fed by the progressive state transition module meets the aforementioned re-determination conditions.

[0050] The vehicle interaction module converts the normal, warning, or alarm output from the adaptive decision-making module into visual, audible, or takeover risk warnings, and saves the current warning status. The warning status levels, from lowest to highest, are no warning, warning, and alarm. Normal, warning, and alarm correspond to no warning, warning, and alarm respectively. This module, according to the aforementioned warning status management rules, raises, lowers, or cancels the warning status based on the number of consecutive frames of the same result. The threshold used for cancellation is one lower than the set hysteresis threshold used for triggering. If the total prediction variance is higher than the uncertainty threshold for a consecutive set number of frames, the warning status is prohibited from shifting from a higher level to a lower level, but shifting from a lower level to a higher level is not restricted.

[0051] The processing performed by each module corresponds one-to-one with the steps of the aforementioned method. The output of the image acquisition and preprocessing module enters the spatiotemporal feature extraction module, the output of the spatiotemporal feature extraction module enters the progressive state transition module and the uncertainty perception output module, the outputs of the progressive state transition module and the uncertainty perception output module jointly enter the adaptive decision module, the total prediction variance output by the uncertainty perception output module is fed back to the progressive state transition module as the observation noise covariance of the next time step; the output of the adaptive decision module enters the vehicle interaction module.

[0052] The present invention also provides a computer-readable storage medium on which a computer program stored is executed by a processor to implement the aforementioned driver state progressive determination method based on an explicit state-space model.

[0053] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention explicitly models the continuous evolution of driver states, outputting trajectories of alertness, fatigue, and distraction with clear physical meaning. It also uses forward prediction, incorporating extrapolated inputs, to identify risk growth trends in advance. The constraint that each diagonal element of a row in the state transition matrix is ​​greater than any off-diagonal element restricts state changes between adjacent time points, avoiding frequent jumps near category boundaries in discrete classification models. Each element of the matrix corresponds to the transition intensity between alertness, fatigue, and distraction. The state trajectory and transition path can be output step-by-step, making it suitable for driving safety analysis and accident retrospective analysis.

[0054] 2. The sum of the elements in each row of the state transition matrix is ​​one and the elements are non-negative. The correction result is normalized by taking the non-negative part and then normalizing by the sum of the components. During the training phase, the soft constraint of attention resource conservation is used to punish deviations of the sum of the three components from one. The sum of the three components does not drift with time throughout the recursion. The proportion relationship between alertness, fatigue and distraction can be compared throughout the entire process. An increase in fatigue or distraction will necessarily correspond to a decrease in alertness.

[0055] 3. The judgment threshold is adjusted based on the driver's individual baseline and the mean and standard deviation within the sliding window, with the judgment value also including a multiple of the predicted standard deviation. When there are large fluctuations during the state transition period, the judgment margin automatically increases, suppressing false prompts caused by single-frame noise in sudden situations such as gaze deviation; when the state continuously deviates from the individual baseline, the judgment threshold decreases, improving judgment sensitivity. The threshold input is taken from the statistics of the state layer rather than the original image features, and the judgment is based on the driver's deviation from their own baseline and recent fluctuations, thus reducing the difference in judgment performance between drivers with different facial anatomical features.

[0056] 4. The inherent noise variance and model uncertainty variance characterize the quality of the input image itself and the novelty of the input relative to the training data, respectively. Their sum constitutes the total prediction variance. The system uses this variance to identify situations such as sudden changes in lighting, facial occlusion, face detection offset, and insufficient model experience. It also reduces the weight of abnormal observations on state correction by increasing the observation noise covariance. The decision margin is automatically expanded by the sliding window standard deviation term in the decision threshold, thus reducing the impact of abnormal observations on the prompting results, allowing for reliable judgments even under the aforementioned conditions. If the total prediction variance is higher than the uncertainty threshold for a consecutive set number of frames, the system prohibits the prompting state from transitioning from a high level to a low level, but does not restrict its transition from a low level to a high level.

[0057] 5. The spatial encoder uses a lightweight convolutional neural network, which can reduce the frequency of estimation of model uncertainty variance. While maintaining continuous state reasoning and uncertainty perception capabilities, it reduces the computational burden. The solution can be deployed on vehicle computing platforms or edge intelligent terminals. Attached Figure Description

[0058] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0059] Figure 1 This is a block diagram of the driver state progressive determination system based on an explicit state-space model according to the present invention. Figure 2 This is the overall flowchart of the driver state progressive determination method based on an explicit state-space model according to the present invention; Figure 3 This is a schematic diagram illustrating the relationship between state variables, judgment thresholds, and prompt state transitions in this invention.

[0060] In the diagram: 1. Image acquisition and preprocessing module; 2. Spatiotemporal feature extraction module; 3. Progressive state transition module; 4. Uncertainty perception output module; 5. Adaptive decision-making module; 6. Vehicle interaction module; 7. Vehicle-mounted camera equipment; 8. Vehicle-mounted human-machine interaction device. Detailed Implementation

[0061] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of the invention. However, those skilled in the art will understand that the invention may be implemented in other embodiments without these specific details.

[0062] Example 1 This embodiment describes the driver condition monitoring process under conditions of gradual fatigue accumulation. An onboard camera is installed in front of the driver, with the imaging range covering the driver's face and upper body. The processor performs face detection, key point localization, affine correction, and size normalization on each video frame to obtain a standardized face image sequence. The data acquisition protocol used in this embodiment is shown in Table 1.

[0063] Table 1 Data Acquisition Protocol Twenty valid participants generated 120 original videos according to the above protocol. After checks for facial visibility, video integrity, and task completion, 112 valid videos were retained. Training, validation, and testing data were divided into driver subsets, with no video clips from the same driver simultaneously entering different subsets. The valid sample size and data partitioning are shown in Table 2.

[0064] Table 2 Effective Sample Size and Data Division During annotation, two annotators determine the start and end times of normal, fatigue, visual distraction, and manual distraction events respectively according to the unified annotation specifications; if the event categories of the two annotators are inconsistent or the boundary deviation exceeds 1 second, a third reviewer determines the final label by combining the complete video, task instruction log and KSS record.

[0065] During training, the discrete state labels in the training video are first smoothed component-by-component using the aforementioned Gaussian smoothing kernel. The smoothing width for the fatigue dimension is 150 frames to represent the gradual accumulation process, for the distraction dimension it is 30 frames to represent event-driven changes, and for the alertness dimension it is 90 frames to account for both types of transitions. The half-width of the smoothing window for each of the three dimensions is three times its respective smoothing width, i.e., 450 frames, 90 frames, and 270 frames. After component-by-component normalization, normalization is then performed between the three components. For datasets that do not contain distraction labels, the discrete state label is taken as the component corresponding to fatigue when labeled as fatigued, and as the component corresponding to alertness when labeled as non-fatigue. The component corresponding to distraction is always zero, and the remaining processing is the same as described above.

[0066] In the first stage, after generating the pseudo-continuous state labels, the spatial encoder, temporal encoder, and input projection matrix are pre-trained using these labels. The second stage jointly optimizes the state estimation loss, orthogonality loss, state transition self-consistency loss, uncertainty calibration loss, and attention resource conservation soft constraint. The state transition matrix is ​​obtained by parameterizing the learnable parameter matrix using a row-wise normalized exponential function. Its learnable parameter matrix is ​​initialized with the sum of twice the size of the 3rd-order identity matrix and a 3rd-order square matrix with all elements equal to one (with a margin of 0.5). After each parameter update, a diagonally dominant projection is performed row-wise. The process noise covariance matrix and observation noise covariance matrix are parameterized to ensure non-negative definiteness. After each parameter update, the input projection matrix is ​​subtracted column-wise from its mean to keep the sum of all column elements zero. The optimizer uses the AdamW adaptive moment estimation optimizer with decoupled weight decay, with an initial learning rate of 1×10⁻⁶. -4 And it is scheduled according to cosine annealing.

[0067] During system deployment, the trained spatial encoder, temporal encoder, state transition matrix, input projection matrix, observation projection matrix, and noise covariance parameters are loaded. Prior thresholds for the crowd, sliding window length, warning and alarm multipliers, consecutive occurrence frame count conditions, and uncertainty thresholds are set. The spatial encoder uses a lightweight convolutional neural network, MobileNetV3-small, with approximately 2.5 million parameters and three parallel output heads. The temporal encoder uses a four-layer gated temporal convolutional network with dilation factors of 1, 2, 4, and 8 respectively. Each layer contains two causal convolutions with a kernel length of 9 followed by residual connections, and the receptive field is 241 frames. The initial values ​​of the three-dimensional state vector are set as follows: alertness equals 1, fatigue and distraction are both equal to zero, and the sum of all components is 1. The current alert state is initialized to no alert. Baseline acquisition is then initiated, collecting the state variables from the first 300 frames and recording the individual baseline mean values ​​for fatigue and distraction. The video frame rate in this embodiment is 30 frames per second. The following time window length, smoothing width and continuous frame count are all in frames. 30 frames correspond to approximately 1 second.

[0068] After deployment, steps S1 to S7 are executed sequentially for each frame, with the order of the steps as follows: Figure 2 As shown.

[0069] Step S1: Acquire and preprocess facial image sequences. A multi-task cascaded convolutional neural network (MTCNN) is used to detect the driver's facial region, obtain bounding box coordinates, and then perform affine transformation to correct the facial region into a standardized 224×224 pixel image.

[0070] Step S2, Spatiotemporal Decoupling Feature Extraction. The standardized image is input into the spatial encoder. The facial keypoint regression head outputs the coordinates of 468 facial keypoints, which are then dimensionality-reduced to obtain 125-dimensional keypoint features. The eye region feature extraction head outputs 128-dimensional eye features, and the head pose estimation head outputs 3-dimensional head pose features. The three are concatenated to obtain 256-dimensional spatial features. Simultaneously, the difference images and keypoint velocity sequences of adjacent frames are calculated and input into the temporal encoder, which outputs 128-dimensional temporal dynamic features.

[0071] Step S3: Multi-scale spatiotemporal cross-attention fusion. Temporal feature caches are maintained for short time windows of 30 frames, medium time windows of 150 frames, and long time windows of 600 frames. In the long time window, features are downsampled to 120 features every 5 frames. Scaling dot product attention is calculated for the historical temporal features of each window using spatial features as the query, resulting in three attention results. These are then weighted and fused using adaptive weights output by a gating network to obtain a 128-dimensional spatiotemporal interaction representation. From system startup until each window is filled, the actual number of available historical frames replaces the window length.

[0072] Step S4, Asymptotic State Transition and State Correction. The spatiotemporal interaction representation is input into the asymptotic state transition module 3. First, the state prior is calculated according to the state prior formula. Then, the Kalman gain is calculated according to the prior error covariance formula and the Kalman gain formula. Subsequently, the corrected state is obtained according to the correction formula. The negative components in the corrected state are set to zero and normalized according to the sum of the components to obtain the alertness, fatigue, and distraction at the current moment. Finally, the posterior error covariance matrix is ​​updated according to the posterior error covariance formula. In this step, the asymptotic state transition module 3 also calculates the predicted state vector according to the extrapolation formula and the prediction formula, with a prediction step count of 30 steps. The observation noise covariance matrix is ​​taken as the sum of the training-obtained observation noise basis and the state layer prediction variance mapping term. The mapping term is obtained by multiplying the diagonal matrix with the total prediction variance of the three components at the previous time step as the diagonal element by the observation projection matrix and its transpose. The total prediction variance of the alertness component is taken as the sum of fatigue and distraction, and the value of the previous time step is taken to avoid the interdependence between the state correction and uncertainty estimation at the same time step. When the observation quality deteriorates, the mapping term increases, the Kalman gain decreases accordingly, and the effect of abnormal observations on state correction is suppressed.

[0073] Alternatively, the observation noise covariance matrix in this step can be directly taken as a diagonal matrix, with each diagonal element representing the total prediction variance of the state component corresponding to that observation component at the previous time step, and the off-diagonal elements set to zero. This eliminates the need for multiplying the observation projection matrix by its first and second transformations, as well as the superposition of the observation noise basis. Among these, the eyelid aspect ratio, mouth aspect ratio, and head roll angle correspond to fatigue, while the gaze yaw angle and gaze pitch angle correspond to distractibility. This approach is suitable for vehicle-mounted platforms with limited computing power.

[0074] Step S5, uncertainty-aware output. The inherent noise variance of the predicted data is obtained through the network output layer in each frame; a random drop-out forward propagation is performed every 15 frames, with 10 sampling times and a drop-out rate of 0.1. Each sampling uses the three-dimensional state vector output by that forward propagation as the statistical object, and the model uncertainty variance is estimated accordingly. The remaining frames use the most recent estimation result. This frequency reduction method reduces the inference computation from 10 times to about 1.7 times.

[0075] The random dropout forward propagation in this step, also known as Monte Carlo dropout, involves keeping the random dropout layer used during training active during the inference phase. Multiple forward propagations are performed on the same input, with each propagation dropping a different neuron. The variance of the resulting multiple three-dimensional state vectors is the model uncertainty variance. The number of samples and the dropout rate affect the stability of this variance estimation and the computational cost of inference.

[0076] Step S6: Determining the Threshold and Classifying the Judgment. After the baseline acquisition period, the judgment threshold is calculated using a weighted summation formula, where the prior threshold for the population is 0.50, the weight of the first threshold term is 1.00, the weight of the second threshold term is 0.30, the weight of the third threshold term is 0.25, and the sliding window length is 150 frames. During judgment, the warning multiple is set to 1, and the alarm multiple is set to zero. That is, if fatigue or distraction exceeds the judgment threshold, an alarm is output; if neither exceeds the judgment threshold, but the sum of either and its predicted standard deviation exceeds the judgment threshold, a warning is output; if neither exceeds the judgment threshold, a normal state is output. The predicted state vector obtained from forward prediction is compared with the judgment threshold using the same alarm multiple. If it is true in any step within 30 steps and the current output is normal, the current output is changed to a warning.

[0077] Step S7, Vehicle Prompt and Prompt Status Update. The vehicle interaction module 6 converts the judgment result into a visual prompt, an audible prompt, or a takeover risk prompt, and saves the current prompt status. The prompt status levels, from low to high, are no prompt, warning prompt, and alarm prompt. The transition relationship between the status quantity, judgment threshold, and prompt status is as follows: Figure 3 As shown, this module counts the number of consecutive frames of the same judgment result. A warning is triggered after 15 consecutive frames, an alarm after 30 consecutive frames, and the warning level is reduced or lifted after 60 consecutive low-level judgment results. The judgment threshold used for lifting the warning is 0.05 lower than the threshold used for triggering, to suppress repeated switching of the warning state near the threshold. When the total prediction variance of fatigue and distraction is higher than the uncertainty threshold of 0.010 within 30 consecutive frames, the warning state is prohibited from transitioning from a high level to a low level, but transitioning from a low level to a high level is not restricted.

[0078] Figure 3This illustrates the transition relationship between state variables, decision thresholds, and prompt states in this embodiment. Figure 3 The upper part uses time as the horizontal axis and state variables as the vertical axis: the solid line represents the trajectory of fatigue degree over time, the band enclosed by the two dashed lines above and below it represents the range of positive and negative one-times the predicted standard deviation of fatigue degree, the horizontal dashed line represents the judgment threshold for the driver, and the dotted line represents the forward prediction trajectory obtained from the extrapolation and prediction formulas starting from the current moment. At the vertical dashed line at the current moment, the fatigue degree itself and its sum with one-times the predicted standard deviation do not exceed the judgment threshold, and according to the direct judgment in step S6, it should output "normal"; however, the forward prediction trajectory crosses the judgment threshold within the prediction steps, and this crossing point is... Figure 3 The predicted trigger time is marked in the image. Therefore, the output will be changed from normal to warning at the current time. The advance time is the difference between the crossing point and the current time. Afterwards, the fatigue level itself will trigger the crossing threshold. Figure 3 The alarm time marked in the figure indicates when the alarm condition is met. Figure 3 The lower half, using the horizontal axis as the same timeline, shows the transition paths between three alert states: no alert, warning alert, and alarm alert. When output results at a level higher than the current alert state occur consecutively for a set number of upgrade frames, the alert state is upgraded along the upgrade path. When output results at a level lower than the current alert state occur consecutively for a set number of downgrade frames, the alert state is downgraded along the downgrade path, i.e., from alarm alert to warning alert, or from warning alert to no alert (i.e., alert is deactivated). Deactivation is based on a threshold minus a set hysteresis value. When the level corresponding to the output result is two levels higher than the current alert state, the alert state is directly upgraded from no alert to alarm alert without a warning alert. Figure 3 The text refers to a sudden, excessive upgrade. When the total prediction variance for both fatigue and distraction exceeds the uncertainty threshold for 30 consecutive frames, [the text abruptly ends here]. Figure 3 In the lower half, transfers from higher to lower levels are prohibited, while transfers from lower to higher levels are unrestricted.

[0079] The above values ​​are selected to correspond to the time scale of facial behavior. A single blink of the human eye lasts about 0.1 to 0.4 seconds, and a short time window of 30 frames can cover several complete blinks; a single yawn lasts several seconds, and a medium time window of 150 frames can encompass a typical yawn event; a long time window is used to evaluate the trend of eyelid closure time ratio, and after downsampling, it takes into account both the sufficiency of trend detection and the computational load. The prior threshold for the population is set to 0.50, which serves as the boundary between normal and abnormal states in the three-dimensional state space. It is obtained by: statistically analyzing the fatigue level of all subjects at the start of each fatigue event frame by frame on the training set, taking the median as the initial value; then searching around this initial value on the validation set with a false alarm rate not exceeding twice per hour and a step size of 0.01, selecting the value that results in the highest F1 score. The prior threshold for distractibility is obtained in the same way based on the distractibility at the start of each distraction event; in this embodiment, both are 0.50. When the population composition changes, this threshold should be recalculated on the new population data in the above manner. For older subjects or those with a higher proportion of nighttime driving, the fatigue level distribution shifts upwards, and this threshold should be adjusted accordingly. The values ​​are adjusted accordingly. When the sample size is insufficient for re-statistical analysis, the values ​​from this embodiment can be reused, and the weight of the second threshold term can be appropriately increased to allow the individual baseline to play a greater calibration role. The weight of the first threshold term is set to 1.00, so that the judgment threshold is based on the prior threshold of the population. The second and third terms are modified to a limited extent on this basis. The weight of the second term is 0.30, so that about one-third of the baseline deviations are fed back to the threshold adjustment. During the transition period of gradual fatigue accumulation, the standard deviation within the sliding window is about 0.03 to 0.05. The weight of the third term is 0.25, which provides a threshold increase of about 0.008 to 0.013. During the transition period of sudden gaze deviation, the standard deviation is larger, and the threshold increase provided is increased accordingly. The half-width of the smoothing window is three times the smoothing width, so that the value of the Gaussian kernel at the edge of the window is negligible. The execution interval of random discarding is much smaller than the time constant of fatigue accumulation, so the timeliness of the estimation is not affected. The weights of the three loss terms are set to 0.01, 0.10 and 0.05, respectively, so that each constraint term does not dominate the total loss. The parameter values ​​of this embodiment are shown in Table 3.

[0080] Table 3 Key Parameter Values The observation vector consists of five parameters: eyelid aspect ratio, mouth aspect ratio, gaze yaw angle, gaze pitch angle, and head roll angle. The eyelid closure time ratio, continuous eye closure duration, and head yaw and pitch angles are derived from the above observations and keypoint coordinates and are not included in the observation vector. They are used for event discrimination in the annotation process, manual verification of implementation results, and conditional input to the gating network described in Example 2. Therefore, observable indicators do not directly determine vehicle prompts but first act on three-dimensional state quantities with clear physical meaning, and then the state layer determines the vehicle action. The correspondence between each step is shown in Table 4.

[0081] Table 4. Correspondence between state variables, observable indicators, and vehicle actions In a scenario where fatigue gradually accumulates from normal driving, the performance of this embodiment and the two comparative examples is shown in Table 5. Comparative Example 1 uses a fixed threshold method for the aspect ratios of the eyelids and mouth, while Comparative Example 2 uses a temporal classification model combining a convolutional neural network and a long short-term memory network. The advance warning time is calculated based on the moment when each method first outputs a warning of any level, with that moment in Comparative Example 1 as the zero point.

[0082] The definitions and statistical methods of the above evaluation indicators are as follows. The F1 score is a frame-level indicator: the output is determined by comparing each frame with the manually labeled target event interval as the true value. If the output is a warning or alarm and the frame falls within the labeled event interval, it is considered a true positive. If the output is a warning or alarm but the frame does not fall within any labeled event interval, it is considered a false positive. If the output is normal and the frame falls within the labeled event interval, it is considered a false negative. Precision and recall are calculated accordingly, and the F1 score is the harmonic mean of the two. The false alarm rate is a frequency-based indicator: the number of times each method triggers a warning or alarm level prompt within a time period in the test set that is not marked as a fatigue event or distraction event, divided by the total duration of that time period, in times per hour; for this embodiment and Comparative Examples 5 and 6, which have prompt status management, one trigger is counted as one transition from no prompt to a warning or alarm prompt; for Comparative Examples 1 to 4, which do not have prompt status management, one trigger is counted as one change in their judgment output from normal to a warning or alarm; the statistical range of this indicator does not overlap with the positive example interval of the F1 score. The smoothness of the state trajectory is taken as the mean of the absolute value of the difference between two moments 30 frames apart (approximately 1 second) on the test set. The smaller the value, the smoother the change of the state trajectory over time. For comparative examples 1 to 4, which do not output the three-dimensional state vector, the indicators used for judgment are normalized to the zero-to-one interval and calculated in the same way. For comparative examples 1 and 3, the mean of the sliding window of the eyelid aspect ratio is taken, and for comparative examples 2 and 4, the fatigue category probability is taken. The test set contains 3 subjects, 17 videos, and a total duration of 2.9 hours, totaling approximately 310,000 frames according to the frame rate in Table 2. All methods are evaluated under the same test set, the same test time axis, and the same event definition. All indicators are single statistical results on the test set.

[0083] Table 5. Judgment performance under the scenario of gradual fatigue accumulation. Comparative Example 1 triggered the first alarm when the proportion of eyelid closure time exceeded 0.15, at approximately 48 seconds, with no prior indication. Comparative Example 2 triggered the first alarm near the discrete category switching boundary, at approximately 44.5 seconds, about 3.5 seconds earlier than Comparative Example 1, still indicating that it was detected only after fatigue had significantly changed the category distribution.

[0084] The fatigue trajectory output in this embodiment shows that the fatigue level increases monotonically from about the 33rd second, from 0.12 to 0.28. At this time, the eyelid closure time ratio is only about 0.08, which is far below the fixed threshold of 0.15. At about the 43.6th second, the fatigue level is about 0.40, and its prediction standard deviation is about 0.08. The individual baseline mean of the driver's fatigue level is 0.12. At this moment, the mean of fatigue level within the current sliding window is 0.34, and the standard deviation is 0.04. According to the weighted summation formula, the first term of the judgment threshold is the product of the prior threshold of the population (0.50) and the weight of the first term of the threshold (1.00), which is 0.50. The second term is the product of the difference between the individual baseline mean and the sliding window mean and the weight of the second term of the threshold (0.30), which is -0.066. The third term is the product of the standard deviation of the sliding window and the weight of the third term of the threshold (0.25), which is 0.010. The sum of the three terms gives the judgment threshold of the driver to be approximately 0.44. The sum of fatigue level and one predictive standard deviation is 0.48, which exceeds the judgment threshold, triggering the first warning, approximately 4.4 seconds earlier than Comparative Example 1. In the frames prior to this, the sum of fatigue level and one predictive standard deviation did not exceed the judgment threshold at that time. The smoothness of the state trajectory in this embodiment is 0.062, meaning that the average change of the state quantity between two moments approximately 1 second apart is about 0.062, which translates to an average change of about 0.002 between adjacent frames. This is better than 0.128 in Comparative Example 1 and 0.085 in Comparative Example 2. The state trajectory generated by continuous state space modeling and Kalman-style recursive reasoning changes gradually rather than jumps. The comparison results for gaze deviation and cross-individual consistency are shown in Examples 2 and 3, respectively.

[0085] In the scenario of gradual fatigue accumulation, the complete comparison results between this embodiment and each comparative example are shown in Table 6. Comparative examples 1 to 4 are existing technical solutions, among which comparative example 3 corresponds to the image feature layer adaptive thresholding scheme described in the background art; comparative examples 5 and 6 are obtained by removing one constraint respectively based on this embodiment. Each comparative example uses the same face cropping results, task labels, test timeline, and event definitions as this embodiment.

[0086] Table 6 Performance comparison of this embodiment and other comparative examples under the scenario of gradual fatigue accumulation. Table 5 is an excerpt from Comparative Examples 1 and 2 in Table 6 and three rows of this embodiment. Both are taken from the same test set, the same test time axis, and the same event definition, and the statistical methods are the same. After removing the row-wise normalization constraint of the state transition matrix, the F1 score decreased from 0.92 to 0.89, and the smoothness of the state trajectory increased from 0.062 to 0.067. Applying structural constraints to the state transition matrix has a direct effect on the continuity of the state trajectory. After removing the self-consistent loss of the state transition, the F1 score decreased from 0.92 to 0.91, and the false alarm rate increased from 1.8 times per hour to 4.0 times per hour. This loss constrains the state transition matrix to reproduce the changes of pseudo-continuous state labels in adjacent time steps. After removal, the reciprocal crossing of state variables near the decision threshold increases, and the false alarms increase accordingly. This loss is effective in constraining the state evolution law. The false alarm rate of this embodiment is lower than that of Comparative Examples 1 to 4 because the judgment threshold is adjusted frame by frame, the judgment value includes a multiple of the prediction standard deviation, and the prompt status must be updated only after continuous frame counting. Although Comparative Examples 5 and 6 retain the above mechanisms, their status trajectories traverse back and forth near the judgment threshold due to the lack of corresponding constraints. This back and forth is continuous rather than single-frame, and continuous frame counting is insufficient to filter them out. Therefore, their false alarm rate is still higher than that of this embodiment. It should be noted that the F1 score is a frame-level indicator while the false alarm rate is a frequency-level indicator. The two have different statistical granularities. A multiple change in the false alarm rate corresponds to a change of one percent in the F1 score. There is no contradiction between the two.

[0087] Example 2 This embodiment describes the driver state monitoring process under sudden and recovery situations of gaze deviation from the road. Its acquisition protocol, system deployment and frame-by-frame processing flow are the same as those in Embodiment 1. The model parameters are also the same as those in Embodiment 1 except for the third adjustment described below. The following only describes the differences, focusing on the rapid response and recovery characteristics of the distraction dimension, as well as the change of the prediction standard deviation during the sudden event.

[0088] Unlike the slow changes in fatigue, events involving gaze deviation or head turning away from the road are sudden; a driver's gaze may shift dramatically from looking ahead to a point of significant deviation within a second. The pseudo-continuous state label smoothing width for the distraction component remains at 30 frames as described in Example 1, significantly smaller than the 150 frames for the fatigue component, allowing distraction to respond quickly to deviation events. Based on this, this embodiment makes three adjustments compared to Embodiment 1: First, as a variation, when the absolute value of the head yaw angle exceeds 20 degrees or the absolute value of the head pitch angle exceeds 15 degrees in a short time window, its weight in the gating fusion is automatically increased. In addition to splicing the attention results of each time window, the gating network uses the head attitude angle as a conditional input. When the absolute value of the head attitude angle exceeds the above threshold, it is regarded as entering a deviation event. Second, the estimation of the model uncertainty variance is increased from once every 15 frames to once every 5 frames during the deviation event, so as to more accurately track the uncertainty in the rapid change process. Third, the initial value of the element corresponding to the shift from distraction to alertness in the learnable parameter matrix is ​​set to 0.5, and the initial value of the element corresponding to the shift from alertness to fatigue is set to 0.05.

[0089] The speed of the recovery process depends on the intensity of the distractibility component's transition to itself and the intensity of its transition to the alertness component in the state transition matrix. The parameter differences between the fatigue dimension and the distractibility dimension are shown in Table 7.

[0090] Table 7. Parameter differences between fatigue and distraction dimensions. In a scenario where a gaze deviation event of approximately 5 seconds occurs at the 20th second, the judgment performance of this embodiment and two comparative examples is shown in Table 8, where the detection delay is calculated from the start time of the deviation event, i.e., the 20th second.

[0091] Table 8. Judgment performance in gaze deviation and recovery scenarios. Comparative Example 1 failed to detect the deviation in this scenario. When the driver looks at the handheld device, their eyes are usually open, and the eyelid aspect ratio is within the normal range. Therefore, the detection delay in Table 8 is recorded as undetected, indicating that the method of judgment based solely on eye features is not applicable in this situation. Comparative Example 2 was able to detect the deviation. Its temporal classification model learned the temporal pattern of gaze deviation from the head posture sequence, but there was a detection delay of approximately 2.2 seconds; after the deviation ended, the distraction level took approximately 4.5 seconds to return to normal.

[0092] In this embodiment, an alert is triggered approximately 1.2 seconds after the deviation begins, specifically at the 21.2-second mark. The transition strength from distraction to alertness in the state transition matrix is ​​an explicit parameter. Starting at the 25-second mark after the deviation event ends, the distraction level drops below 0.2 within approximately 1.8 seconds. The predicted standard deviation of the distraction level increases from approximately 0.05 to approximately 0.12 during the deviation period, reflecting a decrease in the model's confidence in the judgment results for this period. Consequently, the alert conditions are met earlier during this period. The alarm condition, however, is not affected by the predicted standard deviation and is still based on whether the distraction level itself is close to the judgment threshold, thus avoiding premature escalation to the alarm level at the edge of the state.

[0093] Example 3 This embodiment describes the individualized calibration process for the judgment threshold of a driver group with significant differences in facial anatomical features. The difference between this embodiment and Embodiment 1 lies in the individualized process of the driver group and the judgment threshold. The other acquisition protocols, model parameters, and frame-by-frame processing procedures are the same.

[0094] Three groups of drivers with representative eye shape differences were selected. Group A consisted of drivers with smaller palpebral fissures, whose baseline eyelid aspect ratio was approximately 0.18 ± 0.02, lower than the general fixed threshold of 0.20. Group B consisted of drivers with common eye shapes, whose baseline eyelid aspect ratio was approximately 0.25 ± 0.02, close to the population mean. Group C consisted of drivers with larger palpebral fissures, whose baseline eyelid aspect ratio was approximately 0.32 ± 0.02, higher than the general fixed threshold of 0.20. The first 300 frames of each driver's normal driving state were used for baseline acquisition. During this period, the system used the population prior threshold of 0.50 as the judgment threshold. After acquisition, the average fatigue level of each driver during this period was taken as their individual baseline mean, and then the judgment threshold was calculated using a weighted summation formula.

[0095] Group A's eyelid aspect ratio baseline was lower than the population mean. Using a unified population-wide judgment standard, determined by the population distribution in the training set, individuals with baselines below the population mean had their normal blinks counted as closed-eye statistics. This resulted in a systematically higher fatigue estimate for this group, leading to frequent false alarms. For Group C, with baselines above the population mean, the same judgment standard was conservative, resulting in fewer false alarms but reduced detection timeliness. This embodiment recorded the individual baseline fatigue mean of drivers in this group during the baseline acquisition period. Subsequently, the judgment threshold was adjusted frame-by-frame based on the difference between this mean and the mean within the current sliding window, as well as the window standard deviation. The second term of the judgment threshold was zero when the driver was at their own baseline, at which point the threshold returned to near the population's prior threshold. Therefore, individual differences were significantly suppressed rather than completely eliminated, as reflected in Table 9, where Group A's false alarm rate was still slightly higher than Groups B and C. Group C had larger palpebral fissures, and the change in eyelid aspect ratio from 0.32 to 0.20 was discernible, making it a stronger indicator of fatigue, thus requiring no overly sensitive judgment threshold.

[0096] The performance of the three groups of drivers is shown in Table 9. The statistical range is the representative drivers of the three groups of eye shapes mentioned above. Group B is a common eye shape, and its value is consistent with the statistical results in Tables 5 and 6.

[0097] Table 9. Performance Assessment of the Three Groups of Drivers As shown in Table 9, this embodiment reduces the cross-individual standard deviation of the F1 score from 0.065 in Comparative Example 2 to 0.021, and reduces the cross-individual standard deviation of the false alarm rate from 2.21 to 0.47. After baseline acquisition and sliding window updates, the judgment thresholds for the three groups of drivers all fall within a limited range near the population prior threshold of 0.50. Individualized calibration is a correction based on the population prior, rather than resetting the threshold.

[0098] Example 4 This embodiment describes a specific implementation of the aforementioned driver state progressive determination system based on an explicit state-space model. The composition relationship of each module is as follows: Figure 1 As shown. Except for the hardware implementation, internal module structure, inter-module interfaces and deployment methods described below, the network structure, parameter values ​​and frame-by-frame processing flow of this embodiment are the same as those of Embodiment 1.

[0099] The system in this embodiment is hosted on an in-vehicle domain controller, which includes an automotive-grade system-on-a-chip (SoC) and non-volatile memory. The SoC includes a general-purpose processor core and a neural network inference acceleration unit, whose computing power meets the real-time inference requirements of the aforementioned spatial encoder, temporal encoder, and fusion unit at 30 frames per second. The non-volatile memory stores the trained network weights, the initial values ​​of the aforementioned matrices, and the deployment parameters listed in Table 3. After power-on, the system loads the aforementioned weights and parameters from the non-volatile memory, then completes initialization as described in the steps and enters the baseline acquisition period.

[0100] The vehicle-mounted camera device 7 serves as the signal source for the image acquisition and preprocessing module 1. It is mounted on the upper edge of the dashboard, with the lens facing the driver's face at a distance of approximately 0.75 meters. The tilt angle is approximately 5 degrees, and it outputs RGB video at 30 frames per second and 640×480 pixels. Its light source includes an 850nm infrared illumination unit, which activates under low-light conditions. The vehicle-mounted camera device sends video frames to the domain controller via a serial interface or an Ethernet interface.

[0101] The in-vehicle human-machine interface device 8 serves as the execution end of the vehicle interaction module 6, including the instrument panel display area, in-vehicle speakers, and vibrators on the steering wheel or seat. The prompts output by the vehicle interaction module 6 are transmitted to the in-vehicle human-machine interface device via the in-vehicle bus. This device then acts on the driver in the form of visual prompts, audible prompts, and takeover risk warnings. The driver's subsequent responses are then processed by the in-vehicle camera equipment and input into the image acquisition and preprocessing module 1, constituting... Figure 1 The signal path shown is from image acquisition, through status determination, to vehicle prompt, and then from driver response back to image acquisition.

[0102] The image acquisition and preprocessing module 1 consists of the image signal processing unit of the on-chip system and a preprocessing program running on a general-purpose processor core. This module receives video frames sent by the vehicle-mounted camera device frame by frame, and outputs a standardized 224×224 pixel face image sequence after preprocessing as described in step S1. One path of this sequence is sent to the spatiotemporal feature extraction module 2, and the other path is used to calculate the difference image and key point velocity sequence of adjacent frames and send them to this module.

[0103] The spatiotemporal feature extraction module 2 runs within the neural network inference acceleration unit and includes a spatial encoder, a temporal encoder, a fusion unit, and three circular buffers. The spatial encoder outputs 256-dimensional spatial features, and the temporal encoder outputs 128-dimensional temporal dynamic features. The three circular buffers respectively store temporal dynamic features for a short time window of 30 frames, a medium time window of 150 frames, and a long time window of 600 frames. The buffer corresponding to the long time window is written to every 5 frames and has a capacity of 120 features. The fusion unit calculates and gates the attention results of each window as described in step S3, and outputs a 128-dimensional spatiotemporal interaction representation to the progressive state transition module 3 and the uncertainty perception output module 4.

[0104] The asymptotic state transition module 3 includes a parameter storage area and a recursive operation unit. The parameter storage area stores the state transition matrix, input projection matrix, observation projection matrix, process noise covariance matrix, observation noise basis, and posterior error covariance matrix. The recursive operation unit sequentially performs the operations of state prior, prior error covariance, Kalman gain, and state correction as described in step S4, and updates the posterior error covariance matrix. It also obtains the predicted state vector for each step within the prediction step number using extrapolation and prediction formulas. The input of this module is the spatiotemporal interaction representation output by the spatiotemporal feature extraction module 2, and the total prediction variance of the previous moment fed back by the uncertainty perception output module 4. The latter is mapped by the observation projection matrix and added to the observation noise basis to form the observation noise covariance matrix used at the current moment. The output of this module is a three-dimensional state vector and a predicted state vector, both of which are sent to the adaptive decision module 5.

[0105] The uncertainty perception output module 4 includes a data inherent noise variance output layer and a model uncertainty estimation unit. The data inherent noise variance output layer shares front-end features with the spatial encoder and temporal encoder, outputting the data inherent noise variance of fatigue and distraction levels frame by frame. The model uncertainty estimation unit is triggered every 15 frames, performing 10 forward propagations by randomly discarding some neurons. During gaze deviation events, the trigger interval of this unit is increased to once every 5 frames. This module uses the sum of the two types of variances as the total prediction variance, sends the square root of the total prediction variance to the adaptive decision module 5, and sends the total prediction variance back to the asymptotic state transition module 3 as the observation noise covariance for the next time step.

[0106] The adaptive decision-making module 5 includes a baseline statistics unit, a sliding window statistics unit, and a hierarchical judgment unit. The baseline statistics unit accumulates fatigue and distraction values ​​during the baseline acquisition period, ending the period when the accumulated frame count reaches 300 frames and outputting the individual baseline mean. The sliding window statistics unit updates the mean and standard deviation of each component online with a window length of 150 frames. This is achieved by saving the sum and square of the values ​​within the window and updating by adding and removing values ​​frame by frame, without needing to save the entire window's historical values. The hierarchical judgment unit determines the judgment threshold for each component using a weighted summation formula and performs hierarchical judgment and reclassification as described in step S6. Simultaneously with outputting the judgment results, this module sends the total prediction variance of fatigue and distraction to the vehicle interaction module 6. The total prediction variance is the square of the prediction standard deviation sent by the uncertainty perception output module 4.

[0107] The vehicle interaction module 6 includes a prompt status register, a continuous occurrence frame counter, an uncertainty gating unit, and a prompt instruction generation unit. The prompt status register stores one of three levels: no prompt, warning prompt, and alarm prompt. The continuous occurrence frame counter counts the same result output by the adaptive decision module 5; the counter is reset when the result changes. The prompt instruction generation unit updates the prompt status register according to the triggering, downgrading, and deactivation rules described in step S7. The uncertainty gating unit blocks the transition of the prompt status from a high level to a low level, but does not block the transition from a low level to a high level, when the total prediction variance of fatigue and distraction is higher than the uncertainty threshold of 0.010 for 30 consecutive frames. The output of the prompt instruction generation unit is sent to the in-vehicle human-machine interaction device via the in-vehicle bus.

[0108] The execution cycles of the above modules are not entirely the same. The image acquisition and preprocessing module 1, spatiotemporal feature extraction module 2, progressive state transition module 3, adaptive decision-making module 5, and vehicle interaction module 6 are executed frame-by-frame according to the video frame rate; the data inherent noise variance output layer of the uncertainty perception output module 4 is executed frame-by-frame, and its model uncertainty estimation unit is executed at the aforementioned interval with reduced frequency. In this embodiment, the video frame rate is 30 frames per second, and the frame interval is approximately 33 milliseconds. The real-time requirement is met if the sum of the processing time of each module within a frame does not exceed this frame interval. All the above modules can be executed on a pipeline according to... Figure 2 The steps shown are executed in sequence.

[0109] The above six modules are logical units divided according to function, and this division does not constitute a limitation on physical boundaries. In one implementation, all six modules are implemented as software modules on the same on-chip system; in another implementation, image acquisition and preprocessing module 1 and spatiotemporal feature extraction module 2 are deployed in the intelligent front-end within the camera module, while the remaining modules are deployed in the domain controller. The two transmit 128-dimensional spatiotemporal interactive representations rather than raw video frames, which can reduce the bandwidth usage of the in-vehicle bus; as long as the modules communicate according to... Figure 1 The relationships shown convey facial image sequences, spatiotemporal interaction representations, state variables, prediction standard deviation, total prediction variance, and prompts. The above-mentioned merging, splitting, and distributed deployment do not change the essence of the technical solution of this invention.

[0110] The above modules can be implemented by a processor executing a computer program stored in memory, or by an application-specific integrated circuit, a field-programmable gate array, or other hardware circuits with corresponding computing capabilities, or by a processor and hardware circuits working together.

[0111] The above are merely preferred embodiments of the present invention and are not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for progressively determining the state of a driver based on an explicit state-space model, characterized in that, include: Acquire driver facial image sequences and extract spatiotemporal interaction representations; Based on the spatiotemporal interaction representation, a three-dimensional state vector consisting of alertness, fatigue, and distraction is maintained, along with the prediction standard deviations of fatigue and distraction. Each component of the three-dimensional state vector is a non-negative continuous quantity, and the sum of the three is one. Each of fatigue and distraction is referred to as the judgment component. The current value of the three-dimensional state vector is obtained by adding the previous value to the state prior obtained by the state transition matrix transformation and the spatiotemporal interaction representation by the input projection matrix transformation, or by the intermediate result derived from the state prior, and then normalizing the components so that their sum is one. Each of the determined components has a determination threshold that acts on the three-dimensional state vector. The determination threshold is based on the population prior threshold of the determined component and is determined by adjusting the statistics of the determined component during the baseline acquisition period and in the current sliding window. The sum of the alarm multiples of each of the determined components and its prediction standard deviation is referred to as the determination quantity. Each determination quantity is compared with the determination threshold of the determined component. If any one of them exceeds the threshold, an alarm is output. If neither exceeds the threshold, the alarm multiple is replaced by a warning multiple greater than the alarm multiple. Each judgment quantity is recalculated and compared with the judgment threshold of the judgment component. If either exceeds the threshold, a warning is output. If neither exceeds the threshold, a normal condition is output.

2. The method for progressively determining the driver's state based on an explicit state-space model according to claim 1, characterized in that, The elements of the state transition matrix are non-negative, the sum of the elements in each row is one, and the diagonal elements of each row are greater than any off-diagonal element of that row. The state transition matrix is ​​obtained by parameterizing the learnable parameter matrix using a row-wise normalized exponential function. in, Let be the state transition matrix. The learnable parameter matrix, Represents a normalized exponential function applied row-wise; The learnable parameter matrix is ​​initialized in such a way that the values ​​of the diagonal elements are greater than the values ​​of the off-diagonal elements, so that at the start of training, the diagonal elements of each row of the state transition matrix are greater than any off-diagonal element of that row. The learnable parameter matrix is ​​subject to the intra-row diagonal dominance constraint during training. The intra-row diagonal dominance constraint is as follows: after each parameter update, for each row of the learnable parameter matrix, if the diagonal element of the row is less than the sum of the maximum value of the off-diagonal elements of the row and a set margin, the diagonal element of the row is set to the sum of the maximum value and the set margin. The set margin is a positive number, so that the diagonal element of each row of the state transition matrix is ​​greater than any off-diagonal element of the row throughout the training process, and the three-dimensional state vector tends to maintain the current value more than it tends to jump to other values. The normalization process involves setting the negative components in the state prior or the intermediate results derived from the state prior to zero, and then normalizing them according to the sum of their components, so that the alertness, the fatigue, and the distraction are non-negative and their sum is one.

3. The method for progressively determining the driver's state based on an explicit state-space model according to claim 1, characterized in that, First, a linear extrapolation is performed on the difference between adjacent moments based on the spatiotemporal interaction representation. in, The spatiotemporal interaction representation is the first time since the current moment. extrapolated value of step, Take one to integers, The prediction step number is a set positive integer. and These are the current time value and the previous time value of the spatiotemporal interaction representation, respectively; Then, the current value of the three-dimensional state vector is predicted by the power of the state transition matrix, and the cumulative value of the extrapolated value after transformation by the input projection matrix is ​​included in the forward prediction to obtain the prediction intermediate value. in, For the first time since the current moment The predicted intermediate quantity of the step, Take one to integers, For the number of prediction steps, Let be the state transition matrix. The input projection matrix, The value of the three-dimensional state vector at the current moment; After setting the negative components in the predicted intermediate quantities to zero and normalizing them according to the sum of their components, we obtain the first value from the current time. The predicted state vector of the step; The sum of the alarm multiple of each of the determined components in the predicted state vector and the predicted standard deviation of the determined component at the current time is referred to as the predicted determination quantity. Each predicted determination quantity is compared with the determination threshold of the determined component. If any one of them causes the predicted determination quantity to exceed the determination threshold of the determined component at any step within the prediction step, and the output at the current time is normal, the output at the current time is changed from normal to warning.

4. The method for progressively determining the driver's state based on an explicit state-space model according to claim 1, characterized in that, The prediction standard deviation of each of the aforementioned decision components is obtained by taking the square root of the total prediction variance of that decision component, where the total prediction variance is the sum of the data-inherent noise variance and the model uncertainty variance of that decision component. in, Take two and three respectively, and assign them to the fatigue level and the distraction level respectively. For the first The predicted standard deviation of each component, For the first The inherent noise variance of each component of the data, representing the variance caused by the imaging noise of the driver's facial image sequence itself, illumination variations, and facial occlusion, is directly predicted by the network output layer. For the first The model uncertainty variance of each component, representing the variance caused by the deviation of the driver's facial image sequence from the training data distribution, is estimated by the variance of the forward propagation results of multiple random discarding of some neurons.

5. The method for progressively determining the driver's state based on an explicit state-space model according to claim 1, characterized in that, The baseline acquisition period ends when the cumulative number of frames reaches a set number of frames. During the baseline acquisition period, the determination threshold of each of the determined components is taken as the prior threshold of the population for that determined component, and the mean of each of the determined components during the period is recorded as their respective individual baseline mean. After the baseline acquisition period ends, the determination threshold of each determined component is obtained by weighted summation of three factors: the population prior threshold of the determined component, the difference between the individual baseline mean of the determined component and the mean of the determined component within the current sliding window, and the standard deviation of the determined component within the current sliding window; the three weights of the weighted summation are set values, and the length of the current sliding window is a set number of frames. It also includes saving the current prompt status, the prompt status level from low to high is no prompt, warning prompt and alarm prompt, the normal, the warning and the alarm correspond to no prompt, warning prompt and alarm prompt respectively; The system counts the number of consecutive occurrences of the same output result. When the number of consecutive occurrences of an output result that is higher than the level corresponding to the current prompt status reaches a set upgrade frame number, the prompt status is upgraded to the level corresponding to that output result. When the number of consecutive frames of output results below the level corresponding to the current prompt status reaches a set number of downgrade frames, the prompt status is reduced or canceled. When the prompt status is canceled, the value after subtracting the set hysteresis from the judgment threshold is used as the comparison benchmark. When the square of the prediction standard deviation of each of the judged components is higher than the uncertainty threshold within a set number of consecutive high uncertainty frames, the prompt status is prohibited from shifting from a high level to a low level, but the shift of the prompt status from a low level to a high level is not restricted.

6. A driver state progressive determination system based on an explicit state-space model, comprising an image acquisition and preprocessing module for acquiring a sequence of driver facial images, and a spatiotemporal feature extraction module for extracting spatiotemporal interaction representations from the driver facial image sequence, characterized in that, Also includes: The progressive state transition module maintains a three-dimensional state vector composed of alertness, fatigue, and distraction based on the spatiotemporal interaction representation. Each component of the three-dimensional state vector is a non-negative continuous quantity and the sum of the three is one. Each of the fatigue and distraction is referred to as the judgment component. The current value of the three-dimensional state vector is obtained by adding the previous value after transformation by the state transition matrix and the spatiotemporal interaction representation after transformation by the input projection matrix, or by the intermediate result derived from the state prior, and then normalizing the components to make them non-negative and their sum one. The uncertainty perception output module outputs the prediction standard deviation of each of the decision components, and each prediction standard deviation is obtained by taking the square root of the total prediction variance of the decision component. The adaptive decision module has a decision threshold for each of the decision components that is to be determined, which is based on the prior threshold of the population for the decision component and is determined by adjusting the statistics of the decision component during the baseline acquisition period and during the current sliding window. The adaptive decision module takes the sum of the alarm multiple of each of the decided components and its prediction standard deviation as the decision quantity, and compares each decision quantity with the decision threshold of the decided component. If any one of them exceeds the threshold, an alarm is output. If neither exceeds the threshold, the alarm multiple is replaced by a warning multiple greater than the alarm multiple. Each judgment quantity is recalculated and compared with the judgment threshold of the judgment component. If either exceeds the threshold, a warning is output. If neither exceeds the threshold, a normal condition is output. The vehicle interaction module converts the normal, warning, or alarm output by the adaptive decision module into visual prompts, audible prompts, or takeover risk prompts, and saves the current prompt status. The output of the image acquisition and preprocessing module enters the spatiotemporal feature extraction module, the output of the spatiotemporal feature extraction module enters the progressive state transition module and the uncertainty perception output module, the outputs of the progressive state transition module and the uncertainty perception output module enter the adaptive decision module, and the output of the adaptive decision module enters the vehicle interaction module.

7. The driver state asymptotic determination system based on an explicit state-space model according to claim 6, characterized in that, The progressive state transition module includes a parameter storage area and a recursive operation unit. The parameter storage area stores the state transition matrix, the input projection matrix, the observation projection matrix, the process noise covariance matrix, the observation noise basis, and the posterior error covariance matrix. The recursive operation unit determines the current value of the three-dimensional state vector using recursive reasoning in the form of Kalman filtering. Specifically, it determines the Kalman gain by using the state prior, the prior error covariance matrix, the observation projection matrix, and the observation noise covariance matrix. The difference between the actual observations obtained from the driver's facial image sequence and the predicted observations obtained by mapping the state prior using the observation projection matrix is ​​weighted by the Kalman gain to correct the state prior, resulting in a corrected state. The negative components in the corrected state are then set to zero and normalized according to the sum of their components to obtain the current value of the three-dimensional state vector. The uncertainty perception output module sends the total prediction variance of each of the decided components back to the progressive state transition module as the observation noise covariance matrix used to determine the Kalman gain at the next moment.

8. The driver state asymptotic determination system based on an explicit state-space model according to claim 6, characterized in that, The progressive state transition module also performs forward prediction on the current value of the three-dimensional state vector, that is, it applies the power of the state transition matrix to the current value, and includes the cumulative value of the extrapolated value obtained by linear extrapolation of the spatiotemporal interaction representation according to the difference between its adjacent times after transformation by the input projection matrix, to obtain the prediction intermediate quantity. Then, the negative component in the prediction intermediate quantity is set to zero and normalized according to the sum of its components to obtain the prediction state vector for each step within the prediction step number from the current time. The adaptive decision module compares the sum of each of the decision components in the prediction state vector and the alarm multiple of the prediction standard deviation of the decision component at the current time with the decision threshold of the decision component. If any one of them exceeds the decision threshold of the decision component at any step within the prediction step number, and the output at the current time is normal, the output at the current time is changed from normal to warning.

9. The driver state asymptotic determination system based on an explicit state-space model according to claim 6, characterized in that, The vehicle interaction module includes a prompt status register, a continuous frame count counter, an uncertainty gating unit, and a prompt instruction generation unit; the prompt status level stored in the prompt status register is from low to high as no prompt, warning prompt, and alarm prompt, and the normal, warning, and alarm prompts correspond to no prompt, warning prompt, and alarm prompt respectively; The consecutive occurrence frame counter counts the number of consecutive occurrences of the same result output by the adaptive decision module; The prompt instruction generation unit raises the prompt status to the level corresponding to the current prompt status when the number of consecutive frames of results with a higher level than the current prompt status reaches a set upgrade frame number. When the number of consecutive frames of results with a lower level than the current prompt status reaches a set downgrade frame number, the prompt status is lowered or canceled. When canceling the prompt status, the value after subtracting a set hysteresis from the judgment threshold is used as the comparison benchmark. The uncertainty gating unit blocks the transition of the prompt status from a high level to a low level, but does not block the transition of the prompt status from a low level to a high level, when the total prediction variance of each of the judged components is higher than the uncertainty threshold within a set number of consecutive high uncertainty frames.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the driver state progressive determination method based on an explicit state-space model as described in any one of claims 1 to 5.

Images