Fatigue detection method based on multilayer perceptron

Abstract

The present application belongs to the physiological state monitoring technical field, disclose a kind of fatigue detection method based on multilayer perception, comprising: the first modal data reflecting the physiological state of controller is collected, and the second modal data reflecting the behavior state of controller;First modal data and second modal data are processed using multi-branch multilayer perception, to generate fusion feature, wherein, processing includes: processing first modal data in the first branch of multi-branch multilayer perception, to generate first feature;Second modal data is processed in the second branch of multi-branch multilayer perception, to generate second feature;And the intermediate layer of multi-branch multilayer perception fuses first feature and second feature, to generate fusion feature, determines the fatigue grade of controller.The present application solves the technical problems that existing fatigue detection method exists when fusing heterogeneous multi-modal data, such as insufficient fusion depth, difficult to capture nonlinear correlation, and model static, lack of adaptive optimization capability.

Description

Technical Field

[0001] This invention relates to the field of physiological state monitoring technology, and in particular to a fatigue detection method based on a multilayer sensor. Background Technology

[0002] In air traffic control scenarios, controller fatigue is a significant hazard affecting operational accuracy and safety decision-making. Even a minor error, such as delayed instructions or misjudgment, can lead to catastrophic consequences. Existing fatigue detection methods either rely on invasive physiological sensors, which interfere with normal operations, or on single, non-invasive indicators (such as analyzing only facial expressions or only operational behaviors). These methods are easily affected by individual differences, emotional fluctuations, or pseudo-fatigue characteristics in complex real-world work environments, resulting in insufficient accuracy and robustness, and failing to meet high safety standards.

[0003] An academic paper titled "A Multimodal Feature Fusion Framework for Sleep-Deprived Fatigue Detection to Prevent Accidents" (Sensors 2023, 23, 4129) discloses a representative technical solution. This solution collects data from four modalities: visual images, thermal imaging, keyboard dynamics, and speech features. First, it extracts preset features from each modal data. Then, it performs weighted fusion by assigning a set of empirical weights to these extracted features. Finally, it inputs the fused feature vector into a classifier such as a multilayer perceptron (MLP) to determine the fatigue state.

[0004] However, in related technological fields, how to efficiently and deeply fuse heterogeneous data with vastly different properties (e.g., eye-tracking data reflecting physiological states and keyboard operation data reflecting external behaviors) to significantly improve detection accuracy and robustness has always been a persistent technical challenge. The aforementioned shallow fusion method of "extracting first and then weighting" often fails to fully capture the deep, non-linear intrinsic relationships between different modalities when processing heterogeneous data, thus limiting the upper limit of model performance. Furthermore, existing models are typically static, meaning they remain unchanged once trained, making it difficult to adapt to differences between individuals and changes in the state of the same individual at different times, resulting in insufficient personalization and adaptability in practical applications.

[0005] Therefore, there is an urgent need in this field for a new fatigue detection method that can deeply fuse heterogeneous multimodal data and adaptively optimize based on feedback from actual applications. Summary of the Invention

[0006] To address the technical problems of insufficient fusion depth, difficulty in capturing nonlinear correlations, and static models lacking adaptive optimization capabilities in existing fatigue detection methods when fusing heterogeneous multimodal data, this invention provides a fatigue detection method with improved accuracy, robustness, and adaptability.

[0007] To achieve the above-mentioned objectives, the technical solution provided by this invention includes:

[0008] Fatigue detection methods based on multilayer perceptrons include:

[0009] Collect first modal data reflecting the physiological state of the controller, and second modal data reflecting the behavioral state of the controller;

[0010] A multi-branch multilayer perceptron is used to process the first modality data and the second modality data to generate a fused feature, wherein the processing includes:

[0011] The first modal data is processed in a first branch of the multi-branch multilayer perceptron to generate a first feature;

[0012] The second modal data is processed in a second branch of the multi-branch multilayer perceptron to generate a second feature;

[0013] The first feature and the second feature are fused in an intermediate layer of the multi-branch neural network model to generate the fused feature;

[0014] Based on the fusion characteristics, the fatigue level of the controller is determined.

[0015] Preferably, the first modal data includes eye movement data, heart rate data, and / or skin conductance data.

[0016] Preferably, the second modal data includes keyboard operation data, mouse operation data, and / or voice response data.

[0017] Preferred options also include:

[0018] When the fatigue level exceeds a preset threshold, an intervention measure is triggered;

[0019] The multi-branch neural network model is updated online based on feedback data related to the intervention.

[0020] Preferably, the training method for the multi-branch multilayer perceptron includes:

[0021] Based on the gradient of the loss function of the multi-branch neural network model on an input data, an adversarial noise is generated;

[0022] The adversarial noise is applied to the input data to generate an adversarial sample.

[0023] Preferably, the training method for the multi-branch multilayer perceptron includes:

[0024] The multi-branch neural network model is trained using a weighted focus loss function, wherein the weighted focus loss function includes:

[0025] A category-balanced weighting term is used to assign higher static loss weights to pre-defined moderate and severe fatigue samples than to ordinary fatigue level samples.

[0026] A dynamic focusing term is used to dynamically decay the loss of simple samples based on the model's predicted probability of the samples.

[0027] Preferably, the training method for the multi-branch multilayer perceptron includes:

[0028] The multi-branch neural network model is trained using an optimization strategy, the optimization strategy including:

[0029] An AdamW optimizer is used, combined with a periodically restarted cosine annealing learning rate schedule, to update the parameters of the multi-branch neural network model.

[0030] In the later stages of training, a random weighted average Gaussian method is used to collect weight parameters of multiple sets of the multi-branch neural network models, and a weight mean and a weight covariance are calculated based on the multiple sets of weight parameters to generate a Bayesian model set for determining the fatigue level.

[0031] Preferably, the multi-branch neural network model further includes a cross-layer jump connection structure, which is used to combine the first feature and / or the second feature with the fused feature.

[0032] Beneficial effects

[0033] 1. The multi-branch network architecture adopted in this invention can perform specialized and parallel feature extraction for physiological and behavioral data with different properties. Then, through the deep fusion mechanism of the intermediate layer of the network (such as cross-modal Transformer), it effectively captures the deep and nonlinear correlation between heterogeneous data that cannot be achieved by the shallow weighted fusion method of the existing technology.

[0034] 2. This invention further includes an online update mechanism that enables continuous, online optimization of the model based on feedback data from intervention measures. This overcomes the shortcomings of existing technologies where models are static and cannot evolve, allowing the model to adapt to individual differences and state changes, becoming increasingly accurate in long-term applications, and achieving truly personalized fatigue monitoring. Attached Figure Description

[0035] Figure 1 This is a flowchart illustrating a fatigue detection method based on a multilayer perceptron provided in a preferred embodiment of the present invention. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described below with reference to the accompanying drawings.

[0037] Example 1

[0038] The core idea of ​​the method proposed in this embodiment is that it does not simply mix various types of data in a later stage, but instead constructs a specially designed multi-branch, multi-layer perceptron with parallel processing capabilities. This model can deeply mine the fatigue characteristics contained in data of different properties and efficiently fuse them within the network, thereby making a more reliable comprehensive judgment than any single data source. Figure 1 As shown, the fatigue detection method based on a multilayer perceptron disclosed in this embodiment includes:

[0039] S1. Collect first modal data reflecting the physiological state of the controller, and second modal data reflecting the behavioral state of the controller.

[0040] Those skilled in the art will understand that the first and second modal data serve as early signals of fatigue in controllers working under high pressure and high cognitive load. Conventional methods often rely on readily available but low-quality data sources.

[0041] For example, facial video analysis using ordinary cameras attempts to capture macroscopic behaviors such as yawning and nodding. The shortcomings of this method are: 1) These behaviors are late indicators of fatigue, and by the time they appear, it is often too late; 2) These behaviors are easy to fake or suppress, and controllers, driven by a high sense of responsibility, will subconsciously cover up these signs of fatigue; 3) The signal-to-noise ratio is low, and it is easily interfered with by irrelevant factors such as changes in lighting and facial expressions.

[0042] Instead of focusing on macroscopic behavior, this invention makes a more insightful technological choice, with a core consideration of finding "high-fidelity" signal sources directly related to cognitive state and fine motor skills. This step involves the non-invasive simultaneous acquisition of at least two different types of data. The first type is first modal data reflecting the controller's physiological state, which directly reveals changes in physiological indicators caused by fatigue. In a preferred embodiment, the core of this first modal data includes eye-tracking behavior data, such as fixation duration, saccade speed, and pupil diameter change rate acquired via an eye tracker deployed on a display. Eye-tracking behavior is a direct reflection of cognitive load and alertness levels and is highly sensitive to fatigue. As a supplement or alternative, the first modal data may also include heart rate data (especially heart rate variability HRV) and / or skin conductance data acquired via wearable devices (such as smart bracelets), which also effectively characterize the activity state of the autonomic nervous system and are closely related to fatigue levels.

[0043] Meanwhile, the second type of data collected is second modal data reflecting the controller's behavioral state. This type of data can capture the impact of fatigue on fine motor skills and reaction time. For example, the second modal data may include keyboard operation data (such as keystroke rate, interval duration, error rate), mouse operation data (such as movement trajectory smoothness, click accuracy, pause time), and / or voice response data (such as response delay to instructions, voice fundamental frequency, energy changes).

[0044] It should be understood that the raw data collected needs to be preprocessed before being input into the model. This step includes: smoothing the eye-tracking data using a Kalman filter to remove noise; normalizing all continuous numerical features (such as pupil diameter and heart rate) (e.g., using min-max normalization) to scale them to the appropriate range; and segmenting and extracting features from the keyboard, mouse, and voice data to form fixed-dimensional feature vectors.

[0045] S2. A multi-branch multilayer perceptron is used to process the first modal data and the second modal data to generate a fused feature.

[0046] The multi-branch multilayer perceptron has at least two specialized parallel processing branches.

[0047] The first branch is designed to process the first modality data (such as eye movements, heart rate, etc.) by deeply extracting the first feature reflecting physiological fatigue through several fully connected layers and nonlinear activation functions. Its input is a preprocessed physiological state feature vector. In a specific implementation, this branch consists of three fully connected layers with the number of neurons set to 128, 64, and 32, respectively. Each fully connected layer can be followed by a batch normalization layer and a ReLU activation function. The output of this branch is a 32-dimensional first feature vector.

[0048] Correspondingly, the second branch is dedicated to processing the second modality data (such as keyboard and mouse input, voice, etc.), extracting a second feature reflecting a decline in behavioral performance through its own network layer structure. In a specific implementation, this branch consists of two fully connected layers, with the number of neurons set to 64 and 32 respectively. Each fully connected layer can be followed by a layer normalization (LayerNorm) layer and a tanh activation function. The output of this branch is a 32-dimensional second feature vector.

[0049] After each branch independently extracts deep features, the model fuses these features in its intermediate layers to generate the fused features. Specifically, the fusion step is implemented through a cross-modal Transformer framework. It's worth noting that this fusion is not a simple feature concatenation, but a deep fusion achieved through a more complex network structure. Conventional fusion methods are shallow feature-level fusions. That is, features are first extracted manually or through a simple network from each modality, and then these feature vectors are simply concatenated or weighted summed in the later stages of the network (usually before being input to the final classifier). The drawback of this method is that it assumes the relationships between different modalities are relatively simple and static, failing to capture dynamic, context-dependent dependencies. For example, it struggles to learn complex cooperative patterns such as "only when pupillary reflexes are sluggish does a slight slowing of mouse movement become a strong fatigue signal."

[0050] In a preferred embodiment, the multi-branch multilayer perceptron further includes a cross-layer skip connection structure. Specifically, this structure can combine the features before fusion (i.e., the first feature and / or the second feature) with the features after fusion. For example, the output of the first fully connected layer in the first branch can be directly combined with the fused features generated by the intermediate layers through a linear transformation layer (for dimension matching) (e.g., by element-wise addition or concatenation). This design can effectively enhance gradient flow and information reuse within the network, prevent information loss in deep networks, and thus improve the training efficiency and final detection performance of the model.

[0051] S3. Based on the fusion features, the model's output layer ultimately determines a fatigue level representing the controller's current state. The fusion features (and optional skip features) are then fed into one or more fully connected layers for a final nonlinear transformation, and finally processed by an output layer with N neurons (N being the number of fatigue levels, e.g., N=4, corresponding to alertness, mild fatigue, moderate fatigue, and severe fatigue, respectively). This output layer uses the Softmax activation function and outputs a probability distribution representing the controller's current fatigue level. The level with the highest probability is the final determined fatigue level.

[0052] The level can be a continuous fatigue score (e.g., between 0 and 1) or a discrete level classification (e.g., “awake”, “mild fatigue”, “moderate fatigue”, “severe fatigue”).

[0053] In order to build a closed-loop management system that can self-evolve and continuously adapt, an important embodiment of the present invention also includes a dynamic intervention mechanism, specifically including: S4, when the fatigue level exceeds a preset threshold, triggering an intervention measure, such as issuing a warning to the controller, sending a reminder to the command center management platform, or automatically suggesting a shift rotation.

[0054] Those skilled in the art will know that traditional machine learning models are static; once training is complete, their parameters remain fixed. This leads to two problems: 1) "One-size-fits-all" models cannot adapt to the differences in the physiological and behavioral baselines of different controllers; 2) models cannot adapt to long-term changes in the state of the same controller due to factors such as work and rest schedules and health.

[0055] Therefore, the conventional solution is periodic offline retraining. That is, after collecting data for several months or a year, the entire model is taken offline, and a large-scale training is performed from scratch using all historical and new data, followed by redeployment. The drawbacks of this method are: 1) The feedback cycle is too long; the model is "outdated" between retraining sessions and cannot respond promptly to changes in user status; 2) It consumes huge resources, requiring significant computational resources and time for each retraining session; 3) There is a cold start problem; for newly added controllers, the model needs a long period of data accumulation to begin adapting.

[0056] This invention considers replacing the cumbersome, low-frequency "reconstruction" loop with a lightweight, high-frequency "fine-tuning" loop. In some preferred embodiments, a closed-loop adaptive update mechanism is designed, including: S5, updating the multi-branch multilayer perceptron online based on feedback data related to the intervention. The system records feedback data related to the intervention, such as manual annotations by managers regarding the accuracy of the fatigue assessment.

[0057] Based on this high-quality feedback data, the system can employ incremental learning or transfer learning strategies to update the multi-branch multilayer perceptron online. This closed-loop feedback mechanism enables the model to continuously learn from real-world applications, constantly adapting to individual differences and environmental changes, thus achieving truly personalized fatigue monitoring.

[0058] The multi-branch, multi-layer perceptron described above requires a specialized and refined training process to achieve excellent performance. This invention also provides an effective method for training this model, which includes one or more of the following technical solutions:

[0059] In one implementation, a common approach to improve a model's robustness to sensor noise in real-world environments is data augmentation, such as adding random Gaussian noise to the data. However, this is a "blind" augmentation that fails to specifically improve the model's robustness at its most vulnerable points. The training method provided in this embodiment employs adversarial noise adaptive training. Instead of simply adding random noise to the input data, this method, in each iteration of training, first calculates the adversarial noise that will maximally interfere with the model's decisions based on the gradient of the model's loss function with respect to clean input data. Then, this adversarial noise is applied to the original input data to generate a "hardest" adversarial example, which is then used to update the model's weights. By training under this worst-case perturbation, the model's decision boundaries become smoother and more robust.

[0060] Specifically, during training, a strategy based on a variant of the Fast Gradient Sign Method (FGSM) is employed for the continuous numerical features of the input. Specifically, the gradient of the loss function with respect to the input data is first calculated. Then, a small perturbation within a preset norm is applied to the original input data along the direction of this gradient to generate an adversarial example that is most likely to confuse the model. Finally, this adversarial example is used to update the model's parameters. Training in this way significantly improves the model's robustness to sensor noise.

[0061] In another implementation, considering that fatigue samples (especially moderate to severe fatigue) are typically far fewer than conscious samples in real-world datasets, a solution to this data imbalance problem is needed.

[0062] Conventional methods involve simple class weighting, assigning higher loss weights to minority class samples. However, this fails to distinguish between "easy" and "difficult" samples within the minority class, potentially leading to low training efficiency. The training method provided in this embodiment employs a customized weighted focus loss function. This loss function cleverly combines two mechanisms: First, it includes a class balancing weight term, a static weight used to assign higher base loss weights to preset moderate and severe fatigue samples compared to non-critical fatigue level samples (such as conscious samples). For example, weights of 0.1, 0.2, 0.3, and 0.4 can be set for conscious, mild, moderate, and severe fatigue levels, respectively, forcing the model to focus more on the minority but important classes. Second, it includes a dynamic focus term, which dynamically and non-exponentially decays the loss generated by "easy samples" that the model can easily and correctly classify, based on the model's predicted probability of the samples. This allows computational resources during training to be more concentrated on overcoming the difficult-to-distinguish "difficult samples."

[0063] In another preferred embodiment, to enable the model to converge efficiently and stably and achieve optimal generalization ability, the conventional approach is to use the standard Adam optimizer and stepwise learning rate descent. While effective, this method may converge to a "sharp" local optimum where generalization is not optimal. The training method provided in this embodiment employs a combination of advanced optimization strategies. First, an AdamW optimizer is used, which more effectively prevents overfitting by decoupling weight decay and gradient updates. This optimizer is combined with a periodically restarting cosine annealing learning rate scheduling strategy, allowing the learning rate to decrease and reset periodically and smoothly during training, helping the model escape sharp local optima and converge to a flatter, more generalizable region. Furthermore, in the later stages of training, a stochastic weighted average Gaussian method is activated. This method continuously collects the weight parameters of multiple models and calculates a weight mean and weight covariance based on these parameters. Ultimately, this no longer generates a single, fixed model, but rather a set of Bayesian models. When determining fatigue levels, multiple models can be sampled from this model set for integrated prediction, and the results are more stable and reliable than any single model.

[0064] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A fatigue detection method based on a multilayer perceptron, characterized in that, include: Collect first modal data reflecting the physiological state of the controller, and second modal data reflecting the behavioral state of the controller; A multi-branch multilayer perceptron is used to process the first modality data and the second modality data to generate a fused feature, wherein the processing includes: The first modal data is processed in a first branch of the multi-branch multilayer perceptron to generate a first feature; The second modal data is processed in a second branch of the multi-branch multilayer perceptron to generate a second feature; The first feature and the second feature are fused in an intermediate layer of the multi-branch multilayer perceptron to generate the fused feature; Based on the fusion features, the controller's fatigue level is determined; Also includes: When the fatigue level exceeds a preset threshold, an intervention measure is triggered; The multi-branch multilayer perceptron is updated online based on feedback data related to the intervention measures. The training method for the multi-branch multilayer perceptron includes: The multi-branch multilayer perceptron is trained using a weighted focus loss function, which includes: A category-balanced weighting term is used to assign higher static loss weights to pre-defined moderate and severe fatigue samples than to ordinary fatigue level samples. A dynamic focusing term is used to dynamically decay the loss of simple samples based on the model's predicted probability of the samples. The training method for the multi-branch multilayer perceptron includes: The multi-branch multilayer perceptron is trained using an optimization strategy, the optimization strategy including: An AdamW optimizer is used in conjunction with a periodically restarted cosine annealing learning rate schedule to update the parameters of the multi-branch multilayer perceptron. In the later stages of training, a random weighted average Gaussian method is used to collect multiple sets of weight parameters for the multi-branch multilayer perceptron, and a weight mean and a weight covariance are calculated based on the multiple sets of weight parameters to generate a Bayesian model set for determining the fatigue level.

2. The fatigue detection method based on a multilayer perceptron as described in claim 1, characterized in that, The first modality data includes eye movement data, heart rate data, and / or skin conductance data.

3. The fatigue detection method based on a multilayer perceptron as described in claim 1, characterized in that, The second modal data includes keyboard operation data, mouse operation data, and / or voice response data.

4. The fatigue detection method based on a multilayer perceptron as described in claim 1, characterized in that, The training method for the multi-branch multilayer perceptron includes: Based on the gradient of the loss function of the multi-branch multilayer perceptron for an input data, an adversarial noise is generated; The adversarial noise is applied to the input data to generate an adversarial example, and the adversarial example is used to train the multi-branch multilayer perceptron.

5. The fatigue detection method based on a multilayer perceptron as described in claim 1, characterized in that, The multi-branch multilayer perceptron further includes a cross-layer jump connection structure, which is used to combine the first feature and / or the second feature with the fused feature.

Images