Controller fatigue monitoring method based on facial features

Abstract

The present application relates to a kind of controller fatigue monitoring method based on facial features, steps are, face detection: face key point detection: eye mouth aspect ratio calculation: for the parameter calibration of individual facial feature difference: severe fatigue behavior detection: fatigue discrimination algorithm design based on dynamic attenuation time window: algorithm effectiveness analysis based on electroencephalogram signal.This method incorporates the characteristics of control work scene in fatigue discrimination algorithm, can more effectively determine and alarm the fatigue state of controller, is of great significance for preventing the occurrence of air traffic control unsafe events, provides relevant ideas for subsequent controller non-contact fatigue monitoring research, and the efficiency of algorithm is optimized, meets real-time demand.

Description

Technical Field

[0001] This invention relates to the field of human factors in civil aviation transportation, and in particular to a method for monitoring controller fatigue based on facial features. Background Technology

[0002] With the continuous development of the economy and the increasing travel demands of people, the civil aviation sector has also ushered in unprecedented development opportunities. Air traffic controllers, as crucial figures in air traffic control, shoulder the heavy responsibility of ensuring aviation safety and effective flight management. However, with the increase in the number of flights, the workload faced by controllers has also increased accordingly. Therefore, accurately determining and alerting to controller fatigue is of great significance for preventing unsafe incidents in air traffic control. Monitoring controller fatigue through facial feature information, due to its real-time, non-invasive, and contactless characteristics, has become an important research direction in the field of aviation transport safety. However, the technology for monitoring controller fatigue based on facial feature information is currently not mature enough and is difficult to implement with highly accurate results in real-world scenarios. Specifically, the following problems exist: ① Current research struggles to achieve a good balance between accuracy and real-time performance in face detection, affecting the effectiveness of facial fatigue monitoring. ② Most current research generally uses 2D coordinate information when locating facial key points, ignoring some spatial features, and rarely incorporates the characteristics of the controller's work environment during fatigue determination. ③ Previous studies have mostly used machine learning methods to determine the state of the eyes and mouth, classifying different states by training neural network models. While these methods have a certain generalization ability, they are somewhat inadequate in handling test errors caused by individual differences. In this technical field, the published paper "Research on Fatigue Monitoring Method of Air Traffic Controllers Based on Facial Feature Recognition" (China Safety Science Journal, 2012, Vol. 22, No. 7) uses the OpenCV open-source computer vision library and combines image binarization and module matching techniques to identify the fatigue status of air traffic controllers. However, the template matching scheme used has poor fatigue detection performance. The published paper "Fatigue Driving Detection Based on Eye and Mouth State Recognition" (Journal of Jilin University, 2017, Vol. 35, No. 2) uses the Adaboost algorithm to detect the driver's face and combines it with various prior knowledge, including the geometric proportions of facial features, to locate the coordinates of the mouth and eyes. Then, through the application of grayscale integral projection, the eye opening and mouth roundness are extracted and used as fatigue features. Fatigue is judged according to the PERCLOS principle. However, its accuracy in face detection and key point localization is low, affecting downstream tasks. The published patent application, "A Method and System for Monitoring Fatigue Status of Air Traffic Controllers," (Publication No. CN117152724A), detects controller fatigue based on posture, eye, and mouth features. However, it does not consider the impact of the controller's vocal workload on fatigue levels, nor does it account for false detections caused by the controller's head-down posture while operating the keyboard. Therefore, designing a non-contact, real-time fatigue monitoring method with good accuracy and robustness that incorporates the characteristics of controller work is of great significance for preventing unsafe incidents in air traffic control. Summary of the Invention

[0003] Given the limitations of existing fatigue monitoring methods that rely heavily on facial information and lack real-world scenario integration, resulting in low algorithm robustness, this paper proposes a real-time fatigue monitoring method that incorporates relevant characteristics of air traffic control work. This method first performs face detection on a single-frame image, returning the face confidence score and detection box coordinates. Non-maximum suppression is applied to the detection results, limiting the detection to only one face per frame. Then, the original image is cropped based on the determined face detection box coordinates. The cropped image is then input into a 468-key-point face detection model, returning the coordinates of each keypoint. Using the 3D coordinates of the 468 keypoints obtained from the model inference, the 3D Euclidean distance is calculated to further reconstruct the aspect ratio functions of the eyes and mouth. Finally, the method selects an individual controller with open eyes... The aspect ratio of the eyes and mouth was calculated for five states: closed eyes, half-open eyes, open mouth, and closed mouth. The corresponding EAR_3D and MAR_3D thresholds for each subject were calibrated. The eye and mouth states of controllers were assessed, and the results were written into a time-series array. Continuous frame thresholds were set based on the real-time inference performance of the computer or edge computing development board to detect severe fatigue behavior and issue alarms. A dynamic decay time window mathematical model was designed by integrating the controller's work scenario. Fatigue was judged and alarms were issued by calculating the controller's blink frequency within the dynamic time window and combining it with the corresponding thresholds. Finally, bivariate correlation analysis was performed on the EEG processing results and facial video processing results of 30 controller samples using SPSS to verify the effectiveness of the algorithm. This provides relevant ideas for subsequent research on non-contact fatigue detection for controllers.

[0004] The technical solution adopted in this invention is: a fatigue monitoring method for air traffic controllers based on facial features, implemented on a computer or edge computing development board. The computer or development board includes a face detection module, a facial key point detection module, an eye and mouth aspect ratio calculation module, a parameter calibration module for individual facial features, a severe fatigue behavior detection module, a fatigue discrimination algorithm design module based on a dynamic decay time window, and an algorithm effectiveness analysis module based on electroencephalogram (EEG) signals. The steps are as follows:

[0005] Step A, Face Detection:

[0006] First, face detection is performed based on the input single-frame image, and the face confidence score and detection box coordinates are returned. Non-maximum suppression is applied to the detection results to limit the detection to only one face in the image.

[0007] Step B, facial landmark detection:

[0008] After determining the coordinates of the face detection box in step A, the original image is cropped according to the coordinates. The cropped image is then input into the 468 face key point detection model, which returns the coordinate information of each key point.

[0009] Step C, Calculation of the aspect ratio of the eyes and mouth:

[0010] By using the three-dimensional coordinate information of the 468 key facial points obtained through model reasoning in step B, the three-dimensional Euclidean distance is calculated, and the aspect ratio function of the eyes and mouth is further reconstructed.

[0011] Step D, threshold calibration for individual facial feature differences:

[0012] Using the eye and mouth aspect ratio calculation method in step C, five state images of a single controller were selected for calculation: open eyes, closed eyes, half-open eyes, open mouth, and closed mouth. The corresponding EAR_3D and MAR_3D thresholds for each subject were calibrated, where EAR_3D is the eye aspect ratio and MAR_3D is the mouth aspect ratio.

[0013] Step E, Severe Fatigue Behavior Detection:

[0014] Using the EAR_3D and MAR_3D thresholds obtained in step D, the state of the controller's eyes and mouth is determined, and the results are written into a time series array. The threshold for consecutive frames is set according to the real-time inference performance of the computer or edge computing development board to achieve the effect of detecting severe fatigue behavior.

[0015] Step F, Design of fatigue discrimination algorithm based on dynamic decay time window:

[0016] Using the fatigue behavior and eye and mouth status information obtained in step E, a dynamic decay time window mathematical model is designed by integrating the controller's work scenario. By calculating the controller's blinking frequency within the dynamic time window and combining it with the fatigue threshold, fatigue is judged and an alarm is issued.

[0017] Step G, Algorithm effectiveness analysis based on EEG signals:

[0018] SPSS was used to perform correlation analysis between the controller's EEG signal processing results and the facial video processing results in step F to verify the effectiveness of the algorithm.

[0019] By following the steps above, the fatigue status of air traffic controllers can be monitored, and alarms can be issued when fatigue is detected, in order to prevent unsafe air traffic control incidents caused by physical fatigue.

[0020] The face detection model described in step A integrates depthwise separable convolution with Vision Transformer to build a new lightweight feature extraction network, simplifying the encoding method of global feature extraction and balancing the model's inference accuracy and speed.

[0021] The face landmark detection algorithm described in step B performs a second-stage inference based on the face detection results in step A. It uses Google's Attention Mesh face 468 landmark detection algorithm, which is an improvement on Face Mesh, to obtain the three-dimensional spatial coordinate information of facial features.

[0022] The method for calculating the aspect ratio of the eyes and mouth described in step C uses three-dimensional facial coordinate information. By calculating the three-dimensional Euclidean distance, the aspect ratio function of the eyes and mouth is further reconstructed. The specific formula for calculating the three-dimensional Euclidean distance is as follows:

[0023]

[0024] Where (x) i y i , z i ) and (x j y j , z j The coordinates of two points in space are given. Ten key points around the eye are selected for 3D aspect ratio function reconstruction. The aspect ratios of the left and right eyes are calculated separately and then averaged. The final result, EAR_3D, is used as the basis for evaluating the characteristics of the eye's condition. Specifically, the aspect ratio of the left eye is...

[0025] EAR-3D left =P i＝160，j＝144 +P i＝159，j＝145 +P i＝158，j＝153 +P i＝157，j＝154 / 4×P i＝33，j＝133 ,

[0026] Right eye aspect ratio

[0027] EAR_3D right =P i＝384，j＝381 +P i＝385，j＝380 +P i＝386，j＝374 +P i＝387，j＝373 / 4×P i＝363，j＝263 ,

[0028] Eye aspect ratio

[0029]

[0030] In terms of mouth feature data acquisition, eight key points were selected for three-dimensional aspect ratio function reconstruction.

[0031] MAR_3D=P i＝81,j＝178 +P i＝13,j＝14 +P i＝311,j＝402 / 3×P i＝62，j＝293 .

[0032] The threshold calibration method described in step D first selects five states of images from the controller's facial video data: open eyes, closed eyes, half-open eyes, open mouth, and closed mouth. Then, the key point detection model and the formula for calculating the aspect ratio of the eyes and mouth are used to determine the corresponding EAR_3D and MAR_3D thresholds for each subject, and the above five states are judged by the threshold range.

[0033] Step E describes a method for determining the state of an air traffic controller's eyes and mouth. This method uses EAR_3D and MAR_3D thresholds, determined based on individual controller differences, to adjust the state determination range. It distinguishes five states: open eyes, closed eyes, half-open eyes, open mouth, and closed mouth. It sets continuous frame parameters to identify and issue alarms for three severe fatigue behaviors: prolonged closed eyes, prolonged half-open eyes and drowsiness, and yawning.

[0034] The method for identifying the three severe fatigue behaviors—prolonged eye-closing, prolonged drowsiness with eyes half-open, and yawning—is as follows:

[0035] 1) Prolonged eye closure: If the number of consecutive frames is less than the middle threshold, it is determined to be blinking and blink count is performed. If the number of consecutive frames is less than the middle threshold, it is determined to be a prolonged eye closure state and an alarm is issued.

[0036] 2) Prolonged half-open eye drowsiness: If the aspect ratio of the eyes is greater than the middle threshold in a single frame, it is determined to be an open eye state. If the aspect ratio of the eyes is within the middle threshold for 50 consecutive frames and the value range of the middle threshold is slightly adjusted upward, it is determined to be a half-open eye drowsy state, and an alarm will be issued.

[0037] 3) Yawning: If the aspect ratio of the mouth is greater than the closed aspect ratio for three consecutive frames, it is determined to be an open mouth state, and the mouth opening count is performed; if the aspect ratio of the mouth is greater than the open mouth threshold value for 120 consecutive frames, it is determined to be a yawning state, and an alarm is issued.

[0038] The mathematical model method for designing a dynamic decay time window for controller work scenarios described in step F involves introducing three influencing factors: controller on-duty time, real-time air-to-ground communication load, and the number of fatigue events. A dynamic decay scheme with a time window is adopted, using nonlinear least squares to fit the decay process to an exponential decay form. It is assumed that the controller's fatigue level increases with real-time increases in on-duty time and voice frequency. The decay coefficient λ of the dynamic time window W is affected by the above two factors, as well as a bias term representing fatigue events. The specific formula is as follows:

[0039] λ=kp / (τ-0.1b), W=αe -λt +β,

[0040] Where p represents the total frequency of voice calls, which characterizes the real-time air-to-ground communication load; k is a scaling factor, which maps p appropriately, and here it is taken as 2e-6; τ is the attenuation rate; b is the fatigue event record value, and 0.1 is the mapping coefficient; t is the test time, in seconds / s; α is the function saturation value; β is the dynamic time window function bias term, which represents the lower limit of the time window size. Before the experiment, the initial time window was set to 15 minutes, the minimum time window was set to 2 minutes, and the average FPS was 55, so the value of W was calculated to be 49500. By using the nonlinear least squares method to fit the function, the parameters α were obtained as 43544.18, τ as 5.96, and β as 6600. These initial parameters provide an important reference for the subsequent dynamic time window attenuation calculation, and can be fine-tuned based on the results.

[0041] Step F, which involves combining a corresponding threshold for fatigue assessment, uses the ratio F of the number of blinks within a variable time window to the size of the time window. δ As the final criterion for fatigue assessment, F δ Influenced by on-duty time, real-time voice frequency, and the number of fatigue events, the specific formula is as follows:

[0042] F δ =μN blink / W,

[0043] Where N blink The blink frequency is within the dynamic time window; μ is the scale adjustment coefficient, which is set to 1000 here to amplify the value of this indicator to the same scale as the subsequent EEG fatigue indicator. δ The threshold for the indicator is set to 6.5. If the threshold is exceeded, the system will determine fatigue and issue a warning.

[0044] The algorithm effectiveness analysis method based on EEG signals described in step G, and the F calculated in step F δ Based on the indicators, four different frequency waves of EEG signals—δ wave, θ wave, α wave, and β wave—were collected using a multi-channel physiological analyzer. The relative power spectrum ratio of slow waves to fast waves was used as the fatigue state assessment indicator F. ε The specific formula is as follows

[0045] F ε =P′ θ / (P′ α +P′ β )

[0046] Where P′ α For the relative power spectrum of the α wave, P′ β For the relative power spectrum of the β wave, P′ θ The relative power spectrum of the theta wave; the two index data F δ With Fε Alignment was performed along the time dimension, and F-squared values ​​were analyzed for 30 samples using SPSS. δ With F ε The data were subjected to bivariate correlation analysis one by one, and the effectiveness of the controller fatigue discrimination method based on facial features was judged by observing the degree of correlation. The beneficial effects of this invention are: in the face detection part, this invention integrates depthwise separable convolution with VisionTransformer to build a new lightweight feature extraction network and simplifies the encoding method of global feature extraction. Performance achieved AP values ​​of 0.923, 0.902, and 0.813 on the three difficulty levels of the WiderFace dataset; by adopting the Attention Mesh facial landmark detection algorithm, 468 three-dimensional coordinate information of the face is obtained, making the algorithm excellent in terms of stability, robustness, and inference speed, significantly improving the quality of downstream tasks; by using controller on-duty time, real-time air-ground communication load, and the number of fatigue events as influencing factors of the dynamic time window decay rate, the time window size is dynamically adjusted, thereby providing different detection sensitivities at different fatigue stages, effectively mitigating misjudgments caused by detection errors. The F-index obtained through the algorithm is... δ Compared with the "gold standard" EEG signal index F, fatigue detection ε Correlation analysis was conducted. In the bivariate cross-correlation analysis of 30 samples, the Pearson correlation coefficients ranged from 0.462 to 0.785, and the Sig two-tailed significance test results were all at the 0.01 level, indicating significant correlation, thus verifying the effectiveness and rationality of the invention. Attached Figure Description

[0047] Figure 1 This is a schematic diagram of the controller fatigue monitoring method based on facial features involved in the present invention;

[0048] Figure 2 This is a schematic diagram of the face detection network structure in this invention;

[0049] Figure 3 This is a schematic diagram illustrating the decay trend of the ideal time window in this invention;

[0050] Figure 4 This is a graph showing the relationship between the frequency of speech in a single subject sample and the size of the dynamic time window in this invention.

[0051] Figure 5 This is a graph showing the relationship between fatigue indices of a single test sample in this invention;

[0052] Figure 6 This is a distribution diagram of fatigue index data for a single test sample in this invention. Detailed Implementation

[0053] This method for monitoring controller fatigue based on facial features is implemented on a computer capable of running the NVIDIA TensorRT inference framework or a Jetson series edge computing development board. It consists of a face detection module, a facial key point detection module, an eye and mouth aspect ratio calculation module, a parameter calibration module for individual facial features, a severe fatigue behavior detection module, a fatigue discrimination algorithm design module based on dynamic decay time windows, and an algorithm effectiveness analysis module based on EEG signals.

[0054] In this embodiment, the method identifies the fatigue state of controllers through facial feature information and evaluates the acquired detection data using electroencephalogram (EEG) indicators, thereby verifying the effectiveness of the proposed fatigue monitoring algorithm. To this end, facial video data and EEG data were collected from 30 single-shift controllers in a control unit during control testing 35-60 minutes after their shift. The specific end time of the test was determined by the instructor based on the control situation. Video data was acquired in high definition using an HP-w300 camera via OBS Studio. EEG data was collected using a Manglod-10 multi-channel physiological analyzer.

[0055] The present invention is as follows Figure 1 As shown, it includes five steps, A through G. The specific implementation methods for each step are described in detail below.

[0056] Step A, Face Detection Module: This module fuses depthwise separable convolutions with a Vision Transformer, uses FPN for feature fusion, and introduces a context module from SSH to expand the receptive field. This constructs a novel lightweight feature extraction network and simplifies the encoding method for global feature extraction. The overall network structure is shown below. Figure 2 Depthwise separable convolutions reduce the number of parameters and improve computational efficiency compared to traditional convolution operations. Meanwhile, the Transformer relies on self-attention, making it superior to CNNs in extracting global semantic information. Combining the two allows for a better balance between detection performance and speed.

[0057] Step B, Facial Keypoint Detection Module: The facial keypoint detection algorithm in Step B performs a second-stage inference based on the facial detection results from Step A. It uses Google's Attention Mesh 468-keypoint detection algorithm, an improvement on Face Mesh, to obtain the three-dimensional coordinate information of facial features, including x-axis, y-axis, and z-axis coordinates. This algorithm has unique advantages, particularly in its finer-tuning and localization of different regions of the face, especially key areas such as the lips, eyes, and eyebrows. This characteristic allows the algorithm to provide higher accuracy when modeling feature points in important regions in subsequent tasks. By applying this algorithm, more accurate detection and recognition of various facial parts can be achieved, thereby improving the performance of the entire fatigue monitoring system.

[0058] Step C, Eye and Mouth Aspect Ratio Calculation Module: Previous studies mostly used two-dimensional keypoint coordinates as variables to construct aspect ratio functions, resulting in a loss of spatial feature information. To address this issue, three-dimensional facial coordinate information was used, and the aspect ratio functions of the eyes and mouth were further reconstructed by calculating the three-dimensional Euclidean distance. This algorithm significantly enhances the richness of spatial semantic features, thereby ensuring the quality of downstream tasks. The specific calculation formula is as follows:

[0059]

[0060] Where (x) i y i , z i ) and (x j y j , z j The coordinates of two points, i and j, are given, where i and j are indices among 468 keypoints. To ensure computational accuracy and data smoothness, 10 keypoints around the eye were selected for 3D aspect ratio function reconstruction. The aspect ratios of the left and right eyes were calculated separately and then averaged. The final result, EAR_3D, was used as the basis for evaluating eye condition characteristics. Specifically, the aspect ratio of the left eye was...

[0061] EAR_3D left =P i＝160，j＝144 +P i＝159，j＝145 +P i＝158，j＝153 +P i＝157，j＝154 / 4×P i＝33，j＝133 ,

[0062] Right eye aspect ratio

[0063] EAR_3D right =P i＝384，j＝381 +P i-385，j＝380 +P i＝386，j＝374 +P i＝387，j＝373 / 4×P i＝363，j＝263,

[0064] Eye aspect ratio

[0065]

[0066] This method can more accurately reflect eye condition, providing more reliable data support for subsequent fatigue monitoring tasks. Regarding mouth feature data acquisition, eight key points were selected for three-dimensional aspect ratio function reconstruction.

[0067] MAR_3D=P i＝81，j＝178 +P i＝13，j＝14 +P i＝311，j＝402 / 3×P i＝62，j＝293 .

[0068] Step D involves a threshold calibration module addressing individual facial feature differences. Previous studies primarily employed machine learning methods to determine eye and mouth states, classifying different states using trained neural network models. While these methods possess some generalization ability, they fall short in handling test errors caused by individual facial feature differences. To address this issue, while maintaining video stream processing efficiency, a feature matching method is used for sample-by-sample threshold calibration. First, images of five states—open eyes, closed eyes, half-open eyes, open mouth, and closed mouth—are selected from a dataset of 30 controllers. Second, a keypoint detection model and the eye / mouth aspect ratio calculation formula are used to determine the corresponding EAR_3D and MAR_3D thresholds for each subject. The threshold range is then used to determine the five states, as detailed in Table 1. This calibration method better addresses threshold detection errors caused by individual facial feature differences. Furthermore, the reduced number of related models significantly decreases the processing time per frame.

[0069] Table 1 shows the results of the aspect ratio calibration for the eye and mouth conditions.

[0070]

[0071]

[0072] Step E, Severe Fatigue Behavior Detection Module: Using the EAR_3D and MAR_3D thresholds obtained in Step D, the controller's eye and mouth states are assessed, and the results are written into a time series array. The eye and mouth state discrimination is based on the two aspect ratio thresholds obtained in Step D. Taking Sample 1 as an example, for the eyes, the aspect ratio of the half-open eye state is taken as the middle threshold of 0.25, fluctuating around it, with 0.3 as the upper limit threshold. For the mouth, the aspect ratio of the closed and open mouths forms a numerical range of 0.008-0.08. The specific severe fatigue behavior judgment scheme is as follows:

[0073] 1) Prolonged eye closure: Traverse the time sequence array. If three consecutive frames (which can be fine-tuned based on test results) are less than the middle threshold of 0.25, it is considered a blink, and blink count is performed. If 50 consecutive frames (which can be fine-tuned) are less than the middle threshold of 0.25, it is considered a prolonged eye closure state, and an alarm is triggered.

[0074] 2) Prolonged half-open eye drowsiness: Traverse the time sequence array. If a single frame is greater than the middle threshold, it is determined to be an open eye state. If the aspect ratio of the eyes is within the middle threshold for 50 consecutive frames (which can be finely adjusted), and the value range of the middle threshold is within 0.25-0.3 when it is finely adjusted upward, it is determined to be a half-open eye drowsy state, and an alarm prompt is issued.

[0075] 3) Yawning: Traverse the time sequence array. If the aspect ratio of the mouth is greater than 0.03 for three consecutive frames (which can be finely adjusted), it is determined to be an open mouth state. When setting this value, it is necessary to consider that some samples have a habit of slightly opening their mouths. This behavior is filtered out before counting the open mouths. Since the aspect ratio of the mouth in the yawning state is greater than the normal value, it is set that if the aspect ratio of the mouth is greater than the open mouth threshold value of 0.08 for 120 consecutive frames (which can be finely adjusted), it is determined to be a yawning state and an alarm is issued.

[0076] Step F, Fatigue Judgment Algorithm Design Module Based on Dynamic Decay Time Window: The algorithm design aims to monitor the controller's fatigue state in real time by dynamically adjusting the time window size, providing different detection sensitivities in the early and late stages of fatigue, thereby mitigating misjudgments caused by detection errors. A dynamic time window mathematical model is designed by integrating the controller's work scenario. Specifically, considering the controller's shift scheduling in actual work, three influencing factors are proposed: controller on-duty time, real-time air-to-ground communication load, and the number of fatigue events. A dynamic decay scheme for the time window is adopted to adjust the fatigue detection sensitivity in real time while ensuring the correlation of time-series information. Since the time window decay is a nonlinear process, after extensive experiments, a nonlinear least squares method is selected to fit the decay process to an exponential decay form, which helps to adapt to changes in fatigue state. The exponential decay function is a function in which the dependent variable decreases exponentially as the independent variable increases. It can be used to deal with time series data and model parameter decay problems. In physics, it is often used to describe the rate of kinetic energy decay of a moving object, such as the process by which the kinetic energy of an object continuously decays over time. Transferring the scenario, suppose that the fatigue level of the controller will increase with the real-time increase of on-duty time and voice frequency. The decay coefficient λ of the dynamic time window W is affected by the above two factors, as well as the bias term representing fatigue events (yawning, prolonged half-open drowsiness).

[0077] The specific formula is as follows:

[0078] λ=kp / (τ-0.1b), W=αe -λt +β,

[0079] Where p represents the total frequency of voice calls, which characterizes the real-time air-to-ground communication load; k is a scaling factor, which maps p appropriately, and here it is taken as 2e-6; τ is the decay rate; b is the fatigue event record value, and 0.1 is the mapping coefficient; t is the test time, in seconds / s; α is the function saturation value; β is the dynamic time window function bias term, representing the lower limit of the time window size. Before the experiment, the initial time window was set to 15 minutes, and the minimum time window was set to 2 minutes. Using an average FPS of 55 (adjusted according to actual conditions), the value of W was calculated to be 49500. By using the nonlinear least squares method for function fitting, the parameters α were obtained as 43544.18, τ as 5.96, and β as 6600. The ideal time window decay curve is shown in [reference needed]. Figure 3 This can achieve the effect of slow deceleration rate in the early stage of fatigue, faster deceleration rate in the middle stage, and the curve approaching an asymptote in the later stage, stabilizing at a small value. The initial parameter provides an important reference for the subsequent dynamic time window decay calculation, and can be fine-tuned based on the results.

[0080] The fatigue assessment method, specifically considering that the processing object of this fatigue assessment task is a real-time video stream sequence with temporal relationships, needs to simultaneously focus on the spatial dimension information of a single frame and the temporal dimension information between multiple frames. To address these issues and better measure fatigue state, a method is proposed using the ratio F of the number of blinks within a variable time window to the size of the time window. δ As the final criterion for fatigue assessment. Among them, F δ It is affected by on-duty hours, real-time voice volume, and the number of fatigue events. The specific formula is as follows:

[0081] F δ =μN blink / W,

[0082] Where N blink F represents the blink frequency within a dynamic time window; μ is the scale adjustment coefficient, set to 1000 here, used to amplify the value of this indicator to the same scale as the subsequent EEG fatigue indicator. Considering that the blink frequency of a person in a conscious state is usually between 15 and 20 times per minute, the F value is calculated by converting time to frame rate. δ The threshold for the indicator is set to 6.5. If the threshold is exceeded, the system will determine fatigue and issue a warning.

[0083] Step G, Algorithm effectiveness analysis module based on EEG signals: F calculated in step F δBased on the indicators, the method for processing EEG signal data synchronously acquired with facial video was specifically to collect four different frequency waves of EEG signals using a multi-channel physiological analyzer: delta waves, theta waves, alpha waves, and beta waves. Given that the controller's EEG signal collection covered multiple time periods, and the mental state presented at these different time points varied, to make theta and beta wave characteristics more comparable, the method of relative power spectrum analysis of EEG signals from the field of driver fatigue research was adopted, drawing on the concept of transfer learning, to analyze the controller's EEG signal data. This method can improve the objectivity and accuracy of the obtained fatigue indicators. Since the subjects in the experiment were not in a deep sleep state, and delta waves typically appear in adults under anesthesia, in a deep sleep, or in a state of hypoxia, the ratio of the relative power spectrum of slow waves to fast waves was used as the fatigue state judgment indicator F, without considering delta waves. ε The specific formula is as follows

[0084] F ε =P′ θ / (P′ α +P′ β ),

[0085] Where P′ α For the relative power spectrum of the α wave, P′ β For the relative power spectrum of the β wave, P′ θ The relative power spectrum of the theta wave is given. The algorithm's effectiveness verification method considers EEG signals as the "gold standard" for contactless fatigue monitoring; therefore, by analyzing facial video data and EEG data, the fatigue index F is obtained. δ With F ε And align the two indicator data along the time dimension, see Figure 4 Experiments were conducted using facial video data. Data analysis yielded curves showing the increase in the number of voice calls and the actual changes in the size of the dynamic time window. Specific single-sample (subject) data were also provided. Figure 5 As can be observed from the graph, the number of voice calls gradually increases during the control process; simultaneously, within the 0-20 minute time range, the time window size remains stable (the duration varies slightly depending on the single-frame processing speed). This period is used for F... δ The indicator data is initially accumulated. After data accumulation, the size of the dynamic time window, influenced by the accumulation period and the frequency of the speech, experiences a small, sharp decrease, followed by a non-linear decreasing trend, with the lower limit of the window set at 1 minute. The relevant parameters within the algorithm are fine-tuned based on the ideal parameters described above. Due to the characteristics of the algorithm, the F-value during the data accumulation phase... δ The value inevitably shows a monotonically increasing trend, therefore this time period's data is ignored in subsequent analyses. SPSS was chosen to be used to analyze the F-values ​​of 30 samples. δ With F εBivariate correlation analysis was performed on each data point individually. Figure 6 The data distribution of fatigue index for a single subject was presented intuitively in the form of a violin plot. By observing the median position of the two sets of data and the width of the violin plot, it can be clearly found that the two variables show significant similarity in the central trend, and the data are relatively evenly distributed on the horizontal axis without obvious skewness or clustering.

[0086] Table 2 shows F δ With F ε Correlation analysis results

[0087]

[0088]

[0089] In Table 2, ** indicates a significant correlation at the 0.01 level (two-tailed).

[0090] As can be seen from the complete calculation results in Table 2, in the bivariate cross-correlation analysis of 30 samples, the Pearson correlation coefficient generally ranged from 0.462 to 0.785, and the Sig two-tailed significance test results were all at the 0.01 level, indicating significant correlation. Therefore, this invention verifies its effectiveness in real-time fatigue monitoring based on facial feature information, approaching the results of fatigue monitoring based on EEG data, and providing relevant insights for subsequent research on non-contact fatigue monitoring for air traffic controllers.

Claims

1. A method for controller fatigue monitoring based on facial features, implemented based on a computer or an edge computing development board, characterized in that, In the computer or development board, set up face detection module, face key point detection module, eye and mouth aspect ratio calculation module, individual face feature parameter calibration module, severe fatigue behavior detection module, fatigue discrimination algorithm design module based on dynamic attenuation time window, algorithm effectiveness analysis module based on electroencephalogram signal, steps are as follows: Step A, face detection: First, face detection is performed according to the input single frame image, and face confidence and detection box coordinates are returned. Non-maximum suppression is performed on the detection result, and it is limited that only one face is detected in the picture. Step B, face key point detection: After determining the face detection box coordinates in step A, the original picture is cropped according to the coordinates, and the cropped image is input into the face 468 key point detection model to return the key point coordinate information. Step C, eye and mouth aspect ratio calculation: Through the three-dimensional coordinate information of the face 468 key points obtained by using the model inference in step B, the three-dimensional Euclidean distance is calculated, and the aspect ratio function of the eyes and mouth is further reconstructed. Step D, threshold calibration for individual face feature difference: According to the eye and mouth aspect ratio calculation method in step C, select open eye, closed eye, half open eye, open mouth and closed mouth 5 state images for each controller to calculate and calibrate the corresponding EAR_3D and MAR_3D threshold values of each subject, wherein EAR_3D is the eye aspect ratio and MAR_3D is the mouth aspect ratio. Step E, severe fatigue behavior detection: According to the EAR_3D and MAR_3D threshold values obtained in step D, the eye and mouth states of the controller are determined, and the results are written into a time series array. According to the real-time inference performance of the computer or edge computing development board, the continuous frame threshold is set to achieve the effect of severe fatigue behavior detection. Step F, fatigue discrimination algorithm design based on dynamic attenuation time window: According to the fatigue behavior and eye and mouth state information obtained in step E, a dynamic attenuation time window mathematical model is designed by combining the controller's working scene. The controller's blink frequency in the dynamic time window is calculated, and the fatigue threshold is combined to discriminate fatigue and simultaneously alarm. Step G, algorithm effectiveness analysis based on electroencephalogram signal: Using SPSS, the electroencephalogram signal processing results of the controller are analyzed for correlation with the face video processing results in step F to verify the effectiveness of the algorithm. Through the above steps, the fatigue state of the controller is monitored, and an alarm is given when the fatigue discrimination condition is met to prevent air traffic control safety incidents caused by physical fatigue.

2. The face feature based controller fatigue monitoring method of claim 1, wherein, The face detection model part in step A combines depth separable convolution with Vision Transformer to build a new lightweight feature extraction network, simplifies the encoding method of global feature extraction, and balances the inference accuracy and speed of the model.

3. The face feature based controller fatigue monitoring method of claim 1, wherein, The face key point detection algorithm in step B is based on the second stage inference of the face detection result in step A. The Attention Mesh face 468 key point detection algorithm based on Google's Face Mesh improvement is used to obtain the three-dimensional spatial coordinate information of the face features.

4. The face feature based controller fatigue monitoring method of claim 1, wherein, The eye-mouth aspect ratio calculation method in step C adopts human face three-dimensional coordinate information, further reconstructs the eye-mouth aspect ratio function by calculating three-dimensional Euclidean distance, and the specific three-dimensional space Euclidean distance calculation formula is where (x i , y i , z i ) and (x j , y j , z j ) are the coordinate information of two points in space, respectively; 10 key points of the eye are selected for three-dimensional aspect ratio function reconstruction, the aspect ratios of the left and right eyes are calculated respectively and then averaged, and the final result EAR_3D is taken as the basis for evaluating the eye state characteristics. The left eye aspect ratio is EAR_3D left = P i＝160，j＝144 + P i＝159，j＝145 + P i＝158，j＝153 + P i＝157，j＝154 / 4 x P i＝33，j＝133 , Right eye aspect ratio EAR_3D right = P i＝384，j＝381 + P i＝385，j＝380 + P i＝386，j＝374 + P i＝387，j＝373 / 4 x P i＝363，j＝263 , Eye aspect ratio In the aspect of obtaining mouth feature data, eight key points are selected for three-dimensional aspect ratio function reconstruction, and the specific formula is MAR_3D = P i＝81，j＝178 + P i＝13，j＝14 + P i＝311，j＝402 / 3 x P i＝62，j＝293 .

5. The face feature based controller fatigue monitoring method of claim 1, wherein, The threshold calibration method in step D first selects five state images of open eyes, closed eyes, half-open eyes, open mouth, and closed mouth from the controller face video data; secondly, uses the key point detection model and the eye-mouth aspect ratio calculation formula to determine the EAR_3D and MAR_3D threshold values corresponding to each subject, and determines the above five states by the threshold range.

6. The face feature based controller fatigue monitoring method of claim 1, wherein, The method for determining the state of the controller's eyes and mouth in step E uses the EAR_3D and MAR_3D threshold values determined for individual controllers to adjust the state determination interval; distinguishes five states of open eyes, closed eyes, half-open eyes, open mouth, and closed mouth; sets a continuous frame parameter to identify and alarm three severe fatigue behaviors of long-time closed eyes, long-time half-open eyes, and yawning.

7. The face feature based controller fatigue monitoring method of claim 6, wherein, The method for identifying the three severe fatigue behaviors of long-time closed eyes, long-time half-open eyes, and yawning is as follows: 1) Long-time closed eyes: three consecutive frames less than the intermediate threshold value are determined as blinking, and blinking counting is performed; 50 consecutive frames less than the intermediate threshold value are determined as long-time closed eye state, and alarm prompt is performed; 2) Long-time half-open eyes: a single frame greater than the intermediate threshold value is determined as open eye state, and 50 consecutive frames of eye aspect ratio in the intermediate threshold value and the numerical interval of the intermediate threshold value are determined as half-open eye sleep state, and alarm prompt is performed; 3) Yawning: three consecutive frames greater than the closed mouth aspect ratio value are determined as open mouth state, and open mouth counting is performed; 120 consecutive frames of mouth aspect ratio greater than the open mouth threshold value are determined as yawning state, and alarm prompt is performed.

8. The face feature based controller fatigue monitoring method of claim 1, wherein, The method for fusing the dynamic decay time window mathematical model of the controller working scene design in step F is to introduce the controller on-duty time, real-time air-ground communication load, and the number of fatigue events as influence factors; adopt a time window dynamic decay scheme, use a nonlinear least squares method to fit the decay process to an exponential decay form; It is assumed that the fatigue degree of the controller will increase with the real-time increase of the on-duty time and the speech frequency, and the decay coefficient λ of the dynamic time window W is affected by the above two factors and the bias term representing the fatigue event; The specific formula is λ = kp / (τ - 0.1b), W = αe -λt + β, Wherein p represents the total frequency of voice, the index represents the real-time air-ground communication load; k is a scaling factor, which reasonably maps p, and 2e-6 is taken here; tau is the decay rate; b is the fatigue event record value, and 0.1 is the mapping coefficient; t is the test time, the unit is second / s; alpha is the function saturation value; beta is the dynamic time window function bias term, which represents the lower limit of the time window size, the initial time window is set to 15 minutes before the experiment starts, the minimum time window is set to 2 minutes, the average FPS is 55, and the value of W is 49500; by using the nonlinear least squares method for function fitting, the parameters alpha is 43544.18, tau is 5.96, and beta is 6600; the initial parameters are prior values, which provide an important reference for subsequent dynamic time window decay calculation, and the results can be fine-tuned.

9. The face feature based controller fatigue monitoring method of claim 1, wherein, The fatigue discrimination method of step F is to use the ratio of the number of blinks in a variable time window to the size of the time window F δ As the final fatigue discrimination basis, where F δ Affected by the on-duty time, real-time speech frequency, and the number of fatigue events, the specific formula is F δ = μN blink / W, Where N blink is the blink frequency within the dynamic time window; μ is the scale adjustment coefficient, which is 1000 here, used to amplify the numerical value of the index to the same scale as the subsequent electroencephalogram fatigue index, and F δ The index threshold is set to 6.5, and after exceeding the threshold, the system will make a fatigue determination and give a warning prompt.

10. The face feature based controller fatigue monitoring method of claim 1, wherein, The method for analyzing effectiveness of the algorithm based on the brain electrical signals in step G is based on the F δ On the basis of the index, four different frequency waves of the brain electrical signals, i.e., δ wave, θ wave, α wave and β wave, are collected by using a multi-lead physiological instrument, and the relative power spectrum ratio of the slow wave and the fast wave is taken as the fatigue state judgment index F ε , and the specific formula is F ε = P' θ / (P' α + P' β ), Where P′ α For the relative power spectrum of the α wave, P′ β For the relative power spectrum of the β wave, P′ θ The relative power spectrum of the theta wave; the two index data F δ With F ε Alignment was performed along the time dimension, and F-squared values ​​were analyzed for 30 samples using SPSS. δ With F ε The data were subjected to bivariate correlation analysis one by one, and the effectiveness of the controller fatigue judgment method based on facial features was judged by observing the degree of correlation.

Images