Pre-post air traffic controller fatigue objective prediction method and system

Abstract

The invention relates to the technical field of civil aviation air traffic management, and discloses a pre-post air traffic controller fatigue objective prediction method and system, and the method comprises the steps: presenting a mental movement alertness task to a user; the method comprises the following steps: in the process of executing a mental movement alertness task by a user, acquiring eye image data containing the user through image acquisition equipment, and recording the response time and error rate of the user to the mental movement alertness task; calculating to obtain an eyelid closure parameter based on the eye image data; calculating a reaction time parameter and an error rate parameter based on the reaction time and the error rate; and determining a comprehensive fatigue value of the user based on a weighted combination of the eyelid closure parameter, the response time parameter and the error rate parameter. The invention aims to provide an objective fatigue prediction method and system for a pre-post air traffic controller so as to overcome the defects that a traditional subjective scale is insufficient in reliability, a standard objective test consumes too long time and is not suitable for pre-class rapid screening.

Description

Technical Field

[0001] The present invention relates to the technical field of civil aviation air traffic management, and in particular to a method and system for objectively predicting fatigue of air traffic controllers before taking up their posts. Background Art

[0002] Air traffic control is a critical component of ensuring national air defense security and the efficient operation of civil aviation. Its safety relies heavily on air traffic controllers (hereinafter referred to as "controllers") maintaining high levels of alertness and cognitive ability while performing their duties. However, due to the shift work required of controllers, particularly during the night, their circadian rhythms are easily disrupted, leading to fatigue. Fatigue significantly reduces controllers' alertness, decision-making ability, and reaction speed, creating a significant risk of control errors and even serious safety incidents.

[0003] Therefore, effective monitoring and management of fatigue status of practitioners has become an important technical direction in the field of aviation safety. Currently, related technologies mainly focus on real-time monitoring of the status of personnel during the execution of tasks.

[0004] For example, patent application CN113994403A discloses an in-flight risk management system for pilots. This system pre-generates a predicted fatigue profile based on the pilot's historical information (such as rest and wake-up activity) and current flight information. During flight, the system continuously and passively collects real-time fatigue indicator values ​​through sensors in the cockpit (such as eye sensors), and uses this real-time data to update and correct the original fatigue prediction profile. When the updated fatigue level exceeds the risk threshold, the system can take intervention measures such as warnings or automatic control. This solution provides a closed-loop control approach for dynamic fatigue risk management during flight missions.

[0005] For example, patent application CN116504105A discloses a human-machine collaborative safety monitoring system for air traffic control operations. This system includes an AI supervisor unit that uses physiological data collection equipment to conduct real-time assessments of the controller's control effectiveness. It also generates machine-generated decision-making solutions based on air traffic control situation data to assist the controller. This solution aims to improve the safety and efficiency of air traffic control operations through human-machine collaboration.

[0006] However, in the existing technical field, how to quickly, objectively, and effectively screen key personnel (especially air traffic controllers) for fatigue before they start work (pre-duty) remains a technical challenge that needs to be solved. Existing methods mainly fall into two categories: one is subjective rating scales, such as the Stanford Sleepiness Scale (SSS). Although this method is simple, its results are easily influenced by factors such as the test subject's subjective will, emotions, and interests, and lack objectivity and reliability. The other is laboratory-level objective testing methods, such as the standard psychomotor vigilance test (PVT) or electroencephalogram (EEG) test. Although objective and accurate, these methods are often time-consuming, require complex equipment, or have high environmental requirements, making them unsuitable for large-scale, efficient, rapid screening of air traffic control units before they start work.

[0007] Therefore, this field urgently needs a new technical solution that can overcome the above-mentioned defects and realize objective, accurate, fast and convenient detection of the fatigue status of controllers before they take up their posts, thereby preventing safety risks caused by fatigue at the source. Summary of the Invention

[0008] This invention aims to address the existing technical problem of a lack of rapid, objective fatigue detection methods suitable for pre-shift air traffic controllers. Specifically, it aims to provide a method and system for objectively predicting air traffic controller fatigue before taking up their shifts, overcoming the drawbacks of traditional subjective rating scales, which are unreliable, and standard objective tests, which are time-consuming and unsuitable for rapid pre-shift screening.

[0009] To solve the above technical problems, the present invention provides an objective prediction method for air traffic controller fatigue before taking up a post, the method comprising the following steps: presenting a psychomotor vigilance task to a user; During the process of the user performing the psychomotor vigilance task, collecting eye image data of the user by an image acquisition device, and recording the reaction time and error rate of the user to the psychomotor vigilance task; Calculating an eyelid closure parameter based on the eye image data; Calculating a reaction time parameter and an error rate parameter based on the reaction time and the error rate; and Based on a weighted combination of the eyelid closure parameter, the reaction time parameter, and the error rate parameter, a comprehensive fatigue value of the user is determined.

[0010] Preferably, the psychomotor vigilance task comprises: A target letter sequence is randomly selected from a preset letter set; and a series of letters are displayed on the screen in sequence, and the user is required to respond when the target letter sequence appears.

[0011] Preferably, the step of calculating an eyelid closure parameter includes: using a preset neural network model to process the eye image data to determine the eyelid closure state.

[0012] Preferably, the step of determining the comprehensive fatigue value of the user is specifically: calculating the comprehensive fatigue value F by the following formula: ; Wherein, P is the normalized eyelid closure parameter, R is the normalized reaction time parameter, E is the normalized error rate parameter, and A1, A2 and A3 are preset weight coefficients.

[0013] Preferably, the duration of the psychomotor vigilance task is 5 minutes.

[0014] Preferably, the eyelid closure parameter is determined based on the number of image frames in which the eyelid closure degree exceeds 70%; and before performing the weighted combination, the method also includes a step of normalizing the eyelid closure parameter and the reaction time parameter.

[0015] The present invention also provides an objective prediction system for fatigue of air traffic controllers before taking up their posts, comprising: a presenting unit, configured to present a designed psychomotor vigilance task to a user, wherein the execution of the psychomotor vigilance task is used to guide the user's gaze toward an image acquisition device; a data acquisition unit, working in conjunction with the presentation unit, for acquiring, through the image acquisition device, eye image data of the user while the user performs the psychomotor vigilance task, and obtaining the user's reaction time and error rate for the psychomotor vigilance task; a processing unit, connected to the data acquisition unit, and configured to: determining an eyelid closure parameter based on the eye image data; determining a reaction time parameter and an error rate parameter based on the reaction time and the error rate; and Based on a preset neural network model, the eyelid closure parameter, the reaction time parameter, and the error rate parameter are processed to determine the user's comprehensive fatigue value.

[0016] Beneficial effects 1. This invention eliminates assessment scales susceptible to subjective factors and instead conducts a comprehensive assessment based entirely on objectively collected physiological indicators (eyelid closure) and behavioral metrics (reaction time, error rate). By integrating multi-dimensional data, the test results more accurately and reliably reflect the user's true fatigue state.

[0017] 2. The entire testing process is designed to be completed in a short time, greatly improving testing efficiency. This "rapid screening" feature allows it to be seamlessly integrated into the controller's pre-job preparation process, providing high field application value and operability, and resolving the technical pain point of traditional objective testing methods that are too time-consuming.

[0018] 3. This invention uniquely integrates cognitive task design with data collection requirements. By actively guiding users to focus their attention on the screen through a designed psychomotor alertness task, the image acquisition device can consistently acquire high-quality facial and eye images. This collaborative design effectively addresses the low data quality commonly encountered in non-contact, portable testing without adding additional hardware constraints (such as head-mounted devices).

[0019] 4. The present invention ultimately outputs a single, quantified, comprehensive fatigue value. This result is intuitive and clear, allowing managers to directly compare it with preset safety thresholds, thereby providing a clear and decisive basis for decision-making on whether to allow controllers to take up their posts, simplifying the fatigue risk management process. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a flow chart of an objective prediction method for fatigue of pre-job air traffic controllers provided in a preferred embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of an objective prediction system for pre-job air traffic controller fatigue provided in a preferred embodiment of the present invention; Figure 3 This is a schematic diagram of a curve showing a change in fatigue value versus test time provided in an experimental example of the present invention; Figure 4 This is a schematic diagram of fatigue detection results provided in an experimental example of the present invention. DETAILED DESCRIPTION

[0021] To make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that the specific embodiments herein are only used to explain the present invention and are not intended to limit the scope of the present invention.

[0022] Example 1 Reference Figure 1 This embodiment provides an objective fatigue prediction method for pre-job air traffic controllers (hereinafter referred to as "users"). This method aims to address the inaccuracies of existing subjective assessment methods and the complex, expensive, and time-consuming nature of objective physiological testing equipment, providing a fast, objective, and easily deployable fatigue detection solution.

[0023] S101: Presenting a psychomotor vigilance task (PVT) to a user.

[0024] In this embodiment, the device presenting the task may be a computing device with a screen, such as a laptop, desktop, tablet computer, or smartphone.

[0025] In an optional embodiment, the psychomotor vigilance task is specifically designed to more effectively assess cognitive abilities related to air traffic control work (e.g., alertness, working memory, and judgment). In a preferred embodiment, the task is designed as a letter sequence recognition task: first, a target letter sequence consisting of two letters is randomly selected from a preset character set (e.g., containing 10 different English letters); then, a letter randomly selected from the character set is displayed on the screen in sequence at preset time intervals (e.g., every 1000 milliseconds), with each letter displayed on the screen for 100 milliseconds. The task requires the user to maintain attention and respond as quickly as possible when the two letters displayed consecutively on the screen exactly form the preset target letter sequence.

[0026] It should be understood that the above task design is only an example. In other embodiments, the stimuli may not be limited to letters, but may also be numbers, graphics, colors, or a combination thereof. The length of the target sequence may also be limited to 2, but may be 3 or more. The user's response method is also not limited to pressing keys, but may be clicking the mouse, touching the screen, or issuing voice commands. Through this task design, the user's sight and attention can be effectively guided and focused on the central area of ​​the screen, which creates favorable conditions for the subsequent high-quality acquisition of eye image data.

[0027] S102: While the user is performing the psychomotor vigilance task, an image acquisition device is used to acquire eye image data of the user, and the reaction time and error rate of the user to the psychomotor vigilance task are recorded.

[0028] This step is performed simultaneously with S101. The image acquisition device can be integrated into the computing device performing the task, such as the front-facing camera of a laptop. This device continuously captures images of the user's face, particularly the eye area, forming a video stream or a series of image frames. Simultaneously, the program running the task accurately records the user's reaction time to each target sequence and counts errors, such as missed responses (no response to the target sequence) and false responses (responding to a non-target sequence).

[0029] In a preferred embodiment, in order to improve detection efficiency while ensuring detection accuracy, the duration of the psychomotor vigilance task can be set to 5 minutes. Figure 3 As shown, user fatigue data stabilizes around three minutes after the test begins. Setting the timeframe to five minutes ensures data validity while also meeting the needs of rapid pre-job testing. When processing data, consider excluding the first minute of data to eliminate the impact of the user's initial adaptation period.

[0030] S103: Calculate an eyelid closure parameter based on the eye image data.

[0031] In this embodiment, the processing unit (eg, a processor in a computing device) analyzes the eye image data collected in S102 to quantify the user's eye fatigue characteristics.

[0032] In one specific implementation, this step can utilize a pre-trained deep learning model, such as a convolutional neural network (CNN) or a multi-task CNN model, to process the eye image data. This model can automatically locate the eye region within each image frame and accurately calculate the degree of eyelid closure. The implementation details of this technology can be designed with reference to existing technologies in the field and are not further specified by this invention.

[0033] The eyelid closure parameter can be specifically a value calculated based on the PERCLOS (Percentage of Eyelid Closure Over the Pupil Over Time) indicator, such as the P70 indicator, which is the percentage of time when the eyelid closure exceeds a preset threshold (e.g., 70%) in the total monitoring time. Specifically, the eyelid closure parameter P can be calculated using the following formula: ; Among them, FPs represents the frame rate of image acquisition (Frames Per Second), t represents the total length of the video (seconds), P i represents the coverage of the pupil by the eyelid in the i-th frame (i.e., the degree of eyelid closure). sign{·} is an indicator function that takes the value 1 when its internal condition is true and 0 when it is false. The result P here is a normalized eyelid closure parameter.

[0034] S104: Calculate a reaction time parameter and an error rate parameter based on the reaction time and the error rate.

[0035] In this embodiment, the processing unit further needs to quantify the behavior data recorded in S102.

[0036] The reaction time parameter (R) can be calculated based on the time it takes for the user to correctly react each time and normalized. For example, it can be calculated using the following formula: ;in is the average of all correct reaction times of the user; is a preset reaction time threshold (e.g., 500 milliseconds) that represents the typical reaction time in a normal state of alertness.

[0037] The error rate parameter (E) can be calculated based on the number of missed reports and false reports by the user in the task. For example, it can be calculated using the following formula: ; Among them, N o Indicates the number of omitted target letters; N e Indicates the number of wrong letters selected; N t Represents the total number of times the target letter sequence appears in the entire task.

[0038] S105: Determine a comprehensive fatigue value of the user based on a weighted combination of the eyelid closure parameter, the reaction time parameter, and the error rate parameter.

[0039] This step aims to integrate information from the three dimensions of physiology, cognition, and behavior to obtain a comprehensive and objective fatigue assessment result.

[0040] In a specific embodiment, the comprehensive fatigue value F can be calculated by the following linear weighted fusion model: ; Where P and R are the normalized eyelid closure and reaction time parameters (in the previous steps, the calculation method of P and R already includes the normalization concept), and E is the error rate parameter. A1, A2, and A3 are preset weight coefficients, which can be calibrated based on a large amount of experimental data to reflect the contribution of different parameters to the comprehensive fatigue assessment. For example, in a preferred configuration, the weight coefficients can be set as: A1=6, A2=2, A3=2. This shows that in this application scenario, the eyelid closure feature is considered to be the core indicator for assessing fatigue.

[0041] The comprehensive fatigue value F can be set within an intuitive range, such as 0-10, with higher scores indicating more severe fatigue. Furthermore, managers can make decisions based on this value. For example, a safety threshold of 5.0 could be set. When the F value is below 5.0, the controller is in good condition and suitable for duty. When the F value is above 5.0, the system issues an alarm, indicating that the controller is at high fatigue risk and recommends that they rest and adjust before returning to duty.

[0042] The method in this embodiment utilizes a modified PVT task that combines assessments of alertness, memory, and judgment. This not only effectively captures behavioral cognitive data such as reaction time and error rate, but also ensures that the user maintains a constant orientation toward the camera through the task's guidance, thereby acquiring high-quality eye image data. This method then processes the images using an advanced CNN model to obtain an objective physiological fatigue metric (eyelid closure). This is then combined with behavioral metrics (reaction time and error rate) through a weighted fusion model to ultimately output a single, quantified, comprehensive fatigue value.

[0043] Example 2 This embodiment provides an objective fatigue prediction system for pre-job air traffic controllers, which is configured to execute the method described in Example 1.

[0044] Reference Figure 2 The system 200 may be an independent hardware device or a software system installed on a general-purpose computing device (such as a laptop). The system 200 may include: A presentation unit 210 is configured to present a designed psychomotor vigilance task to a user. The presentation unit may be a display screen. As described in Example 1, the execution of the task is used to guide the user's gaze toward an image acquisition device to ensure data acquisition quality.

[0045] A data acquisition unit 220 works in conjunction with the presentation unit 210. This unit is used to capture eye image data of the user through an image acquisition device (e.g., a camera) while the user is performing a task, and obtain the user's response to the task (e.g., via a keyboard, mouse, or touch screen) to record reaction time and error rate.

[0046] A processing unit 230 is connected to the data acquisition unit 220. The processing unit is the core of the system and can be implemented by one or more processors (such as CPU, GPU, DSP), or dedicated chips (such as ASIC, FPGA). The processing unit 230 is configured to perform the following operations: An eyelid closure parameter is determined based on the eye image data received from the data acquisition unit 220. In one implementation, the processing unit 230 loads and runs a pre-trained CNN model to perform this calculation.

[0047] Based on the reaction time and error rate data received from the data acquisition unit 220 , a reaction time parameter and an error rate parameter are determined.

[0048] Based on a preset neural network model (for example, the linear weighted model described in Example 1), the eyelid closure parameter, the reaction time parameter, and the error rate parameter are processed to determine the user's comprehensive fatigue value.

[0049] A storage unit (not shown) is used to store preset program codes, pre-trained CNN models, preset weight coefficients (such as A1, A2, A3), temporary data generated during the detection process, and the final fatigue value result.

[0050] In an optional embodiment, the processing unit 230 of the system 200 is further configured to implement the various optional schemes and specific calculation methods described in Example 1, for example, customization of the psychomotor vigilance task, calculation of eyelid closure using a CNN model, calculation of the comprehensive fatigue value using a specific weighted formula, controlling the task's continuous detection time to 5 minutes, and using a specific P70 indicator and normalization method.

[0051] Those skilled in the art will understand that the various units of the above-mentioned system 200 can be connected via a bus or other means, and the processing unit 230 calls the programs and data in the storage unit to complete the fatigue prediction described in the present invention through interaction with the presentation unit 210 and the data acquisition unit 220.

[0052] In addition, the method and system of the present invention are not only suitable for air traffic controllers, but can also be promoted and applied to other safety-critical fields that require high operator alertness, such as long-distance freight drivers, pilots, heavy machinery operators, surgeons, etc., and have broad application prospects.

[0053] Experimental example In order to verify the effectiveness and sensitivity of the objective detection method and system for pre-job air traffic controller fatigue disclosed in the present invention, the following comparative experiments were conducted.

[0054] 1. Experimental Design and Procedure This experiment was conducted at a regional air traffic control center, recruiting several healthy, licensed air traffic controllers as volunteers. The volunteers were divided into two groups: 1. Pre-duty group (equivalent to the non-fatigue group): This group consists of controllers who will start their shifts around 18:00. These controllers have had a good rest before taking up their posts and are in a state of high energy.

[0055] 2. Post-duty group (equivalent to the fatigue group): This group consists of controllers who have just completed a 6-hour shift (for example, from 12:00 to 18:00) and are waiting for handover. These controllers have experienced the actual workload and safety pressures during their time on duty.

[0056] Both groups of controllers underwent a complete fatigue test using the system 200 described in Example 1. The specific process was as follows: Each volunteer independently performed a 5-minute psychomotor vigilance task based on letter sequence recognition. During this time, the data acquisition unit 220 simultaneously recorded their eye image data, reaction time, and error rate. After the task was completed, the processing unit 110 calculated each volunteer's comprehensive fatigue value F in real time based on preset weighting coefficients (in this experiment, illustratively set to A1=6, A2=2, and A3=2).

[0057] 2. Experimental Results and Analysis like Figure 4 As shown, the statistical results of the experiment are as follows: 1. Pre-job group: The mean value of the comprehensive fatigue value (F) was 2.0, with a standard deviation (SD) of 0.3. The data showed that the values ​​in this group were relatively low and concentrated.

[0058] 2. Post-job group: The mean value of the comprehensive fatigue value (F) was 3.7, with a standard deviation (SD) of 1.5. The data showed that the values ​​in this group were significantly higher, with a large degree of dispersion.

[0059] Independent sample t-test analysis showed that there was a statistically significant difference between the two groups of data (p=0.011<0.05).

[0060] Further analysis revealed that the standard deviation of the post-duty group (1.5) was much greater than that of the pre-duty group (0.3). This is reasonably explained by the fact that individual fatigue levels in the post-duty group varied significantly due to the specific workload, traffic volume, and safety pressures they faced during their shifts, resulting in a wider distribution of fatigue values. In contrast, the pre-duty group, after resting, had a more consistent physiological and mental state, resulting in a more concentrated distribution of fatigue values.

[0061] 3. Experimental Conclusion The results of this validation experiment clearly demonstrate that the detection method and system provided by the present invention can effectively distinguish between controllers in a normal resting state and those experiencing post-duty fatigue. The average fatigue value of the post-duty group (3.7) was significantly higher than that of the pre-duty group (2.0), demonstrating that the comprehensive fatigue value F calculated by the present invention has a high positive correlation with the user's actual physiological fatigue level. Therefore, the experimental data strongly confirms that the present invention can serve as a rapid, objective, and sensitive detection tool for pre-duty fatigue risk screening of air traffic controllers, thereby verifying the practicality and significant beneficial effects of the present invention.

[0062] The basic principles, main features, and advantages of the present invention are shown and described above. Those skilled in the art should understand that the present invention is not limited to the foregoing embodiments. The foregoing embodiments and descriptions are merely illustrative of the principles of the present invention. Various changes and modifications may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and modifications are intended to fall within the scope of the present invention. The scope of protection claimed in the present invention is defined by the appended claims and their equivalents.

Claims

1. An objective prediction method for air traffic controller fatigue before taking up a post, characterized by: include: presenting a psychomotor vigilance task to a user; During the process of the user performing the psychomotor vigilance task, collecting eye image data of the user by an image acquisition device, and recording the reaction time and error rate of the user to the psychomotor vigilance task; Calculating an eyelid closure parameter based on the eye image data; Based on the reaction time and the error rate, calculating a reaction time parameter and an error rate parameter; as well as Based on a weighted combination of the eyelid closure parameter, the reaction time parameter, and the error rate parameter, a comprehensive fatigue value of the user is determined.

2. The objective prediction method for pre-job air traffic controller fatigue according to claim 1, characterized in that: The psychomotor vigilance task included: A target letter sequence is randomly selected from a preset letter set; and a series of letters are displayed on the screen in sequence, and the user is required to respond when the target letter sequence appears.

3. The objective prediction method for pre-job air traffic controller fatigue according to claim 1, characterized in that: The step of calculating an eyelid closure parameter includes: using a preset neural network model to process the eye image data to determine the eyelid closure state.

4. The objective prediction method for pre-job air traffic controller fatigue according to claim 1, characterized in that: The step of determining the comprehensive fatigue value of the user is specifically as follows: calculating the comprehensive fatigue value F by the following formula: ; Wherein, P is the normalized eyelid closure parameter, R is the normalized reaction time parameter, E is the normalized error rate parameter, and A1, A2 and A3 are preset weight coefficients.

5. The objective prediction method for pre-job air traffic controller fatigue according to claim 1, characterized in that: The psychomotor vigilance task lasted for 5 minutes.

6. The objective prediction method for pre-job air traffic controller fatigue according to claim 1 or 2, characterized in that: The eyelid closure parameter is determined based on the number of image frames in which the eyelid closure degree exceeds 70%. Moreover, before performing the weighted combination, the method further includes a step of normalizing the eyelid closure parameter and the reaction time parameter.

7. Pre-job air traffic controller fatigue objective prediction system, characterized by: include: a presenting unit, configured to present a designed psychomotor vigilance task to a user, wherein the execution of the psychomotor vigilance task is used to guide the user's gaze toward an image acquisition device; a data acquisition unit, working in conjunction with the presentation unit, for acquiring, through the image acquisition device, eye image data of the user while the user performs the psychomotor vigilance task, and obtaining the user's reaction time and error rate for the psychomotor vigilance task; a processing unit, connected to the data acquisition unit, and configured to: determining an eyelid closure parameter based on the eye image data; determining a reaction time parameter and an error rate parameter based on the reaction time and the error rate; and Based on a preset neural network model, the eyelid closure parameter, the reaction time parameter, and the error rate parameter are processed to determine the user's comprehensive fatigue value.

Images