A pilot visual fatigue detection system and method based on artificial intelligence

Abstract

The application discloses a kind of pilot visual fatigue detection system and method based on artificial intelligence, belong to flight safety monitoring technical field, first by image sensor obtains pilot face video, face positioning and tracking are carried out by neural network, and then blink frequency and critical flicker fusion frequency are extracted from eye area, and head posture motion is analyzed;After the fusion of multidimensional physiological data and behavior characteristics, the initial fatigue index is obtained by inputting the pre-trained visual fatigue evaluation model;Then, in combination with the individual historical fatigue data of the pilot and the current flight task load, the initial fatigue index is dynamically calibrated to generate an optimized visual fatigue score;Finally, according to the score level, a graded warning is started, a personalized and forward-looking fatigue state intervention is realized, and flight safety is effectively improved.

Description

Technical Field

[0001] This invention relates to the field of flight safety monitoring technology, specifically to an artificial intelligence-based pilot visual fatigue detection system and method. Background Technology

[0002] In the field of aviation flight safety, pilots' workload and mental state are core elements in ensuring flight safety. Among them, visual fatigue caused by prolonged, high-concentration tasks directly reduces pilots' situational awareness and operational reaction ability. Therefore, achieving real-time and accurate monitoring of pilots' visual fatigue status is of great significance for proactively preventing safety hazards caused by human factors.

[0003] Currently, most technologies applied to fatigue monitoring focus on the acquisition and analysis of single-type physiological signals. For example, tracking eye movements using image processing technology or monitoring physiological indicators such as heart rate variability using wearable devices are effective in controlled environments, but their applicability in real dynamic flight scenarios faces challenges. Environmental vibrations, acceleration overload, and necessary head movements during flight can significantly interfere with single signal sources, leading to insufficient stability in feature extraction and making it easy to make biases in fatigue state identification. Furthermore, due to the lack of effective correlation and fusion of multi-dimensional information, existing technologies struggle to distinguish between feature changes caused by physiological fatigue and data fluctuations caused by normal operational behavior. At the same time, most systems focus on snapshot-style judgments of instantaneous states, failing to correlate individual fatigue accumulation patterns of pilots with task loads at different flight stages. This static assessment model makes it difficult to achieve early warning and personalized intervention for fatigue risks. Summary of the Invention

[0004] The purpose of this invention is to provide an artificial intelligence-based pilot visual fatigue detection system and method to solve the problems mentioned in the background art.

[0005] To address the aforementioned technical problems, this invention provides the following technical solution: a pilot visual fatigue detection method based on artificial intelligence, comprising:

[0006] S1. The system continuously acquires facial video streams of the pilots through image sensors deployed in the cockpit, performs facial detection and key point localization on the video frames using a pre-trained convolutional neural network model, and continuously tracks the facial region status based on the Kalman filter algorithm.

[0007] S2. Based on the located key facial feature points, segment the binocular region image sequence from the video frame, perform time-frequency domain analysis on the sequence, and extract physiological indicators characterizing visual function, including the critical flash fusion frequency estimate calculated based on the image brightness change sequence, and the blink frequency obtained by detecting changes in eyelid opening and closing; at the same time, calculate the head posture angle and its motion frequency based on the spatial position changes of facial feature points.

[0008] S3. The extracted critical flash fusion frequency estimate, blink frequency, and head posture angle data are combined to form a multi-dimensional feature vector, which is then input into the pre-trained visual fatigue assessment model to output a preliminary visual fatigue index.

[0009] S4. Access the pilot's historical flight database, retrieve the historical fatigue index sequence and corresponding flight status parameters of the pilot in different flight stages within the past preset period, analyze the individual fatigue evolution pattern, establish a personalized baseline, and dynamically calibrate the initial score based on the matching degree between the current flight stage and the historical pattern to generate an optimized visual fatigue score that integrates the current physiological response and historical behavioral patterns.

[0010] S5. Compare the optimized visual fatigue score with multiple preset fatigue level thresholds to determine the current fatigue level; and automatically trigger the corresponding graded early warning strategy based on the determined level.

[0011] Furthermore, S1 includes:

[0012] S1.1: Deploy high-definition image sensors at preset locations within the cockpit to preprocess the acquired raw video streams, including grayscale conversion, Gaussian filtering for noise reduction, and contrast enhancement.

[0013] S1.2: The preprocessed video frames are input into a pre-trained convolutional neural network model. This model takes the preprocessed video frames as input and includes a feature extraction layer and a keypoint regression layer. The feature extraction layer adopts a residual network structure, extracting global facial features through multiple convolutions and residual connections. The keypoint regression layer outputs the coordinate data of facial contours, periorbital regions, and head bone feature points based on the feature information output by the feature extraction layer, forming a feature point set. The model is trained using a sample dataset of labeled facial feature points, with the deviation between predicted coordinates and true coordinates as the target, and the model parameters are iteratively adjusted through a loss function.

[0014] S1.3: Based on the initial feature point coordinates obtained from the localization, a Kalman filter state equation is constructed. The feature point pixel coordinates and motion velocity are set as state variables. The displacement error of the feature points in adjacent frames is used as the observation value. The feature point position offset is corrected in real time through recursive calculation, so as to realize continuous tracking of the facial region in space.

[0015] Furthermore, S2 includes:

[0016] S2.1: Select the periorbital feature point set E from the feature point set output by S1, which includes the upper eyelid edge point U(i), the lower eyelid edge point L(i), and the pupil boundary point P(j), where i and j are the feature point indices, i ranges from 1 to n, j ranges from 1 to m, and n and m are the total number of eyelid and pupil feature points, respectively; fit the polygon boundary of the bilateral region using the least squares method, and calculate the minimum bounding rectangle R of the polygon. The parameters of the rectangle are defined by the coordinates of the upper left corner (x_min, y_min) and the lower right corner (x_max, y_max); based on the coordinate range of the rectangle R, segment the bilateral region image sequence I(t) from the video frame, where t is the time frame index, representing the t-th frame in the video sequence;

[0017] S2.2: The binocular region image sequence I(t) is grayscaled and normalized to obtain the sequence L(t) showing the change of average brightness of the region over time; a fast Fourier transform is performed on L(t) to extract the power spectral density characteristics of the brightness sequence L(t) within a preset frequency band related to physiological tremor; based on the power spectral density characteristics, the frequency domain index value is calculated by calculating the ratio of energy in a specific frequency band to the total energy; based on the binocular region image sequence I(t), the estimated value of the critical flash fusion frequency CFF is calculated by analyzing the dynamic response characteristics of the pupil region to changes in light intensity; the dynamic response characteristics are represented by the response delay and amplitude of the pupil region's contraction and expansion under a preset frequency light stimulation mode;

[0018] S2.3: Define the eyelid opening and closing reference value D, which is the average vertical distance between the upper and lower eyelid feature points U(i) and L(i) in multiple consecutive frames when the pilot is in a conscious reference state, i.e., D=mean[d(U(i),L(i))]; calculate the eyelid opening and closing degree Dt of the current frame in real time, which is the ratio of the vertical distance of the current frame to the reference value D, i.e., Dt=d_current(U(i),L(i)) / D; when the value of Dt is less than or equal to the preset ratio threshold and the duration of this state reaches the preset number of consecutive frames N, it is determined as a valid blink; calculate the blink frequency F by the formula F=B / T, where B is the number of valid blinks obtained within the statistical duration T.

[0019] S2.4: Select a set of head feature points H for attitude calculation, including the center of the forehead O, the left temporal point T1, the right temporal point T2, and the mandibular point J; construct a three-dimensional facial coordinate system with the center of the forehead O as the origin; solve the coordinates of the feature points in the three-dimensional coordinate system by the two-dimensional and three-dimensional correspondence between the camera perspective projection model and the feature points; calculate the pitch angle α, yaw angle β, and roll angle γ of the head using the Euler angle transformation formula;

[0020] S2.5: Perform a first-order difference operation on the head pose angle sequence [α(t), β(t), γ(t)] of consecutive frames to obtain the rate of change of each pose angle between adjacent frames Δα, Δβ, Δγ; set an angle change rate threshold Δθ, and when the value of max(Δα, Δβ, Δγ) is greater than or equal to the threshold Δθ, it is recorded as a significant head movement; calculate the head movement frequency M using the formula M=K / T, where K is the number of significant movements obtained within the statistical duration T.

[0021] Furthermore, S3 includes:

[0022] S3.1: Feature processing is performed on the critical flash fusion frequency estimates (CFF), blink frequency (F), and head attitude angle extracted in S2: The correlation between any two features is calculated; for feature pairs with a correlation reaching a preset threshold, feature interaction terms are constructed to reflect the synergistic effect between features; the original features and interaction terms are processed using a standardization formula: X'=(X-μ) / σ, where X is the original feature or interaction term to be standardized, μ is the mean of the object to be standardized in the pilot and flight phase dual-dimensional sample sets, and σ is the standard deviation of the corresponding dimension sample set; the standardized original features are then processed... The feature vector V=[CFF',F',α',β',γ',A',B'] is constructed by combining the features and interaction terms in the order of visual physiological indicators, head motion features, and feature interaction terms; where CFF' is the normalized critical flash fusion frequency estimate, F' is the normalized blink frequency, α' is the normalized head pitch angle, β' is the normalized head yaw angle, and γ' is the normalized head roll angle; A' is the normalized interaction term between the critical flash fusion frequency CFF and the blink frequency F, and B' is the normalized interaction term between the head pitch angle α and the roll angle γ.

[0023] S3.2: Construct a pre-trained visual fatigue assessment model, comprising three parts: feature attention processing, temporal local feature extraction, and bidirectional temporal dependency capture. The feature attention processing takes a multi-dimensional feature vector V as input, and calculates the attention weight Wk for each feature through a fully connected layer, using the formula Wk=Softmax[W・V+b], where W is the weight matrix and b is the bias term. The attention-enhanced feature vector V'=∑(Wk×Vk) is obtained through weighted summation, where Vk is the feature component at the k-th position in the multi-dimensional feature vector V, and Wk is the attention weight corresponding to that feature component. The temporal local feature extraction takes a temporal segment of V' as input, employs a dynamic convolution kernel, and processes the temporal segment through two layers of dynamic convolution and one layer of adaptive pooling. The bidirectional temporal dependency capture adopts a bidirectional structure, with the forward and backward parts processing the output results of the temporal local feature extraction respectively, and a hidden layer is set... Set residual connections; when training the visual fatigue assessment model, first train the basic parameters with a large-scale general pilot fatigue sample set, and then adjust the parameters with the historical physiological data of a single pilot; set dynamic sample weights in the loss function, assign weight Wh to high-risk fatigue samples and weight Wb to normal state samples, where Wh is greater than Wb, and the loss function formula is Loss=∑(Wi×E(Yi,Yi')), where Wi is the weight of a single sample, Yi is the true fatigue level of the sample, and Yi' is the fatigue level predicted by the model; E(Yi,Yi')=-Yi×ln(Yi')-(1-Yi)×ln(1-Yi'), which is used to quantify the difference between the true fatigue level of the sample and the fatigue level predicted by the model; input the feature vector V constructed in step S3.1 into the trained model, and the model outputs the preliminary visual fatigue index S1 in the range of [0,100] and the corresponding confidence label.

[0024] Furthermore, S4 includes:

[0025] S4.1: Access the pilot's historical flight database and retrieve the pilot's historical data within a preset period, including the historical fatigue index sequence H=[h1,h2,…,hk] corresponding to different flight stages, where hk is the fatigue index of the pilot in the corresponding stage of the k-th flight; and historical flight state parameters S=[s1,s2,…,sk], where sk is the flight state parameter of the pilot in the k-th flight, concurrent with hk, including the duration, altitude, cabin temperature, and pre-rest duration of the corresponding stage; classify and statistically analyze the historical fatigue index by flight stage, calculate the mean, standard deviation, and fatigue accumulation rate k for each stage, where the fatigue accumulation rate k is the slope obtained by linearly fitting the relationship between the index and flight duration; combine the "stage identifier, mean, standard deviation, fatigue accumulation rate, and historical index sequence" of each stage to construct an individual fatigue pattern dataset D;

[0026] S4.2: Establishing a personalized baseline based on dataset D: A baseline reference value μ0 is set for each flight phase, and a fluctuation range [μ0-λ×σh, μ0+λ×σh] is set in conjunction with σh, where λ is an adjustment coefficient to adapt to individual fluctuation characteristics. kh is used as the dynamic adjustment coefficient for the baseline, forming the personalized dynamic baseline B=[μ0,σh,kh,λ×σh] for that phase, where kh is the historical fatigue accumulation rate for that flight phase. The current flight state parameters C are obtained, including the current flight phase identifier, flight duration tp, current flight altitude hp, cabin ambient temperature te, and pilot's pre-flight rest duration re. The similarity between the current state parameters and the historical flight state parameters in the individual fatigue pattern dataset D is calculated using the weighted Euclidean distance formula: ;

[0027] Where tk is the duration of the corresponding historical flight phase, hg is the altitude of the corresponding historical flight phase, tek is the cabin temperature of the corresponding historical flight phase, rek is the pre-flight rest duration of the corresponding historical flight, and w1, w2, w3, and w4 are preset weights, with w1 and w4 having a higher weight than w2 and w3. Historical data with similarity Sim(C,sk) ≤ preset similarity threshold are selected, and the average fatigue accumulation rate kh in the corresponding historical data is taken as the dynamic adjustment coefficient kp for the current phase.

[0028] S4.3: Match the personalized dynamic baseline B0 corresponding to the current stage, whose components are [baseline reference value μ0, historical standard deviation σh, historical fatigue accumulation rate kh, fluctuation range half-width λ×σh]; calibrate the preliminary fatigue index S1 output by S3 in conjunction with kp:

[0029] Calculate the ratio r of the current flight time to the standard time of this stage; calculate the calibration offset based on kp and r: Δc = kp × r × (S1 - B0[0]), where B0[0] is the baseline reference value μ0, and Δc is used to correct the deviation between the current fatigue index and the baseline reference value, while incorporating the influence of the fatigue accumulation rate.

[0030] The visual fatigue score is optimized by formula S=S1+Δc; if S exceeds the fluctuation range of B0 [μ0-λ×σh,μ0+λ×σh], then Δc is adjusted twice: Δc'=Δc×(B0[3]-|S1-B0[0]|) / B0[3], where B0[3] is the half width of the fluctuation range λ×σh;

[0031] The final output, S, is an optimized visual fatigue score that integrates current physiological responses and historical behavioral patterns.

[0032] The S5 pre-sets multiple fatigue level thresholds, classifying fatigue into four levels from low to high: no fatigue, mild fatigue, moderate fatigue, and severe fatigue. The threshold ranges for each level are determined by the level boundary features obtained from a large-scale pilot fatigue sample statistical analysis, and are fine-tuned based on the individual fatigue tolerance of different pilots. First, a large-scale pilot fatigue sample data is collected, including optimized fatigue scores for different flight phases and environments, as well as the actual operational performance data of the corresponding samples. The sample scores are clustered using the K-means clustering algorithm, and the cluster centers are extracted as initial level boundary features. Then, the operational performance data is used for verification: the scores corresponding to the initial boundary features are substituted into the sample set, and the correlation between the scores and operational error rates under each boundary is calculated. The boundary values ​​are adjusted to ensure the correlation reaches the preset threshold, thus determining the general threshold range, denoted as no fatigue [G1, G2), mild fatigue [G2, G3), moderate fatigue [G3, G4], and severe fatigue [G4, G5]).

[0033] Subsequently, based on the current pilot's historical flight database, the optimized scores and operational performance data corresponding to each fatigue level over the past few months are extracted. The pilot's historical scores are divided into general threshold intervals, and the average operational error rate R_ind within each general level interval is calculated. At the same time, the average operational error rate R_gen of a large-scale general sample in the corresponding level interval is retrieved. The deviation rate ΔR = |R_ind - R_gen| / R_gen is calculated. If ΔR > the preset deviation threshold, the threshold interval for that level is fine-tuned using the following method:

[0034] If R_ind < R_gen: the lower limit of the interval is increased by step Δq = ΔR × (G3 - G2), the new lower limit is G2 + Δq, and the interval becomes [G2 + Δq, G3); if the adjusted interval overlaps with the previous interval, the upper limit of the previous interval is simultaneously increased to G2 + Δq to make the intervals continuous and non-overlapping.

[0035] If R_ind > R_gen: the lower limit of the interval is lowered by step Δq = ΔR × (G3 - G2), the new lower limit is G2 - Δq, and the interval becomes [G2 - Δq, G3); if the adjusted interval overlaps with the previous interval, the upper limit of the previous interval is lowered to G2 - Δq simultaneously.

[0036] Repeat the above calculations and adjustments until ΔR ≤ preset deviation threshold for all level intervals, forming a personalized threshold interval adapted to the pilot; compare the optimized visual fatigue score output by S4 with the personalized threshold interval to determine the current fatigue level and trigger the corresponding graded warning strategy.

[0037] An artificial intelligence-based visual fatigue detection system for pilots includes a data acquisition module, a feature extraction module, a preliminary fatigue index calculation module, a personalized fatigue scoring calibration module, and a fatigue level determination module.

[0038] The data acquisition module continuously acquires facial video streams from the pilot using image sensors deployed in the cockpit, performs facial detection and key point localization on the video frames using a pre-trained convolutional neural network model, and continuously tracks the facial region status based on the Kalman filter algorithm.

[0039] The feature extraction module segments the binocular region image sequence from the video frame based on the located key facial feature points, performs time-frequency domain analysis on the sequence, and extracts physiological indicators that characterize visual function, including the critical flash fusion frequency estimate calculated based on the image brightness change sequence, and the blink frequency obtained by detecting changes in eyelid opening and closing. At the same time, the module calculates the head posture angle and its movement frequency based on the spatial position changes of facial feature points.

[0040] The preliminary fatigue index calculation module uses the extracted critical flash fusion frequency estimate, blink frequency, and head posture angle data to form a multi-dimensional feature vector, which is then input into the pre-trained visual fatigue assessment model to output a preliminary visual fatigue index.

[0041] The personalized fatigue rating calibration module accesses the pilot's historical flight database, calls up the historical fatigue index sequence and corresponding flight status parameters of the pilot in different flight stages within the past preset period, analyzes the individual fatigue evolution pattern, establishes a personalized baseline, and dynamically calibrates the initial score based on the matching degree between the current flight stage and the historical pattern, generating an optimized visual fatigue score that integrates the current physiological response and historical behavioral patterns.

[0042] The fatigue level determination module compares the optimized visual fatigue score with multiple preset fatigue level thresholds to determine the current fatigue level; and automatically triggers the corresponding graded warning strategy based on the determined level.

[0043] Compared with the prior art, the beneficial effects achieved by the present invention are:

[0044] This invention overcomes the limitations of existing fatigue detection methods that rely solely on single physiological signals or general models and ignore individual differences by employing a complete process design encompassing facial feature tracking, multi-dimensional physiological indicator extraction, and personalized scoring calibration. First, it acquires binocular region features and head posture using convolutional neural networks and Kalman filtering, extracting multi-dimensional visual function indicators such as critical flash fusion frequency. Then, it constructs a personalized fatigue baseline by combining this with the pilot's historical flight data. The scoring is dynamically calibrated by matching the current flight status with historical patterns, achieving an innovative approach from generalized detection to precise individualized assessment, effectively improving the accuracy and adaptability of fatigue detection. Simultaneously, in the fatigue level determination and warning stage, this invention determines a general threshold through large-scale sample statistics and fine-tunes it based on individual operational performance data, ensuring that the level classification aligns with the fatigue tolerance of different pilots. Based on this, a graded warning strategy is designed to provide more targeted protection for flight safety. Compared to existing fixed threshold and single warning schemes, its practicality and safety are significantly improved. Attached Figure Description

[0045] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0046] Figure 1 This is a flowchart of a pilot visual fatigue detection method based on artificial intelligence. Detailed Implementation

[0047] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0048] Please see Figure 1 The present invention provides a technical solution: a pilot visual fatigue detection method based on artificial intelligence, comprising:

[0049] S1. The system continuously acquires facial video streams of the pilots through image sensors deployed in the cockpit, performs facial detection and key point localization on the video frames using a pre-trained convolutional neural network model, and continuously tracks the facial region status based on the Kalman filter algorithm.

[0050] S2. Based on the located key facial feature points, segment the binocular region image sequence from the video frame, perform time-frequency domain analysis on the sequence, and extract physiological indicators characterizing visual function, including the critical flash fusion frequency estimate calculated based on the image brightness change sequence, and the blink frequency obtained by detecting changes in eyelid opening and closing; at the same time, calculate the head posture angle and its motion frequency based on the spatial position changes of facial feature points.

[0051] S3. The extracted critical flash fusion frequency estimate, blink frequency, and head posture angle data are combined to form a multi-dimensional feature vector, which is then input into the pre-trained visual fatigue assessment model to output a preliminary visual fatigue index.

[0052] S4. Access the pilot's historical flight database, retrieve the historical fatigue index sequence and corresponding flight status parameters of the pilot in different flight stages within the past preset period, analyze the individual fatigue evolution pattern, establish a personalized baseline, and dynamically calibrate the initial score based on the matching degree between the current flight stage and the historical pattern to generate an optimized visual fatigue score that integrates the current physiological response and historical behavioral patterns.

[0053] S5. Compare the optimized visual fatigue score with multiple preset fatigue level thresholds to determine the current fatigue level; and automatically trigger the corresponding graded early warning strategy based on the determined level.

[0054] S1 includes:

[0055] S1.1: Deploy high-definition image sensors at preset locations within the cockpit to preprocess the acquired raw video streams, including grayscale conversion, Gaussian filtering for noise reduction, and contrast enhancement.

[0056] S1.2: The preprocessed video frames are input into a pre-trained convolutional neural network model. This model takes the preprocessed video frames as input and includes a feature extraction layer and a keypoint regression layer. The feature extraction layer adopts a residual network structure, extracting global facial features through multiple convolutions and residual connections. The keypoint regression layer outputs the coordinate data of facial contours, periorbital regions, and head bone feature points based on the feature information output by the feature extraction layer, forming a feature point set. The model is trained using a sample dataset of labeled facial feature points, with the deviation between predicted coordinates and true coordinates as the target, and the model parameters are iteratively adjusted through a loss function.

[0057] S1.3: Based on the initial feature point coordinates obtained from the localization, a Kalman filter state equation is constructed. The feature point pixel coordinates and motion velocity are set as state variables. The displacement error of the feature points in adjacent frames is used as the observation value. The feature point position offset is corrected in real time through recursive calculation, so as to realize continuous tracking of the facial region in space.

[0058] S2 includes:

[0059] S2.1: Select the periorbital feature point set E from the feature point set output by S1, which includes the upper eyelid edge point U(i), the lower eyelid edge point L(i), and the pupil boundary point P(j), where i and j are the feature point indices, i ranges from 1 to n, j ranges from 1 to m, and n and m are the total number of eyelid and pupil feature points, respectively; fit the polygon boundary of the bilateral region using the least squares method, and calculate the minimum bounding rectangle R of the polygon. The parameters of the rectangle are defined by the coordinates of the upper left corner (x_min, y_min) and the lower right corner (x_max, y_max); based on the coordinate range of the rectangle R, segment the bilateral region image sequence I(t) from the video frame, where t is the time frame index, representing the t-th frame in the video sequence;

[0060] S2.2: The binocular region image sequence I(t) is grayscaled and normalized to obtain the sequence L(t) showing the change of average brightness of the region over time; a fast Fourier transform is performed on L(t) to extract the power spectral density characteristics of the brightness sequence L(t) within a preset frequency band related to physiological tremor; based on the power spectral density characteristics, the frequency domain index value is calculated by calculating the ratio of energy in a specific frequency band to the total energy; based on the binocular region image sequence I(t), the estimated value of the critical flash fusion frequency CFF is calculated by analyzing the dynamic response characteristics of the pupil region to changes in light intensity; the dynamic response characteristics are represented by the response delay and amplitude of the pupil region's contraction and expansion under a preset frequency light stimulation mode;

[0061] S2.3: Define the eyelid opening and closing reference value D, which is the average vertical distance between the upper and lower eyelid feature points U(i) and L(i) in multiple consecutive frames when the pilot is in a conscious reference state, i.e., D=mean[d(U(i),L(i))]; calculate the eyelid opening and closing degree Dt of the current frame in real time, which is the ratio of the vertical distance of the current frame to the reference value D, i.e., Dt=d_current(U(i),L(i)) / D; when the value of Dt is less than or equal to the preset ratio threshold and the duration of this state reaches the preset number of consecutive frames N, it is determined as a valid blink; calculate the blink frequency F by the formula F=B / T, where B is the number of valid blinks obtained within the statistical duration T.

[0062] S2.4: Select a set of head feature points H for attitude calculation, including the center of the forehead O, the left temporal point T1, the right temporal point T2, and the mandibular point J; construct a three-dimensional facial coordinate system with the center of the forehead O as the origin; solve the coordinates of the feature points in the three-dimensional coordinate system by the two-dimensional and three-dimensional correspondence between the camera perspective projection model and the feature points; calculate the pitch angle α, yaw angle β, and roll angle γ of the head using the Euler angle transformation formula;

[0063] S2.5: Perform a first-order difference operation on the head pose angle sequence [α(t), β(t), γ(t)] of consecutive frames to obtain the rate of change of each pose angle between adjacent frames Δα, Δβ, Δγ; set an angle change rate threshold Δθ, and when the value of max(Δα, Δβ, Δγ) is greater than or equal to the threshold Δθ, it is recorded as a significant head movement; calculate the head movement frequency M using the formula M=K / T, where K is the number of significant movements obtained within the statistical duration T.

[0064] S3 includes:

[0065] S3.1: Feature processing is performed on the critical flash fusion frequency estimates (CFF), blink frequency (F), and head attitude angle extracted in S2: The correlation between any two features is calculated; for feature pairs with a correlation reaching a preset threshold, feature interaction terms are constructed to reflect the synergistic effect between features; the original features and interaction terms are processed using a standardization formula: X'=(X-μ) / σ, where X is the original feature or interaction term to be standardized, μ is the mean of the object to be standardized in the pilot and flight phase dual-dimensional sample sets, and σ is the standard deviation of the corresponding dimension sample set; the standardized original features are then processed... The feature vector V=[CFF',F',α',β',γ',A',B'] is constructed by combining the features and interaction terms in the order of visual physiological indicators, head motion features, and feature interaction terms; where CFF' is the normalized critical flash fusion frequency estimate, F' is the normalized blink frequency, α' is the normalized head pitch angle, β' is the normalized head yaw angle, and γ' is the normalized head roll angle; A' is the normalized interaction term between the critical flash fusion frequency CFF and the blink frequency F, and B' is the normalized interaction term between the head pitch angle α and the roll angle γ.

[0066] S3.2: Construct a pre-trained visual fatigue assessment model, comprising three parts: feature attention processing, temporal local feature extraction, and bidirectional temporal dependency capture. The feature attention processing takes a multi-dimensional feature vector V as input, and calculates the attention weight Wk for each feature through a fully connected layer, using the formula Wk=Softmax[W・V+b], where W is the weight matrix and b is the bias term. The attention-enhanced feature vector V'=∑(Wk×Vk) is obtained through weighted summation, where Vk is the feature component at the k-th position in the multi-dimensional feature vector V, and Wk is the attention weight corresponding to that feature component. The temporal local feature extraction takes a temporal segment of V' as input, employs a dynamic convolution kernel, and processes the temporal segment through two layers of dynamic convolution and one layer of adaptive pooling. The bidirectional temporal dependency capture adopts a bidirectional structure, with the forward and backward parts processing the output results of the temporal local feature extraction respectively, and in the hidden layer... Set up residual connections; when training the visual fatigue assessment model, first train the basic parameters with a large-scale general pilot fatigue sample set, and then adjust the parameters with the historical physiological data of a single pilot; set dynamic sample weights in the loss function, assign weight Wh to high-risk fatigue samples and weight Wb to normal state samples, where Wh is greater than Wb, and the loss function formula is Loss=∑(Wi×E(Yi,Yi')), where Wi is the weight of a single sample, Yi is the true fatigue level of the sample, and Yi' is the fatigue level predicted by the model; E(Yi,Yi')=-Yi×ln(Yi')-(1-Yi)×ln(1-Yi'), which is used to quantify the difference between the true fatigue level of the sample and the fatigue level predicted by the model; input the feature vector V constructed in step S3.1 into the trained model, and the model outputs a preliminary visual fatigue index and corresponding confidence label in the range of [0,100].

[0067] S4 includes:

[0068] S4.1: Access the pilot's historical flight database and retrieve the pilot's historical data within a preset period, including the historical fatigue index sequence H=[h1,h2,…,hk] corresponding to different flight stages, where hk is the fatigue index of the pilot in the corresponding stage of the k-th flight; and historical flight state parameters S=[s1,s2,…,sk], where sk is the flight state parameter of the pilot in the k-th flight, concurrent with hk, including the duration, altitude, cabin temperature, and pre-rest duration of the corresponding stage; classify and statistically analyze the historical fatigue index by flight stage, calculate the mean, standard deviation, and fatigue accumulation rate k for each stage, where the fatigue accumulation rate k is the slope obtained by linearly fitting the relationship between the index and flight duration; combine the "stage identifier, mean, standard deviation, fatigue accumulation rate, and historical index sequence" of each stage to construct an individual fatigue pattern dataset D;

[0069] S4.2: Establishing a personalized baseline based on dataset D: A baseline reference value μ0 is set for each flight phase, and a fluctuation range [μ0-λ×σh, μ0+λ×σh] is set in conjunction with σh, where λ is an adjustment coefficient to adapt to individual fluctuation characteristics. kh is used as the dynamic adjustment coefficient for the baseline, forming the personalized dynamic baseline B=[μ0,σh,kh,λ×σh] for that phase, where kh is the historical fatigue accumulation rate for that flight phase. The current flight state parameters C are obtained, including the current flight phase identifier, flight duration tp, current flight altitude hp, cabin ambient temperature te, and pilot's pre-flight rest duration re. The similarity between the current state parameters and the historical flight state parameters in the individual fatigue pattern dataset D is calculated using the weighted Euclidean distance formula: ;

[0070] Where tk is the duration of the corresponding historical flight phase, hg is the altitude of the corresponding historical flight phase, tek is the cabin temperature of the corresponding historical flight phase, rek is the pre-flight rest duration of the corresponding historical flight, and w1, w2, w3, and w4 are preset weights, with w1 and w4 having a higher weight than w2 and w3. Historical data with similarity Sim(C,sk) ≤ preset similarity threshold are selected, and the average fatigue accumulation rate kh in the corresponding historical data is taken as the dynamic adjustment coefficient kp for the current phase.

[0071] S4.3: Match the personalized dynamic baseline B0 corresponding to the current stage, whose components are [baseline reference value μ0, historical standard deviation σh, historical fatigue accumulation rate kh, fluctuation range half-width λ×σh]; calibrate the preliminary fatigue index S1 output by S3 in conjunction with kp:

[0072] Calculate the ratio r of the current flight time to the standard time of this stage; calculate the calibration offset based on kp and r: Δc = kp × r × (S1 - B0[0]), where B0[0] is the baseline reference value μ0, and Δc is used to correct the deviation between the current fatigue index and the baseline reference value, while incorporating the influence of the fatigue accumulation rate.

[0073] The visual fatigue score is optimized by formula S=S1+Δc; if S exceeds the fluctuation range of B0 [μ0-λ×σh,μ0+λ×σh], then Δc is adjusted twice: Δc'=Δc×(B0[3]-|S1-B0[0]|) / B0[3], where B0[3] is the half width of the fluctuation range λ×σh;

[0074] The final output, S, is an optimized visual fatigue score that integrates current physiological responses and historical behavioral patterns.

[0075] The S5 has multiple preset fatigue level thresholds, which are divided into four levels from low to high fatigue: no fatigue, mild fatigue, moderate fatigue, and severe fatigue. The threshold range for each level is determined by the level boundary characteristics of a large-scale pilot fatigue sample statistical analysis, and is fine-tuned according to the individual fatigue tolerance of different pilots (the correlation data between the pilot's past fatigue level and actual operation performance is extracted from the historical flight database).

[0076] When determining the threshold range, a large-scale sample of pilot fatigue data is first collected, including optimized fatigue scores for different flight stages and environments, as well as actual operational performance data of the corresponding samples. The sample scores are clustered using the K-means clustering algorithm, and the cluster centers are extracted as initial level boundary features. Then, the operational performance data is used for verification: the scores corresponding to the initial boundary features are substituted into the sample set, and the correlation between the scores and operational error rates under each boundary is calculated. The boundary values ​​are adjusted so that the correlation reaches the preset threshold, and a general threshold range is determined, denoted as no fatigue [G1, G2), mild fatigue [G2, G3), moderate fatigue [G3, G4], and severe fatigue [G4, G5]).

[0077] Subsequently, based on the current pilot's historical flight database, the optimized scores and operational performance data corresponding to each fatigue level over the past few months are extracted. The pilot's historical scores are divided into general threshold intervals, and the average operational error rate R_ind within each general level interval is calculated. At the same time, the average operational error rate R_gen of a large-scale general sample in the corresponding level interval is retrieved. The deviation rate ΔR = |R_ind - R_gen| / R_gen is calculated. If ΔR > the preset deviation threshold, the threshold interval for that level is fine-tuned using the following method:

[0078] If R_ind < R_gen: the lower limit of the interval is increased by step Δq = ΔR × (G3 - G2), the new lower limit is G2 + Δq, and the interval becomes [G2 + Δq, G3); if the adjusted interval overlaps with the previous interval, the upper limit of the previous interval is simultaneously increased to G2 + Δq to make the intervals continuous and non-overlapping.

[0079] If R_ind > R_gen: the lower limit of the interval is lowered by step Δq = ΔR × (G3 - G2), the new lower limit is G2 - Δq, and the interval becomes [G2 - Δq, G3); if the adjusted interval overlaps with the previous interval, the upper limit of the previous interval is lowered to G2 - Δq simultaneously.

[0080] Repeat the above calculations and adjustments until ΔR ≤ preset deviation threshold for all level intervals, forming a personalized threshold interval adapted to the pilot; compare the optimized visual fatigue score output by S4 with the personalized threshold interval to determine the current fatigue level and trigger the corresponding graded warning strategy.

[0081] This embodiment uses a pilot of a certain airline operating a passenger plane on the Beijing-Shanghai route as the test object. The specific implementation steps are as follows:

[0082] A 2-megapixel high-definition image sensor is deployed above the cockpit instrument panel to capture video streams of the pilot's face. The original video stream is preprocessed (grayscale conversion, 5×5 Gaussian filtering for noise reduction, and adaptive contrast enhancement). The preprocessed frames are input into a pre-trained residual network model, which outputs a set of feature points for facial contours, periorbital area (12 points on the upper eyelid, 12 points on the lower eyelid, and 8 points on the pupil), and head bones (brow center, left and right temporal regions, and mandible). A Kalman filter equation is constructed based on the feature point coordinates, with state variables including pixel coordinates and motion velocity, to correct offsets in real time and achieve continuous facial tracking.

[0083] The eye periorbital point set is selected from the feature point set, and the polygon of the bilateral region is fitted and the minimum bounding rectangle is taken to segment the bilateral image sequence I(t). The I(t) is grayscale normalized to obtain the brightness sequence L(t). The power spectral density of the 0.1-0.3Hz frequency band is extracted by fast Fourier transform, and the frequency domain index is calculated. Combined with the pupil response to 5Hz light stimulus (delay < 0.5s, amplitude > 15% is valid), the CFF estimate is obtained. The eyelid opening and closing benchmark value D = 10mm is defined in the awake state. Dt is calculated in real time. When Dt ≤ 0.3 and lasts for 3 frames, it is determined to be a valid blink. The number of blinks within 60s is counted to obtain F. A three-dimensional coordinate system is constructed by selecting head feature points. After solving the coordinates, α, β, and γ are obtained by Euler angle formula. The first difference of the attitude angle of the continuous frames is set to Δθ = 5° / frame. The number of significant movements within 60s is counted to obtain M.

[0084] Initially, the feature correlation degree of CFF, F, α, β, and γ is calculated, and CFF and F, α and γ are taken as correlation pairs to construct interaction terms; normalized according to X'=(X-μ) / σ, μ and σ are taken from the "pilot-cruise phase" sample set, to obtain the vector V=[CFF',F',α',β',γ',CFF'×F',α'×γ']; input into the pre-trained model, after attention processing, W is a 128-dimensional matrix, b=0.1; 2 layers of dynamic convolution, bidirectional structure, output initial exponent S1=42 (confidence level 92%).

[0085] By accessing the pilot's historical flight data from Beijing to Shanghai over the past 6 months, the cruise phase H=[38,40,45] and S include duration of 100 min and altitude of 10 km are obtained. The values ​​μ=41, σh=2.6, and kh=0.05 are calculated, and dataset D is constructed. A baseline B=[41,2.6,0.05,1.3] is established. The current C_state includes 40 min of flight time and an altitude of 10 km. The similarity to historical data Sim=0.8 (≤threshold 1.0) is calculated, and kp=0.045 is set. r=40 / 100=0.4, Δc=0.045×0.4×(42-41)=0.018, S=42+0.018≈42.02, which does not exceed the interval [39.7,42.3], resulting in an optimization score of 42.02.

[0086] With the general threshold set [G1=0,G2=20,G3=45,G4=70,G5=100], the pilot's historical data is extracted, resulting in cruise phase R_ind=3% (R_gen=5%), ΔR=40%>15%, and the lower limit of mild fatigue is increased by Δq=40%×25=10, resulting in a new range [30,45). The optimization score of 42.02 indicates mild fatigue, triggering a level one warning: a white prompt is displayed in the non-core area of ​​the screen, and a low-volume prompt tone is played for 0.5 seconds.

[0087] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. A pilot visual fatigue detection method based on artificial intelligence, characterized in that: The method includes: S1. The system continuously acquires facial video streams of the pilots using image sensors deployed in the cockpit, performs facial detection and key point localization on the video frames using a pre-trained convolutional neural network model, and continuously tracks the facial region status based on the Kalman filter algorithm. S2. Based on the located key facial feature points, segment the binocular region image sequence from the video frame, perform time-frequency domain analysis on the sequence, and extract physiological indicators characterizing visual function, including the critical flash fusion frequency estimate calculated based on the image brightness change sequence, and the blink frequency obtained by detecting changes in eyelid opening and closing; at the same time, calculate the head posture angle and its motion frequency based on the spatial position changes of facial feature points. S3. The extracted critical flash fusion frequency estimate, blink frequency, and head posture angle data are combined to form a multi-dimensional feature vector, which is then input into the pre-trained visual fatigue assessment model to output a preliminary visual fatigue index. S4. Access the pilot's historical flight database, retrieve the historical fatigue index sequence and corresponding flight status parameters of the pilot in different flight stages within the past preset period, analyze the individual fatigue evolution pattern, establish a personalized baseline, and dynamically calibrate the initial score based on the matching degree between the current flight stage and the historical pattern to generate an optimized visual fatigue score that integrates the current physiological response and historical behavioral patterns. S4 includes: S4.1: Access the pilot's historical flight database and retrieve the pilot's historical data within a preset period, including the historical fatigue index sequence H=[h1,h2,…,hk] corresponding to different flight stages, where hk is the fatigue index of the pilot in the corresponding stage of the k-th flight; and historical flight state parameters S=[s1,s2,…,sk], where sk is the flight state parameter of the pilot in the k-th flight, concurrent with hk, including the duration, altitude, cabin temperature, and pre-rest duration of the corresponding stage; classify and statistically analyze the historical fatigue index by flight stage, calculate the mean, standard deviation, and fatigue accumulation rate k for each stage, where the fatigue accumulation rate k is the slope obtained by linearly fitting the relationship between the index and flight duration; combine the "stage identifier, mean, standard deviation, fatigue accumulation rate, and historical index sequence" of each stage to construct an individual fatigue pattern dataset D; S4.2: Establishing a personalized baseline based on dataset D: A baseline reference value μ0 is set for each flight phase, and a fluctuation range [μ0-λ×σh, μ0+λ×σh] is set in conjunction with σh, where λ is an adjustment coefficient to adapt to individual fluctuation characteristics. kh is used as the dynamic adjustment coefficient for the baseline, forming a personalized dynamic baseline B=[μ0,σh,kh,λ×σh] for that flight phase, where kh is the historical fatigue accumulation rate for that flight phase. The current flight state parameters C are obtained, including the current flight phase identifier, flight duration tp, current flight altitude hp, cabin ambient temperature te, and pilot's pre-flight rest duration re. The similarity between the current state parameters and the historical flight state parameters in the individual fatigue pattern dataset D is calculated using a weighted Euclidean distance formula. ; Where tk is the duration of the corresponding historical flight phase, hg is the altitude of the corresponding historical flight phase, tek is the cabin temperature of the corresponding historical flight phase, rek is the pre-flight rest duration of the corresponding historical flight, and w1, w2, w3, and w4 are preset weights, with w1 and w4 having a higher weight than w2 and w3; historical data with similarity Sim(C,sk) ≤ preset similarity threshold are selected, and the average fatigue accumulation rate kh in the corresponding historical data is taken as the dynamic adjustment coefficient kp for the current phase; S4.3: Match the personalized dynamic baseline B0 corresponding to the current stage, whose components are [baseline reference value μ0, historical standard deviation σh, historical fatigue accumulation rate kh, fluctuation range half width λ×σh]; calibrate the preliminary visual fatigue index output by S3 in conjunction with kp; S5. Compare the optimized visual fatigue score with multiple preset fatigue level thresholds to determine the current fatigue level; and automatically trigger the corresponding graded early warning strategy based on the determined level.

2. The pilot visual fatigue detection method based on artificial intelligence according to claim 1, characterized in that: S1 includes: S1.1: Deploy high-definition image sensors at preset locations within the cockpit to preprocess the acquired raw video streams, including grayscale conversion, Gaussian filtering for noise reduction, and contrast enhancement. S1.2: The preprocessed video frames are input into a pre-trained convolutional neural network model. This model takes the preprocessed video frames as input and includes a feature extraction layer and a keypoint regression layer. The feature extraction layer adopts a residual network structure, extracting global facial features through multiple convolutions and residual connections. The keypoint regression layer outputs the coordinate data of facial contours, periorbital regions, and head bone feature points based on the feature information output by the feature extraction layer, forming a feature point set. The model is trained using a sample dataset of labeled facial feature points, with the deviation between predicted coordinates and true coordinates as the target, and the model parameters are iteratively adjusted through a loss function. S1.3: Based on the initial feature point coordinates obtained from the localization, a Kalman filter state equation is constructed. The feature point pixel coordinates and motion velocity are set as state variables. The displacement error of the feature points in adjacent frames is used as the observation value. The feature point position offset is corrected in real time through recursive calculation, so as to realize continuous tracking of the facial region in space.

3. The pilot visual fatigue detection method based on artificial intelligence according to claim 2, characterized in that: S2 includes: S2.1: Select the periorbital feature point set E from the feature point set output by S1, which includes the upper eyelid edge point U(i), the lower eyelid edge point L(i), and the pupil boundary point P(j), where i and j are the feature point indices, i ranges from 1 to n, j ranges from 1 to m, and n and m are the total number of eyelid and pupil feature points, respectively; fit the polygon boundary of the bilateral region using the least squares method, and calculate the minimum bounding rectangle R of the polygon. The parameters of the rectangle are defined by the coordinates of the upper left corner (x_min, y_min) and the lower right corner (x_max, y_max); based on the coordinate range of the rectangle R, segment the bilateral region image sequence I(t) from the video frame, where t is the time frame index, representing the t-th frame in the video sequence; S2.2: The binocular region image sequence I(t) is grayscaled and normalized to obtain the sequence L(t) showing the change in average brightness of the region over time; a fast Fourier transform is performed on L(t) to extract the power spectral density characteristics of the image brightness change sequence L(t) within a preset frequency band related to physiological tremor; based on the power spectral density characteristics, the frequency domain index value is calculated by calculating the ratio of energy in a specific frequency band to the total energy; based on the binocular region image sequence I(t), the estimated value of the critical flash fusion frequency CFF is calculated by analyzing the dynamic response characteristics of the pupil region to changes in light intensity; the dynamic response characteristics are represented by the response delay and amplitude of the pupil region's contraction and expansion under a preset frequency light stimulation mode; S2.3: Define the eyelid opening and closing reference value D, which is the average vertical distance between the upper and lower eyelid feature points U(i) and L(i) in multiple consecutive frames when the pilot is in a conscious reference state, i.e., D=mean[d(U(i),L(i))]; calculate the eyelid opening and closing degree Dt of the current frame in real time, which is the ratio of the vertical distance of the current frame to the reference value D, i.e., Dt=d_current(U(i),L(i)) / D; d_current(U(i),L(i)) represents the vertical distance between the i-th pair of upper eyelid feature points U(i) and lower eyelid feature points L(i) in the current video frame; when the value of Dt is less than or equal to the preset proportional threshold and the duration of this state reaches the preset number of consecutive frames N, it is determined as a valid blink; calculate the blink frequency F by the formula F=B / T, where B is the number of valid blinks obtained within the statistical duration T.

4. The pilot visual fatigue detection method based on artificial intelligence according to claim 3, characterized in that: S2 further includes: S2.4: Select a set of head feature points H for attitude calculation, including the center of the forehead O, the left temporal point T1, the right temporal point T2, and the mandibular point J; construct a three-dimensional facial coordinate system with the center of the forehead O as the origin; solve the coordinates of the feature points in the three-dimensional coordinate system by the two-dimensional and three-dimensional correspondence between the camera perspective projection model and the feature points; calculate the pitch angle α, yaw angle β, and roll angle γ of the head using the Euler angle transformation formula; S2.5: Perform a first-order difference operation on the head pose angle sequence [α(t), β(t), γ(t)] of consecutive frames to obtain the rate of change of each pose angle between adjacent frames Δα, Δβ, Δγ; set an angle change rate threshold Δθ, and when the value of max(Δα, Δβ, Δγ) is greater than or equal to the threshold Δθ, it is recorded as a significant head movement; calculate the head movement frequency M using the formula M=K / T, where K is the number of significant movements obtained within the statistical duration T.

5. The pilot visual fatigue detection method based on artificial intelligence according to claim 4, characterized in that: S3 includes: S3.1: Feature processing is performed on the critical flash fusion frequency estimates (CFF), blink frequency (F), and head attitude angle extracted in S2: The correlation between any two features is calculated; for feature pairs with a correlation reaching a preset threshold, feature interaction terms are constructed to reflect the synergistic effect between features; the original features and interaction terms are processed using a standardization formula: X'=(X-μ) / σ, where X is the original feature or interaction term to be standardized, μ is the mean of the object to be standardized in the pilot and flight phase dual-dimensional sample sets, and σ is the standard deviation of the corresponding dimension sample set; the standardized original features are then processed... The feature vector V=[CFF',F',α',β',γ',A',B'] is constructed by combining the features and interaction terms in the order of visual physiological indicators, head motion features, and feature interaction terms; where CFF' is the normalized critical flash fusion frequency estimate, F' is the normalized blink frequency, α' is the normalized head pitch angle, β' is the normalized head yaw angle, and γ' is the normalized head roll angle; A' is the normalized interaction term between the critical flash fusion frequency CFF and the blink frequency F, and B' is the normalized interaction term between the head pitch angle α and the roll angle γ. S3.2: Construct a pre-trained visual fatigue assessment model, comprising three parts: feature attention processing, temporal local feature extraction, and bidirectional temporal dependency capture. The feature attention processing takes a multi-dimensional feature vector V as input, and calculates the attention weight Wk for each feature through a fully connected layer, using the formula Wk=Softmax[W・V+b], where W is the weight matrix and b is the bias term. The attention-enhanced feature vector V'=∑(Wk×Vk) is obtained through weighted summation, where Vk is the feature component at the k-th position in the multi-dimensional feature vector V, and Wk is the attention weight corresponding to that feature component. The temporal local feature extraction takes a temporal segment of V' as input, employs a dynamic convolution kernel, and processes the temporal segment through two layers of dynamic convolution and one layer of adaptive pooling. The bidirectional temporal dependency capture adopts a bidirectional structure, with the forward and backward parts processing the output results of the temporal local feature extraction respectively, and in the hidden layer... Set up residual connections; when training the visual fatigue assessment model, first train the basic parameters with a large-scale general pilot fatigue sample set, and then adjust the parameters with the historical physiological data of a single pilot; set dynamic sample weights in the loss function, assign weight Wh to high-risk fatigue samples and weight Wb to normal state samples, where Wh is greater than Wb, and the loss function formula is Loss=∑(Wi×E(Yi,Yi')), where Wi is the weight of a single sample, Yi is the true fatigue level of the sample, and Yi' is the fatigue level predicted by the model; E(Yi,Yi')=-Yi×ln(Yi')-(1-Yi)×ln(1-Yi'), which is used to quantify the difference between the true fatigue level of the sample and the fatigue level predicted by the model; input the feature vector V constructed in step S3.1 into the trained model, and the model outputs a preliminary visual fatigue index and corresponding confidence label in the range of [0,100].

6. The pilot visual fatigue detection method based on artificial intelligence according to claim 5, characterized in that: Step S4 further includes: Calculate the ratio r of the current flight duration to the standard flight duration of the flight phase; calculate the calibration offset based on kp and r: Δc = kp × r × (S1 - B0[0]), where B0[0] is the baseline reference value μ0, and Δc is used to correct the deviation between the current fatigue index and the baseline reference value, while also incorporating the influence of the fatigue accumulation rate. The visual fatigue score is optimized by formula S=S1+Δc; if S exceeds the fluctuation range of B0 [μ0-λ×σh,μ0+λ×σh], then Δc is adjusted twice: Δc'=Δc×(B0[3]-|S1-B0[0]|) / B0[3], where B0[3] is the half width of the fluctuation range λ×σh; The final output, S, is an optimized visual fatigue score that integrates current physiological responses and historical behavioral patterns.

7. The pilot visual fatigue detection method based on artificial intelligence according to claim 1, characterized in that: The S5 presets multiple fatigue level thresholds, classifying fatigue into four levels from low to high: no fatigue, mild fatigue, moderate fatigue, and severe fatigue. The threshold ranges corresponding to each level are determined by the level boundary features of a large-scale pilot fatigue sample statistical analysis. Furthermore, it extracts correlation data between the pilot's past fatigue levels and actual operational performance from a historical flight database for fine-tuning: First, a large-scale pilot fatigue sample data is collected, including optimized fatigue scores for different flight phases and environments, as well as corresponding actual operational performance data. The sample scores are clustered using the K-means clustering algorithm, and the cluster centers are extracted as initial level boundary features. This is then verified using operational performance data: The scores corresponding to the initial boundary features are substituted into the sample set, and the correlation between scores and operational error rates under each boundary is calculated. The boundary values ​​are adjusted to ensure the correlation reaches a preset threshold, determining a general threshold range, denoted as no fatigue [G1, G2), mild fatigue [G2, G3), moderate fatigue [G3, G4), and severe fatigue [G4, G5]). Subsequently, based on the current pilot's historical flight database, the optimized scores and operational performance data corresponding to each fatigue level over the past few months are extracted. The average operational error rate of the pilot at each level within the general threshold range is calculated. If the average error rate at a certain level deviates from the average error rate of the general sample at that level by more than a preset deviation threshold, the threshold range for that level is fine-tuned. The optimized visual fatigue score output by S4 is compared with the preset threshold ranges for each fatigue level. The current fatigue level of the pilot is determined according to the range to which the score belongs, and the corresponding graded early warning strategy is triggered.

8. An artificial intelligence-based pilot visual fatigue detection system, characterized in that: The system includes a data acquisition module, a feature extraction module, a preliminary fatigue index calculation module, a personalized fatigue score calibration module, and a fatigue level determination module. The data acquisition module continuously acquires facial video streams from the pilot using image sensors deployed in the cockpit, performs facial detection and key point localization on the video frames using a pre-trained convolutional neural network model, and continuously tracks the facial region status based on the Kalman filter algorithm. The feature extraction module segments the binocular region image sequence from the video frame based on the located key facial feature points, performs time-frequency domain analysis on the sequence, and extracts physiological indicators characterizing visual function, including the critical flash fusion frequency estimate calculated based on the image brightness change sequence, and the blink frequency obtained by detecting changes in eyelid opening and closing; at the same time, it calculates the head posture angle and its movement frequency based on the spatial position changes of facial feature points. The preliminary fatigue index calculation module constructs a multi-dimensional feature vector from the extracted critical flash fusion frequency estimate, blink frequency, and head posture angle data, and inputs it into the pre-trained visual fatigue assessment model to output a preliminary visual fatigue index. The personalized fatigue scoring calibration module accesses the pilot's historical flight database, calls up the historical fatigue index sequence and corresponding flight status parameters of the pilot in different flight stages within the past preset period, analyzes the individual fatigue evolution pattern, establishes a personalized baseline, and dynamically calibrates the initial score based on the matching degree between the current flight stage and the historical pattern, generating an optimized visual fatigue score that integrates the current physiological response and historical behavioral patterns. The personalized fatigue scoring calibration module specifically includes: Access the pilot's historical flight database and retrieve the pilot's historical data within a preset period, including the historical fatigue index sequence H=[h1,h2,…,hk] corresponding to different flight stages, where hk is the fatigue index of the pilot in the corresponding stage of the k-th flight; and historical flight state parameters S=[s1,s2,…,sk], where sk is the flight state parameter of the pilot in the k-th flight, concurrent with hk, including the duration, altitude, cabin temperature, and pre-flight rest duration of the corresponding stage; classify and statistically analyze the historical fatigue index by flight stage, calculate the mean, standard deviation, and fatigue accumulation rate k for each stage, where the fatigue accumulation rate k is the slope obtained by linearly fitting the relationship between the index and flight duration; combine the "stage identifier, mean, standard deviation, fatigue accumulation rate, and historical index sequence" of each stage to construct an individual fatigue pattern dataset D; Personalized baselines are established based on dataset D: a baseline reference value μ0 is set for each flight phase, and a fluctuation range [μ0-λ×σh, μ0+λ×σh] is set in combination with σh, where λ is an adjustment coefficient to adapt to individual fluctuation characteristics. kh is used as the dynamic adjustment coefficient for the baseline, forming a personalized dynamic baseline B=[μ0,σh,kh,λ×σh] for that flight phase, where kh is the historical fatigue accumulation rate for that flight phase. Current flight state parameters C are obtained, including the current flight phase identifier, flight duration tp, current flight altitude hp, cabin ambient temperature te, and pilot's pre-flight rest duration re. The similarity between the current state parameters and the historical flight state parameters in the individual fatigue pattern dataset D is calculated using a weighted Euclidean distance formula. ; Where tk is the duration of the corresponding historical flight phase, hg is the altitude of the corresponding historical flight phase, tek is the cabin temperature of the corresponding historical flight phase, rek is the pre-flight rest duration of the corresponding historical flight, and w1, w2, w3, and w4 are preset weights, with w1 and w4 having a higher weight than w2 and w3; historical data with similarity Sim(C,sk) ≤ preset similarity threshold are selected, and the average fatigue accumulation rate kh in the corresponding historical data is taken as the dynamic adjustment coefficient kp for the current phase; Match the personalized dynamic baseline B0 corresponding to the current stage, whose components are [baseline reference value μ0, historical standard deviation σh, historical fatigue accumulation rate kh, fluctuation range half width λ×σh]; and calibrate the preliminary visual fatigue index output by the preliminary fatigue index calculation module in combination with kp. The fatigue level determination module compares the optimized visual fatigue score with multiple preset fatigue level thresholds to determine the current fatigue level; and automatically triggers the corresponding graded early warning strategy based on the determined level.

Images