Pilot visual fatigue detection system and method based on artificial intelligence

Abstract

The invention discloses a pilot visual fatigue detection system and method based on artificial intelligence, and belongs to the technical field of flight safety monitoring, and the method comprises the steps: obtaining a pilot face video through an image sensor, carrying out the face positioning and tracking through a neural network, extracting a blinking frequency and a critical flash fusion frequency from an eye region, and carrying out the detection of the visual fatigue of a pilot. The head posture movement is analyzed; fusing the multi-dimensional physiological data with the behavior features, and inputting the fused data into a pre-trained visual fatigue evaluation model to obtain an initial fatigue index; performing dynamic calibration on the initial fatigue index in combination with the historical fatigue data of the pilot individual and the current flight mission load, and generating an optimized visual fatigue score; and finally, grading early warning is started according to the grading grade, personalized and foresight fatigue state intervention is realized, and the flight safety is effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of flight safety monitoring, and particularly relates to a pilot visual fatigue detection system and method based on artificial intelligence. BACKGROUND

[0002] In the field of aviation flight safety, the work load and mental state of pilots are core elements to ensure flight safety. Visual fatigue caused by long-time and high-concentration tasks directly reduces the situational awareness and operation response ability of pilots. Therefore, real-time and accurate monitoring of the visual fatigue state of pilots has great significance for actively preventing safety hazards caused by human factors.

[0003] At present, technical solutions applied to fatigue monitoring are mostly focused on the collection and analysis of single type physiological signals. For example, eye movement tracks are tracked through image processing technology, or physiological indicators such as heart rate variability are monitored by wearing devices. Such methods have certain effects in controlled environments, but their applicability faces challenges in real dynamic flight scenarios. Environmental vibration, acceleration overload and necessary head observation movements of pilots during flight will significantly interfere with single signal sources, resulting in insufficient stability of feature extraction and making the discrimination of fatigue state prone to deviation. Further, due to the lack of effective association and fusion of multi-dimensional information, the existing technology is difficult to distinguish the characteristic changes caused by physiological fatigue from the data fluctuations caused by normal operation behaviors. At the same time, most systems focus on snapshot judgment of instantaneous state and fail to associate the individual fatigue accumulation mode of pilots with the task load of different flight stages. Such static evaluation mode is difficult to realize early warning and personalized intervention of fatigue risk. SUMMARY

[0004] The present application aims to provide a pilot visual fatigue detection system and method based on artificial intelligence to solve the problems raised in the background.

[0005] In order to solve the above technical problems, the present application provides the following technical solution: a pilot visual fatigue detection method based on artificial intelligence, comprising: S1. continuously acquiring the face video stream of the pilot through the image sensor deployed in the cockpit, detecting and locating the key points of the face through the pre-trained convolutional neural network model, and continuously tracking the face region state based on the Kalman filtering algorithm; S2. Based on the located facial key feature points, segment the binocular region image sequence from the video frames, perform time-frequency domain analysis on the sequence, and extract physiological indicators representing visual function, including critical flicker fusion frequency estimate value calculated based on image brightness change sequence, and blink frequency obtained by detecting eyelid opening and closing state changes; at the same time, according to the spatial position changes of the facial feature points, calculate the attitude angle of the head and its motion frequency; S3. The extracted critical flicker fusion frequency estimate value, blink frequency and head attitude angle data constitute a multi-dimensional feature vector, which is input into the pre-trained visual fatigue evaluation model, and an initial visual fatigue index is output; S4. Access the pilot's historical flight database, call the historical fatigue index sequence and corresponding flight state parameters of the pilot in the past preset period and different flight stages, analyze the individual fatigue evolution law, establish a personalized baseline, and dynamically calibrate the initial score according to the matching degree of the current flight stage and the historical law, to generate an optimized visual fatigue score that integrates current physiological response and historical behavior pattern; S5. Compare the optimized visual fatigue score with the preset multiple fatigue level thresholds to determine the current fatigue level; and according to the determined level, automatically trigger the corresponding grading warning strategy.

[0006] Further, S1 includes: S1.1: Deploy high-definition image sensors at preset positions in the cockpit, and pre-process the collected raw video stream, including grayscale conversion, Gaussian filter noise reduction and contrast enhancement; S1.2: Input the pre-processed video frames into the pre-trained convolutional neural network model, which takes the pre-processed video frames as input and contains feature extraction layer and key point regression layer; wherein the feature extraction layer adopts residual network structure, extracts facial global features through multiple layers of convolution and residual connection; the key point regression layer outputs the coordinate data of facial contour, eye region and head skeleton feature points based on the feature information output by the feature extraction layer, forming a feature point set; the model training uses a sample data set labeled with facial feature points, and the deviation between predicted coordinates and true coordinates is used as the target to iteratively adjust the model parameters through the loss function; S1.3: Based on the initial feature point coordinates located, construct a Kalman filter state equation, set the feature point pixel coordinates and motion speed as state variables, and take the displacement error of adjacent feature points as observation value, to realize continuous tracking of the facial region in space by recursively calculating and correcting the feature point position offset in real time.

[0007] Further, S2 includes: S2.1: Select eye region feature point set E from the feature point set output from S1, containing upper eyelid edge point U(i), lower eyelid edge point L(i) and pupil boundary point P(j), where i, j are feature point serial numbers, i takes value in the range of 1 to n, j takes value in the range of 1 to m, n and m are the total number of eyelid and pupil feature points respectively; fit the polygon boundary of the binocular region by least square method, calculate the minimum circumscribed rectangle R of the polygon, the parameters of the rectangle are defined by the upper left corner coordinates (x_min, y_min) and the lower right corner coordinates (x_max, y_max); based on the coordinate range of rectangle R, segment the binocular region image sequence I(t) from the video frame, t is the time frame index, representing the t-th frame in the video sequence; S2.2: Perform grayscale and normalization processing on the binocular region image sequence I(t) to obtain the sequence L(t) of the change of the region average brightness with time; perform fast Fourier transform on L(t) to extract the power spectral density feature of the brightness sequence L(t) in the preset frequency band range related to physiological tremor; based on the power spectral density feature, calculate the frequency domain index value by calculating the ratio of the energy of a specific frequency band to the total energy; based on the binocular region image sequence I(t), calculate the estimated value of the critical flicker fusion frequency CFF by analyzing the dynamic response characteristics of the pupil region to the change of light intensity; the dynamic response characteristics are represented by the response delay and amplitude of the contraction and expansion of the pupil region under the light stimulation mode of the preset frequency; S2.3: Define eyelid opening degree reference value D as the average value of the vertical distance between the upper and lower eyelid feature points U(i) and L(i) of the pilot in the clear reference state for a plurality of consecutive frames, that is, D = mean[d(U(i), L(i))]; calculate the eyelid opening degree Dt of the current frame in real time, which is the ratio of the vertical distance of the current frame to the reference value D, that is, Dt = d_current(U(i), L(i)) / D; when the value of Dt is less than or equal to the preset proportion threshold and the duration of this state reaches the preset number of consecutive frames N, it is determined as an effective blink; calculate the blink frequency F by the formula F = B / T, where B is the number of effective blinks counted in the statistical time T.

[0008] S2.4: Select head feature point set H for calculating the attitude, including eyebrow center point O, left temporal point T1, right temporal point T2 and mandibular point J; construct a three-dimensional coordinate system of the face with the eyebrow center point O as the origin; solve the coordinates of the feature points in the three-dimensional coordinate system through the two-dimensional and three-dimensional correspondence of the feature points and the camera perspective projection model; calculate the pitch angle α, yaw angle β and roll angle γ of the head by using the Euler angle conversion formula; S2.5: first-order difference operation is performed on the head posture angle sequence [a(t), b(t), g(t)] of the continuous frames to obtain the change rates of each posture angle between adjacent frames, namely, Da, Db, and Dg; a threshold value of angle change rate, denoted as DQ, is set, and when the value of max(Da, Db, Dg) is greater than or equal to the threshold value DQ, it is recorded as a significant head movement; the head movement frequency M is calculated by the formula M = K / T, wherein K is the number of significant movements obtained in the statistical time T.

[0009] Further, S3 includes: S3.1: feature processing is performed on the critical flicker fusion frequency estimate CFF, the blink frequency F, and the head posture angle extracted in S2: the correlation degree between any two features is calculated, and for the feature pairs with a correlation degree reaching a preset threshold, a feature interaction term is constructed to reflect the synergistic effect between the features; the original features and the interaction term are processed by using a standardization formula, and the formula is X' = (X - m) / s, wherein X is the original feature or the interaction term to be standardized, m is the mean value of the object to be standardized in the pilot and the flight phase two-dimensional sample set, and s is the standard deviation of the corresponding dimensional sample set; the standardized original features and the feature interaction terms are combined in the order of visual physiological indicators, head movement features, and feature interaction terms to construct a multi-dimensional feature vector V = [CFF', F', a', b', g', A', B'], wherein CFF' is the standardized critical flicker fusion frequency estimate, F' is the standardized blink frequency, a' is the standardized head pitch angle, b' is the standardized head yaw angle, g' is the standardized head roll angle, A' is the standardized interaction term of the critical flicker fusion frequency CFF and the blink frequency F, and B' is the standardized interaction term of the head pitch angle a and the roll angle g; S3.2: Construct a pre-trained visual fatigue assessment model, including feature attention processing, time sequence local feature extraction and bidirectional time sequence dependence capture. The feature attention processing takes a multi-dimensional feature vector V as input, calculates the attention weight Wk of each feature through a fully connected layer, and the formula is Wk=Softmax[W·V+b], wherein W is a weight matrix, and b is a bias term. The attention-enhanced feature vector V' is obtained by weighted summation, wherein Vk is the feature component at the kth position in the multi-dimensional feature vector V, and Wk is the attention weight corresponding to the feature component. The time sequence local feature extraction takes the time sequence segment of V' as input, adopts a dynamic convolution kernel, and processes the time sequence segment through 2 layers of dynamic convolution and 1 layer of adaptive pooling. The bidirectional time sequence dependence capture adopts a bidirectional structure, and the forward part and the backward part process the output results of the time sequence local feature extraction respectively, and residual connection is set in the hidden layer. During the training of the visual fatigue assessment model, the basic parameters are first trained using a large-scale general pilot fatigue sample set, and then the historical physiological data of a single pilot is used to adjust the parameters. Dynamic sample weights are set in the loss function, and the high-risk fatigue samples are given a weight Wh, and the normal state samples are given a weight Wb, wherein Wh is greater than Wb, and the loss function formula is Loss=∑(Wi×E(Yi,Yi')), wherein Wi is a single sample weight, Yi is a sample true fatigue grade, Yi' is a model predicted fatigue grade, and E(Yi,Yi')=-Yi×ln(Yi')-(1-Yi)×ln(1-Yi') is used to quantify the difference between the sample true fatigue grade and the model predicted fatigue grade. The feature vector V constructed in step S3.1 is input into the trained model, and the model outputs a preliminary visual fatigue index S1 in the range of [0, 100] and a corresponding confidence label.

[0010] Further, S4 includes: S4.1: Access the pilot historical flight database, call the historical data of the pilot in the past preset period, including the historical fatigue index sequence H=[h1, h2, …, hk] corresponding to different flight stages, wherein hk is the fatigue index of the pilot in the kth flight corresponding to the stage, and the historical flight state parameter S=[s1, s2, …, sk], wherein sk is the flight state parameter of the pilot in the kth flight corresponding to hk, including the duration, altitude, cabin temperature and pre-rest duration of the corresponding stage. The historical fatigue index is classified and counted according to the flight stage, the mean, standard deviation and fatigue accumulation rate k of each stage are calculated, and the fatigue accumulation rate k is the slope obtained by linear fitting the relationship between the index and the flight duration. The "stage identifier, mean, standard deviation, fatigue accumulation rate, historical index sequence" of each stage are combined to construct an individual fatigue rule data set D. S4.2: Establishing an individualized baseline based on the dataset D: set a baseline reference value μ0 for each flight phase, set a fluctuation interval [μ0-λ×σh, μ0+λ×σh] in combination with σh, λ is an adjustment coefficient, adapt to the individual fluctuation characteristics, and take kh as the baseline dynamic adjustment coefficient, form the individualized dynamic baseline B of this phase [μ0, σh, kh, λ×σh], wherein kh is the historical fatigue accumulation rate of this flight phase; obtain the current flight state parameters C, including the current flight phase identifier, the flown time tp, the current flight altitude hp, the cabin temperature te, and the pilot's rest time re before this flight, calculate the similarity of the current state parameters and each historical flight state parameter in the individual fatigue rule dataset D, and use the weighted Euclidean distance formula: ; wherein tk is the duration of the corresponding phase of the historical flight, hk is the altitude of the corresponding phase of the historical flight, tek is the cabin temperature of the corresponding phase of the historical flight, rk is the pre-rest time of the corresponding phase of the historical flight, w1, w2, w3, and w4 are preset weights, wherein the weight proportions of w1 and w4 are higher than those of w2 and w3; filter the historical data with a similarity Sim(C, sk)≤preset similarity threshold, and take the mean value of the fatigue accumulation rate kh in the corresponding historical data as the dynamic adjustment coefficient kp of the current phase.

[0011] S4.3: Match the individualized dynamic baseline B0 corresponding to the current phase, and the constituent elements are [baseline reference value μ0, historical standard deviation σh, historical fatigue accumulation rate kh, and fluctuation interval half-width λ×σh]; calibrate the preliminary fatigue index S1 output by S3 in combination with kp: calculate the ratio r of the current flown time and the standard time of the phase; calculate the calibration offset Δc according to kp and r: Δc=kp×r×(S1-B0[0]), wherein B0[0] is the baseline reference value μ0, and Δc is used to correct the deviation of the current fatigue index from the baseline reference value, and simultaneously incorporate the influence of the fatigue accumulation rate; calculate the optimized visual fatigue score by the formula S=S1+Δc; if S exceeds the fluctuation interval [μ0-λ×σh, μ0+λ×σh] of B0, then perform a second adjustment on Δc: Δc'=Δc×(B0[3]-|S1-B0[0]|) / B0[3], wherein B0[3] is the fluctuation interval half-width λ×σh; Finally, output the adjusted S as the optimized visual fatigue score that comprehensively considers the current physiological response and historical behavior pattern.

[0012] S5 presets multiple sets of fatigue level threshold values, and divides fatigue levels from low to high into four levels of no fatigue, mild fatigue, moderate fatigue, and severe fatigue. The threshold value interval corresponding to each level is determined by the level boundary characteristics of a large-scale pilot fatigue sample statistics, and is fine-tuned according to the individual fatigue tolerance of different pilots: first, collect a large-scale pilot fatigue sample data, including optimized fatigue scores in different flight stages and different environments, and actual operation performance data of the corresponding samples, cluster the sample scores by the K-means clustering algorithm, extract the cluster centers as the initial level boundary characteristics, and then verify them in combination with the operation performance data: substitute the scores corresponding to the initial boundary characteristics into the sample set, calculate the correlation of the scores and the operation failure rate under each boundary, adjust the boundary value to make the correlation reach a preset threshold, determine the universal threshold interval, and record it as no fatigue [G1, G2), mild fatigue [G2, G3), moderate fatigue [G3, G4), and severe fatigue [G4, G5]. Then, based on the historical flight database of the current pilot, extract the optimized scores corresponding to each fatigue level in the past few months and the operation performance data in the same period, divide the pilot's historical scores according to the universal threshold interval, calculate the average operation failure rate R_ind in each universal level interval; At the same time, the average operation failure rate R_gen of the large-scale universal sample in the corresponding level interval is called; Calculate the deviation rate AR = |R_ind - R_gen| / R_gen, if AR > preset deviation threshold, then fine-tune the level threshold interval as follows: If R_ind < R_gen: increase the lower limit value of the level interval, the increase step size is AR x (G3-G2), the new lower limit is G2+Δq, and the interval becomes [G2+Δq, G3); if the adjusted interval overlaps with the previous level interval, the upper limit value of the previous level interval is adjusted to G2+Δq simultaneously to make the intervals continuous and non-overlapping; If R_ind > R_gen: decrease the lower limit value of the level interval, the decrease step size is AR x (G3-G2), the new lower limit is G2-Δq, and the interval becomes [G2-Δq, G3); if the adjusted interval overlaps with the previous level interval, the upper limit value of the previous level interval is adjusted to G2-Δq; Repeat the above calculation and adjustment until AR of all level intervals is less than or equal to the preset deviation threshold, and form the personalized threshold interval adapted to the pilot; compare the optimized visual fatigue score output by S4 with the personalized threshold interval to determine the current fatigue level, and trigger the corresponding graded warning strategy.

[0013] A pilot visual fatigue detection system based on artificial intelligence, comprising a data acquisition module, a feature extraction module, a preliminary fatigue index calculation module, a personalized fatigue score calibration module, and a fatigue level determination module. The data acquisition module continuously acquires the face video stream of the pilot through the image sensor deployed in the cockpit, and detects and locates the face key points of the video frame through the pre-trained convolutional neural network model; and continuously tracks the face region state based on the Kalman filtering algorithm; The feature extraction module segments the eye region image sequence from the video frame based on the located face key feature points, performs time-frequency domain analysis on the sequence, and extracts physiological indicators representing visual function, including the critical flicker fusion frequency estimate value calculated based on the image brightness change sequence, and the blink frequency obtained by detecting the eyelid opening and closing state change; at the same time, according to the spatial position change of the face feature points, the attitude angle and the motion frequency of the head are calculated; The preliminary fatigue index calculation module constructs a multi-dimensional feature vector with the extracted critical flicker fusion frequency estimate value, blink frequency and head attitude angle data, inputs it into the pre-trained visual fatigue evaluation model, and outputs a preliminary visual fatigue index; The personalized fatigue score calibration module accesses the historical flight database of the pilot, calls the historical fatigue index sequence and corresponding flight state parameters of the pilot in the past preset period and different flight stages, analyzes the individual fatigue evolution law, establishes a personalized baseline, and dynamically calibrates the initial score according to the matching degree of the current flight stage and the historical law, to generate an optimized visual fatigue score that comprehensively considers the current physiological response and historical behavior pattern; The fatigue level determination module compares the optimized visual fatigue score with the preset multiple fatigue level thresholds to determine the current fatigue level; and according to the determined level, automatically triggers the corresponding graded warning strategy.

[0014] Compared with the prior art, the present application has the following advantages: The present application breaks through the limitation of the existing fatigue detection method which only relies on a single physiological signal or a general model and ignores individual differences, through the whole process design of face feature tracking, multi-dimensional physiological indicator extraction and personalized score calibration. Firstly, the convolutional neural network and Kalman filter are used to obtain the eye region features and head attitude, and multi-dimensional visual function indicators such as critical flicker fusion frequency are extracted. Then, the individual fatigue baseline is constructed by combining the historical flight data of the pilot, and the score is dynamically calibrated according to the matching degree of the current flight state and the historical law, realizing the innovation from general detection to individual precise evaluation, effectively improving the accuracy and adaptability of fatigue detection. At the same time, in the fatigue level determination and warning link, the general threshold is determined through large-scale sample statistics, and the individual operation performance data is fine-tuned to ensure that the level division fits the fatigue tolerance of different pilots. On this basis, the graded warning strategy is designed to provide more targeted protection for flight safety. Compared with the existing fixed threshold and single warning scheme, the practicality and safety are significantly improved. BRIEF DESCRIPTION OF DRAWINGS

[0015] The accompanying drawings are included to provide a further understanding of the application and are incorporated in and constitute a part of the specification, illustrate embodiments of the application and are used to explain the application, but are not intended to limit the application. In the drawings: Figure 1 A method flowchart of a pilot visual fatigue detection method based on artificial intelligence. DETAILED DESCRIPTION

[0016] The technical solutions in the embodiments of the application will be clearly and completely described below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, rather than all the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the application.

[0017] Please refer to Figure 1 The application provides a technical solution: a pilot visual fatigue detection method based on artificial intelligence, comprising: S1. Continuously acquiring a face video stream of a pilot through an image sensor deployed in a cockpit, performing face detection and key point positioning on a video frame through a pre-trained convolutional neural network model, and continuously tracking a face region state based on a Kalman filtering algorithm; S2. Based on the positioned face key feature points, segmenting a binocular region image sequence from the video frame, performing time-frequency domain analysis on the sequence, and extracting physiological indicators representing visual function, including a critical flicker fusion frequency estimate value calculated based on an image brightness change sequence and a blink frequency obtained by detecting eyelid opening and closing state changes; at the same time, according to the spatial position changes of the face feature points, calculating the attitude angle and motion frequency of the head; S3. Constructing a multi-dimensional feature vector with the extracted critical flicker fusion frequency estimate value, blink frequency and head attitude angle data, inputting the multi-dimensional feature vector into a pre-trained visual fatigue evaluation model, and outputting a preliminary visual fatigue index; S4. Accessing a historical flight database of the pilot, calling a historical fatigue index sequence and corresponding flight state parameters of the pilot at different flight stages in a preset period in the past, analyzing the individual fatigue evolution law, establishing a personalized baseline, and dynamically calibrating the initial score according to the matching degree of the current flight stage and the historical law, to generate an optimized visual fatigue score that comprehensively considers the current physiological response and historical behavior pattern; S5. Comparing the optimized visual fatigue score with a plurality of preset fatigue level thresholds to determine the current fatigue level; and according to the determined level, automatically triggering a corresponding graded early warning strategy.

[0018] S1 includes: S1.1: Deploy high-definition image sensors at preset positions in the cockpit, and preprocess the collected raw video stream, including grayscale conversion, Gaussian filter denoising, and contrast enhancement; S1.2: Input the preprocessed video frames into a pre-trained convolutional neural network model, which takes the preprocessed video frames as input and contains a feature extraction layer and a key point regression layer; the feature extraction layer uses a residual network structure to extract global facial features through multiple layers of convolution and residual connection; the key point regression layer outputs the coordinates of facial contours, eye region, and head skeleton feature points based on the feature information output by the feature extraction layer, forming a feature point set; the model is trained using a sample data set labeled with facial feature points, and the deviation between predicted coordinates and true coordinates is used as the target to iteratively adjust the model parameters through a loss function; S1.3: Based on the initial feature point coordinates obtained by positioning, construct a Kalman filter state equation, set the feature point pixel coordinates and motion velocity as state variables, and use the displacement error of adjacent frames of feature points as observation values to recursively calculate and real-time correct the feature point position offset, achieving continuous tracking of the facial region in space.

[0019] S2 includes: S2.1: Select the eye region feature point set E from the feature point set output by S1, which contains upper eyelid edge points U(i), lower eyelid edge points L(i), and pupil boundary points P(j), where i and j are feature point serial numbers, i ranges from 1 to n, j ranges from 1 to m, n and m are the total number of eyelid and pupil feature points respectively; fit the polygon boundary of the bilateral eye region by least squares method, calculate the minimum circumscribed rectangle R of the polygon, and the parameters of the rectangle are defined by the upper left corner coordinates (x_min, y_min) and the lower right corner coordinates (x_max, y_max); based on the coordinate range of rectangle R, segment the bilateral eye region image sequence I(t) from the video frame, where t is the time frame index, representing the t-th frame in the video sequence; S2.2: Perform grayscale and normalization processing on the bilateral eye region image sequence I(t) to obtain the sequence L(t) of the average brightness of the region varying with time; perform fast Fourier transform on L(t) to extract the power spectral density features of the brightness sequence L(t) in the preset frequency band range related to physiological tremor; based on the power spectral density features, calculate the frequency domain index value by calculating the ratio of the energy in a specific frequency band to the total energy; based on the bilateral eye region image sequence I(t), calculate the estimated value of the critical flicker fusion frequency CFF by analyzing the dynamic response characteristics of the pupil region to light intensity changes; the dynamic response characteristics are represented by the response delay and amplitude of the contraction and expansion of the pupil region under the light stimulation mode of the preset frequency; S2.3: define the eyelid opening degree reference value D as the average of the vertical distance between the upper eyelid feature point U(i) and the lower eyelid feature point L(i) in a plurality of consecutive frames when the pilot is in a sober reference state, i.e. D = mean[d(U(i), L(i))]; calculate the eyelid opening degree Dt of the current frame in real time as the ratio of the vertical distance of the current frame to the reference value D, i.e. Dt = d_current(U(i), L(i)) / D; when the value of Dt is less than or equal to the preset proportion threshold and the duration of this state reaches the preset number of consecutive frames N, it is determined as one valid blink; calculate the blink frequency F by the formula F = B / T, wherein B is the number of valid blinks counted in the time length T.

[0020] S2.4: select the head feature point set H for calculating the posture, including the eyebrow center point O, the left temporal point T1, the right temporal point T2 and the chin point J; construct a three-dimensional coordinate system of the face with the eyebrow center point O as the origin; solve the coordinates of the feature points in the three-dimensional coordinate system through the two-dimensional and three-dimensional correspondence relationship of the feature points and the camera perspective projection model; calculate the pitch angle α, the yaw angle β and the roll angle γ of the head by using the Euler angle conversion formula; S2.5: perform first-order difference operation on the head posture angle sequence [α(t), β(t), γ(t)] of consecutive frames to obtain the change rates Δα, Δβ and Δγ of each posture angle between adjacent frames; set the angle change rate threshold Δθ, and when the value of max(Δα, Δβ, Δγ) is greater than or equal to the threshold Δθ, it is recorded as one significant head movement; calculate the head movement frequency M by the formula M = K / T, wherein K is the number of significant movements counted in the time length T.

[0021] S3 includes: S3.1: Feature processing is performed on the critical flicker fusion frequency estimate CFF, the blink frequency F, and the head posture angle extracted in S2: the correlation degree between any two features is calculated, and for the feature pairs whose correlation degree reaches a preset threshold, a feature interaction term is constructed to reflect the synergistic effect between the features; a standardization formula is used to process the original features and the interaction term, and the formula is X'=(X-μ) / σ, where X is the original feature or the interaction term to be standardized, μ is the mean of the to-be-standardized object in the pilot and flight phase two-dimensional sample set, and σ is the standard deviation of the corresponding dimensional sample set; the standardized original features and the feature interaction term are combined in the order of visual physiological indicators, head movement features, and feature interaction terms to construct a multi-dimensional feature vector V=[CFF', F', α', β', γ', A', B'], wherein CFF' is the standardized critical flicker fusion frequency estimate, F' is the standardized blink frequency, α' is the standardized head pitch angle, β' is the standardized head yaw angle, and γ' is the standardized head roll angle; A' is the standardized interaction term of the critical flicker fusion frequency CFF and the blink frequency F, and B' is the standardized interaction term of the head pitch angle α and the roll angle γ; S3.2: Construct a pre-trained visual fatigue assessment model, including feature attention processing, time sequence local feature extraction and bidirectional time sequence dependence capture. The feature attention processing takes a multi-dimensional feature vector V as input, calculates the attention weight Wk of each feature through a fully connected layer, and the formula is Wk=Softmax[W·V+b], wherein W is a weight matrix, and b is a bias term. The attention-enhanced feature vector V' is obtained by weighted summation, wherein Vk is the feature component at the kth position in the multi-dimensional feature vector V, and Wk is the attention weight corresponding to the feature component. The time sequence local feature extraction takes the time sequence segment of V' as input, adopts a dynamic convolution kernel, and processes the time sequence segment through 2 layers of dynamic convolution and 1 layer of adaptive pooling. The bidirectional time sequence dependence capture adopts a bidirectional structure, and the forward part and the backward part process the output results of the time sequence local feature extraction respectively, and residual connection is set in the hidden layer. During the training of the visual fatigue assessment model, the basic parameters are first trained using a large-scale general pilot fatigue sample set, and then the parameters are adjusted using the historical physiological data of a single pilot. Dynamic sample weights are set in the loss function, and the high-risk fatigue samples are assigned a weight Wh, and the normal state samples are assigned a weight Wb, wherein Wh is greater than Wb, and the loss function formula is Loss=∑(Wi×E(Yi,Yi')), wherein Wi is a single sample weight, Yi is a sample true fatigue grade, Yi' is a model predicted fatigue grade, and E(Yi,Yi')=-Yi×ln(Yi')-(1-Yi)×ln(1-Yi') is used to quantify the difference between the sample true fatigue grade and the model predicted fatigue grade. The feature vector V constructed in step S3.1 is input into the trained model, and the model outputs a preliminary visual fatigue index in the range of [0, 100] and a corresponding confidence label.

[0022] S4 includes: S4.1: Access the pilot historical flight database, call the historical data of the pilot in the past preset period, including the historical fatigue index sequence H=[h1, h2, …, hk] corresponding to different flight stages, wherein hk is the fatigue index of the pilot in the kth flight corresponding to the stage, and the historical flight state parameter S=[s1, s2, …, sk], wherein sk is the flight state parameter of the pilot in the kth flight corresponding to hk, including the duration, altitude, cabin temperature and pre-rest duration of the corresponding stage. The historical fatigue index is classified and counted according to the flight stage, the mean, standard deviation and fatigue accumulation rate k of each stage are calculated, and the fatigue accumulation rate k is the slope obtained by linear fitting the relationship between the index and the flight duration. The "stage identifier, mean, standard deviation, fatigue accumulation rate, historical index sequence" of each stage are combined to construct an individual fatigue rule data set D. S4.2: Establishing an individualized baseline based on the dataset D: set a baseline reference value μ0 for each flight phase, set a fluctuation interval [μ0-λ×σh, μ0+λ×σh] in combination with σh, λ is an adjustment coefficient, adapt to the individual fluctuation characteristics, and take kh as the baseline dynamic adjustment coefficient, form the individualized dynamic baseline B of this phase [μ0, σh, kh, λ×σh], wherein kh is the historical fatigue accumulation rate of this flight phase; obtain the current flight state parameters C, including the current flight phase identifier, the flown time tp, the current flight altitude hp, the cabin temperature te, and the pilot's rest time re before this flight, calculate the similarity of the current state parameters and each historical flight state parameter in the individual fatigue rule dataset D, and use the weighted Euclidean distance formula: ; wherein tk is the duration of the corresponding phase of the historical flight, hk is the altitude of the corresponding phase of the historical flight, tek is the cabin temperature of the corresponding phase of the historical flight, rk is the pre-rest time of the corresponding phase of the historical flight, w1, w2, w3, and w4 are preset weights, wherein the weight proportions of w1 and w4 are higher than those of w2 and w3; filter the historical data with a similarity Sim(C, sk)≤preset similarity threshold, and take the mean value of the fatigue accumulation rate kh in the corresponding historical data as the dynamic adjustment coefficient kp of the current phase.

[0023] S4.3: Match the individualized dynamic baseline B0 corresponding to the current phase, and the constituent elements are [baseline reference value μ0, historical standard deviation σh, historical fatigue accumulation rate kh, and fluctuation interval half-width λ×σh]; calibrate the preliminary fatigue index S1 output by S3 in combination with kp: calculate the ratio r of the current flown time and the standard time of the phase; calculate the calibration offset Δc according to kp and r: Δc=kp×r×(S1-B0[0]), wherein B0[0] is the baseline reference value μ0, and Δc is used to correct the deviation of the current fatigue index from the baseline reference value, and simultaneously incorporate the influence of the fatigue accumulation rate; calculate the optimized visual fatigue score by the formula S=S1+Δc; if S exceeds the fluctuation interval [μ0-λ×σh, μ0+λ×σh] of B0, then perform a second adjustment on Δc: Δc'=Δc×(B0[3]-|S1-B0[0]|) / B0[3], wherein B0[3] is the fluctuation interval half-width λ×σh; Finally, output the adjusted S as the optimized visual fatigue score that comprehensively considers the current physiological response and historical behavior pattern.

[0024] S5 preset multiple sets of fatigue level threshold values, divided into four levels of no fatigue, mild fatigue, moderate fatigue, and severe fatigue from low to high fatigue degree, and the threshold interval of each level is determined by the grade boundary characteristics of large-scale pilot fatigue sample statistics, and is fine-tuned according to the individual fatigue tolerance of different pilots (extracting the correlation data between the past fatigue level and the actual operation performance of the pilot from the historical flight database); When determining the threshold interval, first collect large-scale pilot fatigue sample data, including optimized fatigue scores in different flight stages and different environments, and actual operation performance data of the corresponding samples, cluster the sample scores by K-means clustering algorithm, extract the cluster center as the initial grade boundary feature, and then verify it by combining the operation performance data: put the scores corresponding to the initial boundary feature into the sample set, calculate the correlation between the scores and the operation failure rate under each boundary, adjust the boundary value to make the correlation reach the preset threshold, determine the general threshold interval, and record it as no fatigue [G1, G2), mild fatigue [G2, G3), moderate fatigue [G3, G4), and severe fatigue [G4, G5]); Then, based on the historical flight database of the current pilot, extract the optimized scores of each fatigue level in the past few months and the operation performance data in the same period, divide the pilot's historical scores according to the general threshold interval, calculate the average operation failure rate R_ind in each general level interval; At the same time, the average operation failure rate R_gen of large-scale general samples in the corresponding level interval is called; Calculate the deviation rate AR = |R_ind-R_gen| / R_gen, if AR> preset deviation threshold, then fine-tune the level threshold interval as follows: If R_ind < R_gen: increase the lower limit value of the level interval, the increase step size is AR x (G3-G2), the new lower limit is G2+Δq, and the interval becomes [G2+Δq, G3); if the adjusted interval overlaps with the previous interval, the upper limit value of the previous interval is adjusted to G2+Δq simultaneously to make the intervals continuous and non-overlapping; If R_ind > R_gen: decrease the lower limit value of the level interval, the decrease step size is AR x (G3-G2), the new lower limit is G2-Δq, and the interval becomes [G2-Δq, G3); if the adjusted interval overlaps with the previous interval, the upper limit value of the previous interval is adjusted to G2-Δq; Repeat the above calculation and adjustment until AR≤ preset deviation threshold for all level intervals, forming a personalized threshold interval adapted to the pilot; compare the optimized visual fatigue score output by S4 with the personalized threshold interval to determine the current fatigue level, and trigger the corresponding graded warning strategy.

[0025] The pilot flying the Beijing-Shanghai route of a certain airline is taken as the detection object in this embodiment, and the specific implementation steps are as follows: A 200-megapixel high-definition image sensor is deployed above the cockpit instrument panel to collect the pilot's facial video stream; the original video stream is preprocessed (gray scale conversion, 5x5 Gaussian filter noise reduction, contrast adaptive enhancement); the preprocessed frame is input into the pre-trained residual network model, and the facial contour, eye area (upper eyelid 12 points, lower eyelid 12 points, pupil 8 points) and head skeleton (brow, left and right temporal, mandible) feature point set is output; based on the feature point coordinates, a Kalman filter equation is constructed, the state variable contains pixel coordinates and motion speed, and the offset is corrected in real time to realize continuous tracking of the face; From the feature point set, the eye area point set is selected, the polygon in the eye area is fitted, and the minimum circumscribed rectangle is taken to segment out the eye image sequence I(t); I(t) is normalized by gray scale conversion to obtain the brightness sequence L(t), and the 0.1-0.3Hz frequency band power spectrum density is extracted by fast Fourier transform to calculate the frequency domain index, combined with the response of the pupil to 5Hz light stimulation (delay <0.5s, amplitude >15% effective), to obtain the CFF estimate value; define the eyelid opening degree reference value D=10mm in the awake state, calculate Dt in real time, when Dt≤0.3 and lasts for 3 frames, judge effective blinking, and count the number of times in 60s to get F; select the head feature points to construct a three-dimensional coordinate system, and calculate the coordinates to get α, β, γ by Euler angle formula, and first-order difference of continuous frame attitude angle, set Δθ=5° / frame, count the number of significant movements in 60s to get M; Correlation degree of CFF, F, α, β, γ is calculated, CFF and F, α and γ are taken as correlation pairs, and interaction terms are constructed; according to the standardization X'=(X-μ) / σ, μ and σ are taken from the "pilot-cruise phase" sample set, and the vector V=[CFF', F', α', β', γ', CFF'xF', α'xγ'] is obtained; input the pre-trained model, and after attention processing, W is a 128-dimensional matrix, and b=0.1; 2 layers of dynamic convolution, bidirectional structure, output preliminary index S1=42 (confidence 92%); The pilot's past 6 months of Beijing-Shanghai route history data is called to get the cruise phase H=[38, 40, 45], S includes continuous time length 100min, height 10km, etc., and μ=41, σh=2.6, kh=0.05 are calculated to construct the data set D; the baseline B=[41, 2.6, 0.05, 1.3] is established, the current C_state contains 40min of flight, and the height is 10km, the similarity Sim=0.8 (≤threshold 1.0) with the historical data is calculated, and kp=0.045 is taken; r=40 / 100=0.4, Δc=0.045x0.4x(42-41)=0.018, S=42+0.018≈42.02, which does not exceed the interval [39.7, 42.3], and the optimized score 42.02 is obtained; General threshold setting [G1=0, G2=20, G3=45, G4=70, G5=100], extract the pilot historical data, get cruise phase R_ind=3% (R_gen=5%), ΔR=40%>15%, increase the mild fatigue lower limit Δq=40%×25=10, get the new interval [30, 45); the optimization score 42.02 belongs to mild fatigue, trigger the first level early warning: the display screen non-core area displays white prompt, plays 0.5s low volume prompt sound.

[0026] It will be apparent to those skilled in the art that the application is not limited to the details of the above-exemplified embodiments and that the application can be implemented in other particular forms without departing from the spirit or essential characteristics of the application. The embodiments should therefore be considered in all respects as illustrative and not restrictive, the scope of the application being indicated by the appended claims rather than by the above description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. No reference signs in the claims should be considered as limiting the scope of the claims with respect to the figures of the application.

Claims

1. A pilot visual fatigue detection method based on artificial intelligence, characterized in that: The method includes: S1. The system continuously acquires facial video streams of the pilots through image sensors deployed in the cockpit, performs facial detection and key point localization on the video frames using a pre-trained convolutional neural network model, and continuously tracks the facial region status based on the Kalman filter algorithm. S2. Based on the located key facial feature points, segment the binocular region image sequence from the video frame, perform time-frequency domain analysis on the sequence, and extract physiological indicators characterizing visual function, including the critical flash fusion frequency estimate calculated based on the image brightness change sequence, and the blink frequency obtained by detecting changes in eyelid opening and closing; at the same time, calculate the head posture angle and its motion frequency based on the spatial position changes of facial feature points. S3. The extracted critical flash fusion frequency estimate, blink frequency, and head posture angle data are combined to form a multi-dimensional feature vector, which is then input into the pre-trained visual fatigue assessment model to output a preliminary visual fatigue index. S4. Access the pilot's historical flight database, retrieve the historical fatigue index sequence and corresponding flight status parameters of the pilot in different flight stages within the past preset period, analyze the individual fatigue evolution pattern, establish a personalized baseline, and dynamically calibrate the initial score based on the matching degree between the current flight stage and the historical pattern to generate an optimized visual fatigue score that integrates the current physiological response and historical behavioral patterns. S5. Compare the optimized visual fatigue score with multiple preset fatigue level thresholds to determine the current fatigue level; and automatically trigger the corresponding graded early warning strategy based on the determined level.

2. The pilot visual fatigue detection method based on artificial intelligence according to claim 1, characterized in that: S1 includes: S1.1: Deploy high-definition image sensors at preset locations within the cockpit to preprocess the acquired raw video streams, including grayscale conversion, Gaussian filtering for noise reduction, and contrast enhancement. S1.2: The preprocessed video frames are input into a pre-trained convolutional neural network model. This model takes the preprocessed video frames as input and includes a feature extraction layer and a keypoint regression layer. The feature extraction layer adopts a residual network structure, extracting global facial features through multiple convolutions and residual connections. The keypoint regression layer outputs the coordinate data of facial contours, periorbital regions, and head bone feature points based on the feature information output by the feature extraction layer, forming a feature point set. The model is trained using a sample dataset of labeled facial feature points, with the deviation between predicted coordinates and true coordinates as the target, and the model parameters are iteratively adjusted through a loss function. S1.3: Based on the initial feature point coordinates obtained from the localization, a Kalman filter state equation is constructed. The feature point pixel coordinates and motion velocity are set as state variables. The displacement error of the feature points in adjacent frames is used as the observation value. The feature point position offset is corrected in real time through recursive calculation, so as to realize continuous tracking of the facial region in space.

3. The method for detecting pilot visual fatigue based on artificial intelligence according to claim 2, characterized in that: S2 includes: S2.1: Select the periorbital feature point set E from the feature point set output by S1, which includes the upper eyelid edge point U(i), the lower eyelid edge point L(i), and the pupil boundary point P(j), where i and j are the feature point indices, i ranges from 1 to n, j ranges from 1 to m, and n and m are the total number of eyelid and pupil feature points, respectively; fit the polygon boundary of the bilateral region using the least squares method, and calculate the minimum bounding rectangle R of the polygon. The parameters of the rectangle are defined by the coordinates of the upper left corner (x_min, y_min) and the lower right corner (x_max, y_max); based on the coordinate range of the rectangle R, segment the bilateral region image sequence I(t) from the video frame, where t is the time frame index, representing the t-th frame in the video sequence; S2.2: The binocular region image sequence I(t) is grayscaled and normalized to obtain the sequence L(t) showing the change of average brightness of the region over time; a fast Fourier transform is performed on L(t) to extract the power spectral density characteristics of the brightness sequence L(t) within a preset frequency band related to physiological tremor; based on the power spectral density characteristics, the frequency domain index value is calculated by calculating the ratio of energy in a specific frequency band to the total energy; based on the binocular region image sequence I(t), the estimated value of the critical flash fusion frequency CFF is calculated by analyzing the dynamic response characteristics of the pupil region to changes in light intensity; the dynamic response characteristics are represented by the response delay and amplitude of the pupil region's contraction and expansion under a preset frequency light stimulation mode; S2.3: Define the eyelid opening and closing reference value D, which is the average vertical distance between the upper and lower eyelid feature points U(i) and L(i) in multiple consecutive frames when the pilot is in a conscious reference state, i.e., D=mean[d(U(i),L(i))]; calculate the eyelid opening and closing degree Dt of the current frame in real time, which is the ratio of the vertical distance of the current frame to the reference value D, i.e., Dt=d_current(U(i),L(i)) / D; d_current(U(i),L(i)) represents the vertical distance between the i-th pair of upper eyelid feature points U(i) and lower eyelid feature points L(i) in the current video frame; when the value of Dt is less than or equal to the preset proportional threshold and the duration of this state reaches the preset number of consecutive frames N, it is determined as a valid blink; calculate the blink frequency F by the formula F=B / T, where B is the number of valid blinks obtained within the statistical duration T.

4. The pilot visual fatigue detection method based on artificial intelligence according to claim 3, characterized in that: S2 further includes: S2.4: Select a set of head feature points H for attitude calculation, including the center of the forehead O, the left temporal point T1, the right temporal point T2, and the mandibular point J; construct a three-dimensional facial coordinate system with the center of the forehead O as the origin; solve the coordinates of the feature points in the three-dimensional coordinate system by the two-dimensional and three-dimensional correspondence between the camera perspective projection model and the feature points; calculate the pitch angle α, yaw angle β, and roll angle γ of the head using the Euler angle transformation formula; S2.5: Perform a first-order difference operation on the head pose angle sequence [α(t), β(t), γ(t)] of consecutive frames to obtain the rate of change of each pose angle between adjacent frames Δα, Δβ, Δγ; set an angle change rate threshold Δθ, and when the value of max(Δα, Δβ, Δγ) is greater than or equal to the threshold Δθ, it is recorded as a significant head movement; calculate the head movement frequency M using the formula M=K / T, where K is the number of significant movements obtained within the statistical duration T.

5. The pilot visual fatigue detection method based on artificial intelligence according to claim 4, characterized in that: S3 includes: S3.1: Feature processing is performed on the critical flash fusion frequency estimates (CFF), blink frequency (F), and head attitude angle extracted in S2: The correlation between any two features is calculated; for feature pairs with a correlation reaching a preset threshold, feature interaction terms are constructed to reflect the synergistic effect between features; the original features and interaction terms are processed using a standardization formula: X'=(X-μ) / σ, where X is the original feature or interaction term to be standardized, μ is the mean of the object to be standardized in the pilot and flight phase dual-dimensional sample sets, and σ is the standard deviation of the corresponding dimension sample set; the standardized original features are then processed... The feature vector V=[CFF',F',α',β',γ',A',B'] is constructed by combining the features and interaction terms in the order of visual physiological indicators, head motion features, and feature interaction terms; where CFF' is the normalized critical flash fusion frequency estimate, F' is the normalized blink frequency, α' is the normalized head pitch angle, β' is the normalized head yaw angle, and γ' is the normalized head roll angle; A' is the normalized interaction term between the critical flash fusion frequency CFF and the blink frequency F, and B' is the normalized interaction term between the head pitch angle α and the roll angle γ. S3.2: Construct a pre-trained visual fatigue assessment model, comprising three parts: feature attention processing, temporal local feature extraction, and bidirectional temporal dependency capture. The feature attention processing takes a multi-dimensional feature vector V as input, and calculates the attention weight Wk for each feature through a fully connected layer, using the formula Wk=Softmax[W・V+b], where W is the weight matrix and b is the bias term. The attention-enhanced feature vector V'=∑(Wk×Vk) is obtained through weighted summation, where Vk is the feature component at the k-th position in the multi-dimensional feature vector V, and Wk is the attention weight corresponding to that feature component. The temporal local feature extraction takes a temporal segment of V' as input, employs a dynamic convolution kernel, and processes the temporal segment through two layers of dynamic convolution and one layer of adaptive pooling. The bidirectional temporal dependency capture adopts a bidirectional structure, with the forward and backward parts processing the output results of the temporal local feature extraction respectively, and in the hidden layer... Set up residual connections; when training the visual fatigue assessment model, first train the basic parameters with a large-scale general pilot fatigue sample set, and then adjust the parameters with the historical physiological data of a single pilot; set dynamic sample weights in the loss function, assign weight Wh to high-risk fatigue samples and weight Wb to normal state samples, where Wh is greater than Wb, and the loss function formula is Loss=∑(Wi×E(Yi,Yi')), where Wi is the weight of a single sample, Yi is the true fatigue level of the sample, and Yi' is the fatigue level predicted by the model; E(Yi,Yi')=-Yi×ln(Yi')-(1-Yi)×ln(1-Yi'), which is used to quantify the difference between the true fatigue level of the sample and the fatigue level predicted by the model; input the feature vector V constructed in step S3.1 into the trained model, and the model outputs a preliminary visual fatigue index and corresponding confidence label in the range of [0,100].

6. The pilot visual fatigue detection method based on artificial intelligence according to claim 5, characterized in that: S4 includes: S4.1: Access the pilot's historical flight database and retrieve the pilot's historical data within a preset period, including the historical fatigue index sequence H=[h1,h2,…,hk] corresponding to different flight stages, where hk is the fatigue index of the pilot in the corresponding stage of the k-th flight; and historical flight state parameters S=[s1,s2,…,sk], where sk is the flight state parameter of the pilot in the k-th flight, concurrent with hk, including the duration, altitude, cabin temperature, and pre-rest duration of the corresponding stage; classify and statistically analyze the historical fatigue index by flight stage, calculate the mean, standard deviation, and fatigue accumulation rate k for each stage, where the fatigue accumulation rate k is the slope obtained by linearly fitting the relationship between the index and flight duration; combine the "stage identifier, mean, standard deviation, fatigue accumulation rate, and historical index sequence" of each stage to construct an individual fatigue pattern dataset D; S4.2: Establishing a personalized baseline based on dataset D: A baseline reference value μ0 is set for each flight phase, and a fluctuation range [μ0-λ×σh, μ0+λ×σh] is set in conjunction with σh, where λ is an adjustment coefficient to adapt to individual fluctuation characteristics. kh is used as the dynamic adjustment coefficient for the baseline, forming the personalized dynamic baseline B=[μ0,σh,kh,λ×σh] for that phase, where kh is the historical fatigue accumulation rate for that flight phase. The current flight state parameters C are obtained, including the current flight phase identifier, flight duration tp, current flight altitude hp, cabin ambient temperature te, and pilot's pre-flight rest duration re. The similarity between the current state parameters and the historical flight state parameters in the individual fatigue pattern dataset D is calculated using the weighted Euclidean distance formula: ; Where tk is the duration of the corresponding historical flight phase, hk is the altitude of the corresponding historical flight phase, tek is the cabin temperature of the corresponding historical flight phase, rk is the pre-flight rest duration of the corresponding historical flight, and w1, w2, w3, and w4 are preset weights, with w1 and w4 having a higher weight than w2 and w3. Historical data with similarity Sim(C,sk) ≤ preset similarity threshold are selected, and the average fatigue accumulation rate kh in the corresponding historical data is taken as the dynamic adjustment coefficient kp for the current phase.

7. The pilot visual fatigue detection method based on artificial intelligence according to claim 6, characterized in that: Step S4 further includes: S4.3: Match the personalized dynamic baseline B0 corresponding to the current stage, whose components are [baseline reference value μ0, historical standard deviation σh, historical fatigue accumulation rate kh, fluctuation range half-width λ×σh]; calibrate the preliminary fatigue index S1 output by S3 in conjunction with kp: Calculate the ratio r of the current flight time to the standard time of this stage; calculate the calibration offset based on kp and r: Δc = kp × r × (S1 - B0[0]), where B0[0] is the baseline reference value μ0, and Δc is used to correct the deviation between the current fatigue index and the baseline reference value, while incorporating the influence of the fatigue accumulation rate. The visual fatigue score is optimized by formula S=S1+Δc; if S exceeds the fluctuation range of B0 [μ0-λ×σh,μ0+λ×σh], then Δc is adjusted twice: Δc'=Δc×(B0[3]-|S1-B0[0]|) / B0[3], where B0[3] is the half width of the fluctuation range λ×σh; The final output, S, is an optimized visual fatigue score that integrates current physiological responses and historical behavioral patterns.

8. The method for detecting pilot visual fatigue based on artificial intelligence according to claim 1, characterized in that: The S5 presets multiple fatigue level thresholds, classifying fatigue into four levels from low to high: no fatigue, mild fatigue, moderate fatigue, and severe fatigue. The threshold ranges corresponding to each level are determined by the level boundary features of a large-scale pilot fatigue sample statistical analysis. Furthermore, it extracts correlation data between the pilot's past fatigue levels and actual operational performance from a historical flight database for fine-tuning: First, a large-scale pilot fatigue sample data is collected, including optimized fatigue scores for different flight phases and environments, as well as corresponding actual operational performance data. The sample scores are clustered using the K-means clustering algorithm, and the cluster centers are extracted as initial level boundary features. This is then verified using operational performance data: The scores corresponding to the initial boundary features are substituted into the sample set, and the correlation between scores and operational error rates under each boundary is calculated. The boundary values ​​are adjusted to ensure the correlation reaches a preset threshold, determining a general threshold range, denoted as no fatigue [G1, G2), mild fatigue [G2, G3), moderate fatigue [G3, G4), and severe fatigue [G4, G5]). Subsequently, based on the current pilot's historical flight database, the optimized scores and operational performance data corresponding to each fatigue level over the past few months are extracted. The average operational error rate of the pilot at each level within the general threshold range is calculated. If the average error rate at a certain level deviates from the average error rate of the general sample at that level by more than a preset deviation threshold, the threshold range for that level is fine-tuned. The optimized visual fatigue score output by S4 is compared with the preset threshold ranges for each fatigue level. The current fatigue level of the pilot is determined according to the range to which the score belongs, and the corresponding graded early warning strategy is triggered.

9. An artificial intelligence-based pilot visual fatigue detection system, characterized in that: The system includes a data acquisition module, a feature extraction module, a preliminary fatigue index calculation module, a personalized fatigue score calibration module, and a fatigue level determination module. The data acquisition module continuously acquires facial video streams from the pilot using image sensors deployed in the cockpit, performs facial detection and key point localization on the video frames using a pre-trained convolutional neural network model, and continuously tracks the facial region status based on the Kalman filter algorithm. The feature extraction module segments the binocular region image sequence from the video frame based on the located key facial feature points, performs time-frequency domain analysis on the sequence, and extracts physiological indicators characterizing visual function, including the critical flash fusion frequency estimate calculated based on the image brightness change sequence, and the blink frequency obtained by detecting changes in eyelid opening and closing; at the same time, it calculates the head posture angle and its movement frequency based on the spatial position changes of facial feature points. The preliminary fatigue index calculation module constructs a multi-dimensional feature vector from the extracted critical flash fusion frequency estimate, blink frequency, and head posture angle data, and inputs it into the pre-trained visual fatigue assessment model to output a preliminary visual fatigue index. The personalized fatigue scoring calibration module accesses the pilot's historical flight database, calls up the historical fatigue index sequence and corresponding flight status parameters of the pilot in different flight stages within the past preset period, analyzes the individual fatigue evolution pattern, establishes a personalized baseline, and dynamically calibrates the initial score based on the matching degree between the current flight stage and the historical pattern, generating an optimized visual fatigue score that integrates the current physiological response and historical behavioral patterns. The fatigue level determination module compares the optimized visual fatigue score with multiple preset fatigue level thresholds to determine the current fatigue level. And based on the determined level, the corresponding graded early warning strategy will be automatically triggered.

Images