Flight crew operating status analysis method

Abstract

The present invention discloses a method for analyzing the operating status of flight personnel, including: S1, using an eye tracker to collect eye movement trajectory data of the pilot and generate eye movement time series data; using an electromyography sensor array to collect eye muscle electrical activity signals and convert them into electromyography feature vectors; S2, generating multimodal deep fused eye movement trajectory spatial position encoding data from the eye movement trajectory; S3, fusing the eye movement time series data after importance weight distribution with the eye movement trajectory spatial position encoding data after proliferation to obtain fused data; S4, training the fused data in an XGBoost model to obtain a situation awareness assessment model; S5, assessing the operating status of the flight personnel according to the situation awareness assessment model. The present invention obtains eye movement trajectory data and eye muscle electrical activity signals by high-precision data acquisition means, and deeply processes and fuses them, thereby improving the accuracy and reliability of the assessment of the operating status of the flight personnel.

Description

Technical Field

[0001] The present invention relates to the field of flight simulation monitoring and evaluation, and more particularly to a method for analyzing the operating status of flight personnel. Background Art

[0002] In the aviation field, accurate assessment of the operating status of flight crews is crucial to ensuring flight safety and improving flight efficiency. Traditionally, the methods for assessing the operating status of flight crews are relatively simple, mainly relying on self-reports from flight crews and limited external observations, which makes it difficult to fully, real-timely and accurately capture the true status of flight crews in complex flight environments. With the continuous advancement of science and technology, the use of physiological data to analyze the operating status of flight crews has become a hot research topic, among which the application of eye movement data and electromyography data shows great potential.

[0003] Eye movement data can directly reflect the distribution of visual attention of flight personnel. During the flight, pilots need to constantly pay attention to various instruments, the external environment, and other aircraft. Their eye movement trajectories and gaze point distribution contain rich information, such as the degree of attention to different flight parameters, the perception and response to dangerous situations, etc. Using time as a clue, the changes in the pilot's eye dynamics over time are collected and recorded, and through analysis, the visual behavior patterns of pilots at different stages can be revealed, and their cognitive and decision-making processes can be gained. However, relying solely on eye movement data, it is impossible to gain an in-depth understanding of the intrinsic driving mechanism of eye muscle movement and changes in physiological state.

[0004] At the same time, electromyographic data provides unique information from the perspective of muscle activity. The electrical activity signals of eye muscles can reflect the movement status of eye muscles, including the muscle activity when actively looking in different directions, tracking specific targets, and capturing flight parameters in an instant. It can provide a deeper physiological basis for analyzing the performance of pilots' visual tasks. For example, by analyzing the electrical activity signals of eye muscles, it can be determined whether the eye muscles work normally when pilots perform complex visual tasks, and whether there are signs of fatigue or tension.

[0005] At present, although some studies have focused on the application of eye movement data and electromyography data in the analysis of flight personnel status, there are relatively few studies that integrate eye movement data with electromyography data and use them for the analysis of flight personnel operation status. In addition, the few existing studies have many deficiencies in terms of data acquisition accuracy, data processing accuracy, and model construction effectiveness. For example, traditional data acquisition equipment and methods are difficult to obtain eye movement time series data with picosecond resolution, and cannot accurately capture subtle dynamic changes in the eyes; in the data processing process, the removal of complex noise is not thorough enough, resulting in low data quality, affecting the accuracy of subsequent analysis; when constructing an evaluation model, there is a lack of effective methods for deep fusion of multimodal data, and it is impossible to fully explore the potential connections between data, thereby limiting the accuracy and reliability of the evaluation of flight personnel operation status. Summary of the invention

[0006] An object of the present invention is to solve at least the above problems and to provide at least the advantages which will be described hereinafter.

[0007] Another object of the present invention is to provide a method for analyzing the operating status of flight personnel, which obtains eye movement trajectory data and eye muscle electrical activity signals through high-precision data acquisition means, and deeply processes and fuses them, thereby improving the accuracy and reliability of the evaluation of the operating status of flight personnel.

[0008] In order to achieve these purposes and other advantages according to the present invention, a method for analyzing flight personnel operation status is provided, comprising:

[0009] S1. Use an eye tracker to collect the pilot's eye movement trajectory data, and slice the normalized eye movement trajectory according to a preset picosecond fixed step size to generate eye movement time series data with a resolution of 1 picosecond; at the same time, use an electromyography sensor array to collect eye muscle electrical activity signals and convert them into electromyography feature vectors;

[0010] S2, mapping the gaze point coordinates in the eye movement trajectory into a three-dimensional space, dividing the voxel region and encoding it, combining the voxel region encoding with the electromyographic feature vector, classifying and encoding according to the eye muscle coordinated movement group, and generating multi-modal deep fusion eye movement trajectory spatial position encoding data;

[0011] S3, fusing the eye movement time series data after importance weight allocation with the eye movement trajectory spatial position encoding data after proliferation to obtain fused data;

[0012] S4, training the fused data in the XGBoost model to obtain a situation awareness assessment model;

[0013] S5. Analyzing the operating status of the flight crew according to the situation awareness assessment model;

[0014] Among them, the eye movement time series data is based on a position-aware multi-head self-attention network, which automatically learns the importance weights of eye movement data in different time periods to situation awareness, and highlights key time segment information according to the importance weights to obtain the eye movement time series data after importance weight allocation; the eye movement trajectory spatial position encoding data is enhanced by feature expression through an adversarial generative network to obtain the proliferated eye movement trajectory spatial position encoding data.

[0015] Preferably, the eye movement trajectory data in S1 is the eye dynamic information of the pilot throughout the spatial orientation training, including the gaze point coordinates, eye saccade start and end positions and paths, and pupil scaling changes under different visual stimuli recorded by the nanosecond timestamp of the eye tracker;

[0016] The eye muscle electrical activity signals include eye muscle electrical activity signals when actively looking at different directions, eye muscle electrical activity signals when performing a specific target gaze tracking visual task, and eye muscle electrical activity signals when performing a flight parameter visual task.

[0017] Preferably, the eye movement trajectory data in S1 needs to be denoised before normalization, specifically including:

[0018] For the eye movement trajectory data, a Kalman filter fault-tolerant algorithm based on multimodal data fusion is used to eliminate abnormal data fragments caused by occasional jitter of the equipment, instantaneous head displacement of the pilot and strong light interference, and a Mallat algorithm is used to remove high-frequency noise.

[0019] Preferably, the Kalman filter fault-tolerant algorithm based on multimodal data fusion removes abnormal data segments caused by occasional equipment jitter, instantaneous head displacement of the pilot and strong light interference in the following specific steps:

[0020] A1. Use vibration sensors to collect equipment vibration data, motion capture sensors to collect pilot motion data, and light sensors to collect ambient light data;

[0021] A2. Based on the Kalman filter fault-tolerant algorithm, the data from the vibration sensor, motion capture sensor and light sensor are integrated to process the eye movement trajectory data, and abnormal data fragments caused by occasional device jitter, instantaneous head displacement of the pilot and strong light interference are eliminated.

[0022] Preferably, the use of the Mallat algorithm to remove high frequency noise comprises the following steps:

[0023] B1, dividing the eye movement trajectory data into a plurality of eye movement trajectory segments according to a preset time interval;

[0024] B2. Processing any eye movement trajectory segment using the Mallat algorithm with preset adaptation parameters to decompose the eye movement trajectory segment into a high-frequency sub-band and a low-frequency sub-band;

[0025] B3. Identify the portion of the high frequency sub-band whose amplitude exceeds a preset high frequency noise amplitude threshold as high frequency noise, and remove it.

[0026] Preferably, the collection time of the eye movement trajectory data in B1 is 10-60s, and the preset time interval is 0.1-0.5s; the high-frequency noise amplitude threshold in B3 is 0.05-0.2 volts.

[0027] Preferably, the eye muscle electrical activity signal in S1 is converted into an electromyographic feature vector by the following steps:

[0028] C1. Use blind source separation algorithm to remove environmental electrical noise and common mode interference;

[0029] C2. The eye muscle electrical activity signal is extracted through the Hilbert-Huang transform to extract the characteristic frequency band of muscle contraction and relaxation, and converted into an electromyographic feature vector that can be directly used by the XGBoost model.

[0030] Preferably, the specific steps of training the fused data in the XGBoost model in S4 to obtain the situation awareness assessment model are:

[0031] D1, input the fused data obtained in S3 into the recursive neural network variant structure, use the memory unit of the recursive neural network variant structure to capture the long time series dependency, and output a dynamically updated feature representation with time series;

[0032] D2, then input the feature representation obtained in D1 into the multi-layer convolutional neural network connected by the dynamic routing algorithm, automatically adjust the network depth and convolution kernel parameters according to the feedback during the training process of the multi-layer convolutional neural network, optimize the extraction of spatial features, and finally connect the processed eye feature data to the XGBoost model;

[0033] D3. A feature filter driven by a quantum evolutionary algorithm using the superposition state characteristics of quantum bits is used to filter the eye feature data obtained in D2. Based on the contribution ranking of the eye feature data in improving the prediction accuracy of the model and a preset dynamic adaptive threshold, the current optimal feature subset is selected as a training set, and training is performed in an XGBoost model to obtain a situational awareness assessment model.

[0034] Preferably, the process of training the optimal feature subset as a training set in the XGBoost model also includes:

[0035] d1, dividing a validation set from the fused data obtained in S3;

[0036] d2. The XGBoost model is trained based on the data in the fusion data except the validation set, and uses the triple early stopping strategy and the adaptive learning rate power decay rule to make the situation awareness assessment model converge to the global optimal or approximate optimal solution;

[0037] Among them, in the triple early stopping strategy, whether to stop training early is determined based on the model loss function, the accuracy of the validation set, and the stability of the optimal feature subset during the XGBoost model training process; the model loss function is calculated based on the data during the model training process, the validation set accuracy is calculated based on the validation set data, and the feature subset stability is determined based on the feature subset data involved in the model training process.

[0038] Preferably, the method of enhancing feature expression by using a generative adversarial network to obtain the proliferated spatial position coding data comprises the following steps:

[0039] E1, input the eye movement trajectory spatial position coding obtained in S2 into the constructed adversarial generative network, use the generator in the adversarial generative network to extract and enhance the eye movement trajectory spatial position coding to generate intermediate feature data;

[0040] E2. According to a preset spatial position coding rule, the intermediate feature data is coded and converted to obtain intermediate spatial position coding data;

[0041] E3. Feed the intermediate spatial position coding data obtained in E2 back to the discriminator in the adversarial generative network. The discriminator judges its validity based on the preset judgment criteria. If it does not meet the criteria, it enters the generator to continue the optimization process until the amplified spatial position coding data that meets the criteria is obtained.

[0042] The present invention has at least the following beneficial effects:

[0043] First, the flight crew operation status analysis method provided by the present invention uses an eye tracking instrument to collect the pilot's eye movement trajectory data and an electromyography sensor array to collect the eye muscle electrical activity signal, and combines the pilot's eye movement trajectory data with the pilot's eye muscle electrical activity signal, which not only covers the dynamic behavior of the flight crew's eyes, but also goes deep into the driving mechanism level of muscle physiology. It not only accurately presents the pilot's visual focus and attention allocation at different times, but also refines the position information of the eye movement trajectory in three-dimensional space. The combination of the two greatly improves the comprehensiveness and accuracy of the flight crew operation status evaluation;

[0044] Secondly, the multi-head self-attention network based on position perception in the flight personnel operation status analysis method provided by the present invention can automatically learn the importance weights of eye movement data in different time periods to situation awareness, accurately identify key time segments so that subsequent analysis based on these data can more accurately focus on information that is critical to flight safety and operation status assessment, and avoid being disturbed by a large amount of irrelevant or secondary eye movement data; at the same time, through the adversarial training of the generator and the discriminator in the adversarial generative network, the spatial position encoding data of the eye movement trajectory is feature extracted and enhanced, and more potential spatial features are excavated. In the XGBoost model, richer and higher-quality spatial position encoding data can allow the model to better learn the relationship between the eye movement trajectory and the flight situation, thereby improving the generalization ability and accuracy of the model;

[0045] Third, the flight crew operation status analysis method provided by the present invention generates high-resolution eye movement time series data through picosecond fixed-step slicing, combines the Kalman filter fault-tolerant algorithm of multimodal data fusion and the Mallat algorithm for noise reduction, removes abnormal data and high-frequency noise such as equipment jitter, head displacement, strong light interference, etc., provides a high-quality data basis for subsequent analysis, and improves the accuracy of analysis;

[0046] Fourthly, the flight personnel operation status analysis method provided by the present invention maps the eye movement trajectory coordinates to three-dimensional space coding, classifies and codes them with the electromyographic feature vector according to the eye muscle coordinated movement group, and then fuses them with the eye movement time series data assigned with importance weights, comprehensively integrates visual behavior and muscle physiological information, deeply mines the multi-dimensional characteristics of the pilot's operation status, and enhances the comprehensiveness of the evaluation;

[0047] Fifth, the flight crew operating status analysis method provided by the present invention can evaluate the flight crew operating status in real time and accurately, which helps to promptly detect problems such as pilot fatigue and lack of concentration, take intervention measures in advance, provide strong protection for flight safety, and reduce the risk of flight accidents.

[0048] Other advantages, objectives and features of the present invention will be embodied in part through the following description, and in part will be understood by those skilled in the art through study and practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 The figure is a flow chart of the flight crew operation status analysis method of the present invention. DETAILED DESCRIPTION

[0050] The present invention is further described in detail below in conjunction with the accompanying drawings so that those skilled in the art can implement the invention with reference to the description.

[0051] It should be understood that the terms such as “having”, “including” and “comprising” used herein do not exclude the existence or addition of one or more other elements or combinations thereof.

[0052] like Figure 1 As shown, the present invention provides a method for analyzing the operating status of a flight crew, comprising:

[0053] S1. Use an eye tracker to collect the pilot's eye movement trajectory data, and slice the normalized eye movement trajectory according to a preset picosecond fixed step size to generate eye movement time series data with a resolution of 1 picosecond; at the same time, use an electromyography sensor array to collect eye muscle electrical activity signals and convert them into electromyography feature vectors;

[0054] S2, mapping the gaze point coordinates in the eye movement trajectory into a three-dimensional space, dividing the voxel region and encoding it, combining the voxel region encoding with the electromyographic feature vector, classifying and encoding according to the eye muscle coordinated movement group, and generating multi-modal deep fusion eye movement trajectory spatial position encoding data;

[0055] S3, fusing the eye movement time series data after importance weight allocation with the eye movement trajectory spatial position encoding data after proliferation to obtain fused data;

[0056] S4, training the fused data in the XGBoost model to obtain a situation awareness assessment model;

[0057] S5. Analyzing the operating status of the flight crew according to the situation awareness assessment model;

[0058] Among them, the eye movement time series data is based on a position-aware multi-head self-attention network, which automatically learns the importance weights of eye movement data in different time periods to situation awareness, and highlights key time segment information according to the importance weights to obtain the eye movement time series data after importance weight allocation; the eye movement trajectory spatial position encoding data is enhanced by feature expression through an adversarial generative network to obtain the proliferated eye movement trajectory spatial position encoding data.

[0059] In the above technical solution, the eye movement trajectory data is collected using a high-precision eye tracker with a resolution of 1920×1080 pixels or above and a sampling frequency of 2000Hz or above. The eye tracker collects data from the pilot in a simulated flight training environment. The eye tracker has a nanosecond timestamp recording function and can accurately record the pilot's eye dynamic information throughout the spatial orientation training, including the coordinates of the gaze point, the start and end positions and paths of eye saccades, and the scaling changes of the pupil under different visual stimuli. After collecting the original eye movement trajectory data, the Kalman filter fault-tolerant algorithm based on multimodal data fusion is used to fuse the three sensor data to process the eye movement trajectory data, and eliminate abnormal data fragments caused by occasional equipment jitter, instantaneous head displacement of the pilot, and strong light interference. The Mallat algorithm is used to further remove high-frequency noise. The eye movement trajectory data is divided into multiple eye movement trajectory segments according to a preset time interval (such as 0.2s). Each eye movement trajectory segment is processed using the Mallat algorithm with preset adaptation parameters, decomposing it into high-frequency sub-bands and low-frequency sub-bands. The part of the high-frequency sub-band whose amplitude exceeds the preset high-frequency noise amplitude threshold (such as 0.1 volt) is identified as high-frequency noise and removed. After noise reduction, the eye movement trajectory data is normalized and mapped to a unified numerical range for subsequent analysis. The normalized eye movement trajectory is then sliced ​​according to the preset picosecond fixed step size to generate eye movement time series data with a resolution of 1 picosecond.

[0060] In the above technical scheme, an electromyographic sensor array is arranged at an appropriate position on the pilot's face to collect eye muscle electrical activity signals when the pilot actively looks in different directions, performs specific target tracking visual tasks, and instantly captures flight parameter visual tasks. A blind source separation algorithm is used to remove environmental electrical noise and common-mode interference, and then the characteristic frequency bands of muscle contraction and relaxation are extracted through the Hilbert-Huang transform, and the eye muscle electrical activity signals are converted into electromyographic feature vectors that can be directly used by subsequent models.

[0061] In the above technical scheme, eye movement trajectory spatial position coding data is further generated according to the eye movement trajectory data and electromyographic feature vector, the processed gaze point coordinates in the eye movement trajectory are mapped into three-dimensional space, and voxel areas are divided according to the flight scene and eye movement characteristics, for example, the voxel areas are divided according to the instrument area in the cockpit, the visual field area outside the window, etc. The divided voxel areas are encoded, the voxel area encoding is combined with the electromyographic feature vector obtained previously, and the classification encoding is performed according to the eye muscle coordinated movement group, and the multi-modal deep fusion eye movement trajectory spatial position coding data is generated. The eye movement trajectory spatial position coding data is input into the constructed adversarial generation network, and the generator in the adversarial generation network is used to extract and enhance the features, generate intermediate feature data, and according to the preset spatial position coding rules, the intermediate feature data is encoded and converted to obtain the intermediate spatial position coding data. The intermediate spatial position coding data is fed back to the discriminator in the adversarial generation network, and the discriminator judges its validity according to the preset discrimination criteria. If it does not meet the criteria, the generator continues to optimize the processing until the hyperplastic eye movement trajectory spatial position coding data that meets the criteria is obtained.

[0062] In the above technical solution, the eye movement time series data is further weighted based on the multi-head self-attention network based on position perception, and the generated eye movement time series data is input into the multi-head self-attention network based on position perception, and the network automatically learns the importance weights of eye movement data in different time periods for situation awareness. For example, in key stages such as takeoff and landing, the network learns that the eye movement data during this period is more important for situation awareness and will assign a higher weight. According to the learned importance weights, the eye movement time series data is weighted to highlight the key time segment information, and the eye movement time series data after importance weight allocation is obtained. Finally, the eye movement time series data after importance weight allocation is fused with the eye movement trajectory spatial position encoding data after proliferation to obtain fused data.

[0063] In the above technical solution, the model is further trained and evaluated: the fused data is input into the recursive neural network variant structure, and its memory unit is used to capture the long-term series dependency, and the dynamically updated feature representation with time series is output, and the above feature representation is input into the multi-layer convolutional neural network connected by the dynamic routing algorithm. According to the feedback in the training process of the multi-layer convolutional neural network, the network depth and convolution kernel parameters are automatically adjusted to optimize the extraction of spatial features. Finally, the processed eye feature data is connected to the XGBoost model, and the feature filter driven by the quantum evolutionary algorithm with the superposition state characteristics of quantum bits is used to filter the eye feature data. According to the contribution ranking of the eye feature data in improving the prediction accuracy of the model and the preset dynamic adaptive threshold, the current optimal feature subset is selected as the training set. A validation set is divided from the fused data, and the XGBoost model is trained based on the data in the fused data except the validation set. The triple early stopping strategy and the adaptive learning rate power decay rule are used to promote the situation awareness assessment model to converge to the global optimal or approximate optimal solution. In the triple early stopping strategy, whether to stop training in advance is determined according to the model loss function, the accuracy of the validation set, and the stability of the optimal feature subset during the training process of the XGBoost model. After the situational awareness assessment model is obtained through training, the pilot's eye movement trajectory data and eye muscle electrical activity signals are collected in real time during actual flight training or flight missions, and the corresponding fusion data is obtained according to the same data processing and feature generation steps as mentioned above. The fusion data is input into the situational awareness assessment model, and the situational awareness assessment model outputs the assessment results of the flight personnel's operating status, such as the pilot's concentration, fatigue status, and grasp of the flight situation. According to the assessment results, relevant personnel can take timely measures, such as reminding pilots to rest, adjusting flight mission arrangements, etc., to ensure flight safety.

[0064] A specific example of collecting and processing eye movement trajectory data in step S1 in the above scheme is as follows:

[0065] Simulate the collection of eye tracking data from the eye tracker, use np.random.rand(1000,2) to generate a two-dimensional array of eye tracking data with a shape of (1000,2) as sample data, each row of the sample data represents the eye tracking data of a time point, including x and y coordinates, whose values ​​are randomly generated in the range of [0,1]. Further normalize the x and y coordinates of the two-dimensional array eye tracking data to the range of [0,1], taking the normalize_data function as an example, by calculating the minimum and maximum values ​​of the data, each data point is converted to the form of (data-min_val) / (max_val-min_val) to ensure the consistency and comparability of the data. Next, use the slice_eye_tracking_data function to slice the normalized eye tracking data according to a fixed step size of picoseconds (such as 10), divide the continuous eye tracking data into multiple segments, each segment contains picosecond step data points, and then calculate the average value of each segment as a new data point, finally obtaining the eye movement time series data with a resolution of 1 picosecond.

[0066] A specific example of collecting and processing eye muscle electrical activity signals in step S1 of the above scheme is as follows:

[0067] Simulate the collection of eye muscle electrical activity signals from the EMG sensor array and use np.random.rand(1000,5) to generate a two-dimensional array of eye EMG data with a shape of (1000,5) as sample data. Each row of the sample data represents the eye muscle electrical activity signal at a time point and contains 5 dimensions of features. Its values ​​are randomly generated in the range of [0,1]. The two-dimensional array of eye EMG data is used as the EMG feature vector, which can be further processed according to actual needs.

[0068] A specific example of generating multimodal deep fusion eye movement trajectory spatial position encoding data in step S2 of the above scheme is as follows:

[0069] First, use the map_and_encode_eye_tracking function to map the normalized eye tracking data to three-dimensional space. Assuming that the z coordinate is 0, convert the (x, y) coordinate to the form of (x, y, 0) and store it in the mapped eye tracking data; then use the encode_emg_features function to classify and encode the electromyographic feature vector for eye muscle coordinated movement groups. The encoding method here is to sum the vector elements modulo 10, add each vector element in the electromyographic feature vector and modulo 10, and store the result in the encoded electromyographic feature vector data; finally, combine the mapped eye tracking data and the encoded electromyographic feature vector data to form the eye tracking spatial encoding data, and convert it into a deep learning tensor for subsequent deep learning processing.

[0070] A specific example of the spatial position encoding data of the eye movement trajectory after the proliferation obtained in step S3 of the above scheme is as follows:

[0071] Construction and training of Generative Adversarial Network (GAN): Create a generator and a discriminator. The generator has three fully connected layers, from input dimension to hidden dimension, from hidden dimension to twice hidden dimension, and from twice hidden dimension to output dimension, and uses ReLU activation function. The discriminator also has three fully connected layers to determine whether the input data is real data. Use train_gan function to train GAN. During the training process:

[0072] For each training epoch:

[0073] Training the discriminator: First, input the spatial position encoding data of the eye movement trajectory into the discriminator to obtain real_output, and calculate its binary cross entropy loss d_loss_real with the real label; then generate random noise z with the same shape as spatial_encoding_data, get fake_data through the generator, input fake_data into the discriminator to get fake_output, and calculate its binary cross entropy loss d_loss_fake with the fake label fake_label; finally, set the total discriminator loss d_loss to the sum of d_loss_real and d_loss_fake, and update the discriminator parameters through the optimizer.

[0074] Train the generator: input fake_data into the discriminator again to get fake_output, calculate the binary cross entropy loss g_loss between it and the true label, and update the generator parameters through the optimizer.

[0075] After training, the generator is used to process the eye movement trajectory spatial position encoding data to obtain the augmented eye movement trajectory spatial position encoding data.

[0076] In the above solution, a specific example of obtaining the weighted eye movement time series data in step S3 is as follows:

[0077] First, an attention layer with a 2-head multi-head attention mechanism is created to assign importance weights to the eye movement time series data. Then, the eye movement time series data is converted into a deep learning tensor and a dimension is added as a batch dimension, which is input into the attention layer to obtain the weighted eye movement time series data.

[0078] A specific example of fusing data in step S3 of the above scheme is as follows: the weighted eye movement time series data is flattened and concatenated with the flattened result of the proliferated eye movement trajectory spatial position encoding data to obtain fused data, and a batch dimension is added.

[0079] In the above scheme, a specific example of the situation awareness assessment model obtained in step S4 is as follows:

[0080] Generate random binary classification labels for the fused data, and use np.random.randint(0,2,fusion_data.size(1)) to generate 0 or 1 labels with the same number of samples as the fused data; create an xgb.XGBClassifier model, convert the fused data into a numpy array and input it into the model for training together with the labels to obtain a situation awareness assessment model.

[0081] In the above technical solution, the flight crew operation status analysis method has at least the following technical effects:

[0082] 1. Generate high-resolution eye movement time series data through picosecond fixed-step slicing, which can capture extremely subtle changes in eye movement, providing a high-precision data basis for analyzing the visual behavior of pilots. At the same time, the Kalman filter fault-tolerant algorithm and Mallat algorithm of multimodal data fusion are used for noise reduction processing, which effectively removes abnormal data and high-frequency noise caused by various interference factors, greatly improves the quality of eye movement trajectory data, and ensures the accuracy of subsequent analysis;

[0083] 2. In terms of eye muscle electrical activity signal processing, the application of blind source separation algorithm and Hilbert-Huang transform accurately extracts the characteristic frequency bands of muscle contraction and relaxation. The converted electromyographic feature vector can accurately reflect the movement state of eye muscles, providing reliable physiological data support for multimodal data fusion;

[0084] 3. Map the eye movement trajectory coordinates to three-dimensional space and combine them with electromyographic feature vectors for multi-modal deep fusion. The generated eye movement trajectory spatial position encoding data fully integrates the pilot's visual behavior and eye muscle physiological information. This fusion method can characterize the pilot's operating status from multiple dimensions. For example, in complex flight operations, it can not only understand the position of the pilot's gaze point, but also know the coordinated movement of his eye muscles through electromyographic feature vectors, so as to more accurately judge the pilot's operating intention and perception of the flight situation;

[0085] 4. The fusion of the eye movement time series data after importance weight distribution and the eye movement trajectory spatial position encoding data after proliferation further strengthens the complementarity of multimodal data. The time series data highlights the eye movement information of key time segments, and the spatial position encoding data refines the spatial characteristics of the eye movement trajectory. The fusion of the two provides rich and comprehensive information for subsequent model training, significantly improving the accuracy and reliability of the flight crew's operational status assessment;

[0086] 5. Based on the precise situational awareness assessment model, it can evaluate the operating status of flight personnel in real time and accurately. During the flight, it can timely discover the pilot's fatigue, lack of concentration, misjudgment of flight situation and other problems, and take corresponding intervention measures in advance, such as arranging pilots to rest, adjusting flight missions, etc., effectively reducing the risk of flight accidents, providing a strong guarantee for flight safety, and having important significance for safe operations in the aviation field.

[0087] In another embodiment of the present invention, the eye movement trajectory data in S1 is the eye dynamic information of the pilot throughout the spatial orientation training, including the gaze point coordinates, eye saccade start and end positions and paths, and pupil scaling changes under different visual stimuli recorded by the nanosecond timestamp of the eye tracker;

[0088] The eye muscle electrical activity signals include eye muscle electrical activity signals when actively looking at different directions, eye muscle electrical activity signals when performing a specific target gaze tracking visual task, and eye muscle electrical activity signals when performing a flight parameter visual task.

[0089] In the above technical solution, in the spatial orientation training scenario of simulated flight training, a high-precision eye tracker with nanosecond timestamp recording function is selected. Before the training begins, the technicians accurately calibrate and fix the eye tracker on the pilot's helmet or a specific position in the cockpit to ensure that it can collect data stably and accurately. During the entire spatial orientation training process, the eye tracker records the pilot's eye dynamic information in real time. When the pilot observes various instruments in the cockpit, simulates landmarks outside the window, and responds to different flight instructions, the eye tracker accurately records the coordinates of each gaze point, which enables us to understand the distribution of the pilot's visual focus points in the cockpit. For example, during the simulated takeoff phase, the pilot needs to frequently pay attention to instruments such as the airspeed indicator, altimeter, and attitude indicator. The eye tracker records the coordinates of the pilot's gaze points on these instruments, showing his attention to each instrument at different times. At the same time, for the rapid eye movement of saccades, the eye tracker records the starting position, end position, and movement path of the saccade. For example, when the pilot quickly shifts his gaze from observing one parameter on the dashboard to another, the relevant information of the saccade is fully recorded. In addition, under different visual stimuli, such as simulated strong light exposure or instrument displays of different brightness, the eye tracker captures the changes in pupil expansion. In the simulated night flight scene, when a strong light is suddenly turned on to simulate the external light, the eye tracker records the contraction of the pupil. These data reflect the physiological response of the pilot's eyes to light changes and indirectly reflect their visual processing mechanism.

[0090] In the above technical solution, multiple small, highly sensitive electromyographic sensors are respectively attached to key muscle parts around the pilot's eyes, such as extraocular muscles. During the training process, when the pilot actively looks at different directions, such as looking left, right, up, and down to observe targets in various directions in the simulated flight environment, the electromyographic sensors collect the corresponding eye muscle electrical activity signals. When performing a specific target gaze tracking visual task, such as tracking a moving target (such as another virtual aircraft) in a simulated flight, the sensor records the changes in electrical activity of the pilot's eye muscles to keep tracking the target. When the pilot instantly captures the flight parameter visual task, such as quickly checking the parameters of a key instrument in an emergency, the electromyographic sensor also records the electrical activity signals of the eye muscles during this process. These signals, in the form of electrophysiological data, intuitively reflect the working status and force of the eye muscles under different visual tasks.

[0091] In the above technical solution, the eye movement trajectory data is further optimized, and the eye movement trajectory data covers the coordinates of the gaze point, the start and end positions and paths of eye saccades, and the changes in pupil zoom, providing a full range of perspectives for in-depth analysis of the pilot's visual behavior. The coordinates of the gaze point can accurately locate the targets that the pilot is paying attention to at different times, the eye saccade path reflects the dynamic process of its attention transfer, and the changes in pupil zoom reveal the pilot's physiological response to different visual stimuli, which helps to deeply explore the pilot's visual behavior patterns in spatial orientation training and accurately grasp its attention allocation.

[0092] In the above technical solution, the electrical activity signals of eye muscles under different visual tasks are further clarified, such as signal collection during active gaze, target tracking, and parameter capture, so that physiological data is closely linked to visual behavior in actual flight operations. Through this connection, the intrinsic driving mechanism of pilot visual behavior can be explained from a physiological level. For example, when performing a specific target gaze tracking task, the electrical activity of the eye muscles reflects its ability to lock and track the target, thereby providing a physiological basis for more accurate evaluation of the pilot's operating status.

[0093] In the above technical solution, comprehensive and specific data collection greatly enriches the data dimensions used for analysis. After the fusion of multi-dimensional data, it can more comprehensively reflect the status of pilots in spatial orientation training and reduce the one-sidedness of the evaluation. For example, by combining eye movement trajectory data and eye muscle electrical activity signals, it is possible to comprehensively judge whether the pilot's visual operation and physiological response are coordinated when facing complex flight situations, thereby significantly improving the accuracy of the evaluation of the flight personnel's operating status.

[0094] Based on such detailed data collection and analysis, targeted training can be conducted based on the pilots' performance in different visual tasks. For example, if it is found that a pilot's eye muscle electrical activity is abnormal or the eye saccade path is inefficient when capturing flight parameters in an instant, a special training program can be designed for intensive training. At the same time, for pilots in abnormal conditions, more precise intervention can be carried out based on these data. For example, if it is determined that the pilot's eye muscles are fatigued due to long-term high concentration, rest and adjustment can be arranged in time, thereby effectively improving the effect and safety of flight training.

[0095] In another embodiment of the present invention, the eye movement trajectory data in S1 needs to be denoised before normalization, specifically including:

[0096] For the eye movement trajectory data, a Kalman filter fault-tolerant algorithm based on multimodal data fusion is used to eliminate abnormal data fragments caused by occasional jitter of the equipment, instantaneous head displacement of the pilot and strong light interference, and a Mallat algorithm is used to remove high-frequency noise.

[0097] In the above technical solution, a high-precision vibration sensor is installed on the flight simulation device, which can monitor the vibration of the device in real time during the training process. For example, when the pilot operates the joystick or the flight simulation device vibrates due to the simulated flight scene, the vibration sensor can accurately record the frequency, amplitude and other parameters of the vibration. The motion capture sensor is used to collect the pilot's head movement data in real time. Multiple motion capture sensors can be arranged around the pilot's head to ensure that the displacement, rotation and other movement information of the head in three-dimensional space can be fully captured. During the pilot's spatial orientation training, whether it is normal head rotation observation or instantaneous head displacement caused by sudden changes in flight conditions, the motion capture sensor can accurately record it. A light sensor is installed at a suitable position in the cockpit to monitor the intensity, change frequency and other information of the ambient light. During the simulated flight, when simulating scenes such as sunrise, sunset, strong light exposure or sudden changes in light, the light sensor can capture the changes in light in time. The data collected by the vibration sensor, motion capture sensor and light sensor are fused with the eye movement trajectory data. By establishing a suitable data fusion model, the data from different sensors are aligned and integrated in time and space so that they can complement each other and be used together for subsequent noise reduction processing.

[0098] In the above technical solution, the Kalman filter fault-tolerant algorithm is used to process the fused data. First, according to the characteristics of the sensor data and the dynamic model of the eye movement trajectory data, the parameters of the Kalman filter, including the state estimate, covariance matrix, etc., are initialized. Then, at each moment, according to the state estimate of the previous moment and the sensor measurement value of the current moment, the optimal state estimate of the current moment is calculated through prediction and update steps. In this process, the algorithm can identify and eliminate abnormal data points caused by device jitter, head displacement and strong light interference, and output relatively pure eye movement trajectory data.

[0099] In the above technical solution, the eye movement trajectory data processed by the Kalman filter fault-tolerant algorithm is divided according to a preset time interval. For example, the preset time interval is set to 0.2 seconds, and the continuous eye movement trajectory data is divided into multiple segments, each of which contains the eye movement data within the time period. For each eye movement trajectory segment, the Mallat algorithm with preset adaptation parameters is used for processing. The Mallat algorithm is based on wavelet transform and decomposes the eye movement trajectory segment into high-frequency sub-bands and low-frequency sub-bands. The high-frequency sub-band mainly contains noise and detail information, while the low-frequency sub-band contains the main trends and characteristics of the data. By setting a suitable threshold, the part of the high-frequency sub-band whose amplitude exceeds the preset high-frequency noise amplitude threshold (such as 0.05-0.2 volts) is identified as high-frequency noise and removed. Then, the processed high-frequency sub-band and low-frequency sub-band are reconstructed by inverse wavelet transform to obtain the eye movement trajectory data after removing the high-frequency noise.

[0100] In the above technical solution, the eye movement trajectory data after the Mallat algorithm denoising is normalized. The data is mapped to a unified numerical range, such as the [0,1] interval, so that it can be subsequently fused with other data and used for model training. The normalized data can be further sliced ​​at a fixed step size of picoseconds to generate eye movement time series data, preparing for subsequent multimodal data fusion and model training.

[0101] In the above technical solution, the Kalman filter fault-tolerant algorithm and Mallat algorithm based on multimodal data fusion can effectively remove various types of noise in the eye movement trajectory data. The Kalman filter fault-tolerant algorithm accurately identifies and removes abnormal data fragments caused by occasional jitter of the equipment, instantaneous head displacement of the pilot and strong light interference by fusing the data of the vibration sensor, motion capture sensor and light sensor, ensuring that the data truly reflects the pilot's eye movement. The Mallat algorithm further removes high-frequency noise to make the data smoother and more accurate, providing a high-quality data foundation for subsequent analysis and model training. The data after denoising reduces the interference of outliers and noise, so that the model can more accurately learn the patterns and laws in the data during the training process. When the eye movement trajectory data is used to train the recursive neural network variant structure, the multi-layer convolutional neural network and the XGBoost model, the stable and accurate data helps the model converge to the global optimal or approximate optimal solution faster, improves the training efficiency and stability of the model, and thus improves the accuracy and reliability of the model's assessment of the flight crew's operating status.

[0102] In the above technical solution, noise data may lead to misjudgment and missed judgment when analyzing the pilot's eye movement trajectory. For example, abnormal data generated by equipment jitter may be misjudged as the pilot's real eye movement, resulting in an erroneous analysis conclusion. After noise reduction processing by the method provided by the present invention, these interference factors are effectively eliminated, which greatly reduces the risk of misjudgment and missed judgment, and ensures that the evaluation results of the pilot's operating status are true and reliable. The data after noise removal can more clearly show the pilot's visual behavior pattern, such as the exact position of the gaze point, the real path of the eye saccade, etc. Based on these accurate data, multimodal data fusion and analysis can more reliably judge the pilot's attention allocation, perception of the flight situation, etc., thereby improving the reliability of the evaluation of the flight personnel's operating status and providing a stronger guarantee for flight safety.

[0103] In another embodiment of the present invention, the Kalman filter fault-tolerant algorithm based on multimodal data fusion removes abnormal data fragments caused by occasional jitter of the equipment, instantaneous head displacement of the pilot, and strong light interference in the following specific steps:

[0104] A1. Use vibration sensors to collect equipment vibration data, motion capture sensors to collect pilot motion data, and light sensors to collect ambient light data;

[0105] A2. Based on the Kalman filter fault-tolerant algorithm, the data from the vibration sensor, motion capture sensor and light sensor are integrated to process the eye movement trajectory data, and abnormal data fragments caused by occasional device jitter, instantaneous head displacement of the pilot and strong light interference are eliminated.

[0106] In the above technical solution, the use of vibration sensors, motion capture sensors and light sensors to collect multimodal data is elaborated in detail, providing a rich and comprehensive source of information for the Kalman filter fault-tolerant algorithm. Through these sensors, the source of abnormal data can be accurately located. For example, the vibration sensor can detect occasional jitters of the equipment, the motion capture sensor can capture the instantaneous head displacement of the pilot, and the light sensor can sense strong light interference. This makes it possible to accurately determine which data is due to external interference rather than the pilot's real eye movement when processing eye movement trajectory data, thus laying the foundation for the subsequent accurate elimination of abnormal data.

[0107] In the above technical solution, the eye movement trajectory data is processed through the Kalman filter fault-tolerant algorithm based on multimodal data fusion, effectively eliminating abnormal data fragments caused by equipment jitter, head displacement and strong light interference. This ensures the integrity of the eye movement trajectory data, so that it more realistically reflects the actual eye movement of the pilot. Complete and reliable data is crucial for subsequent analysis and model training. It can avoid analysis deviations and model training errors caused by the presence of abnormal data, and improve the accuracy of the assessment of the flight crew's operating status.

[0108] In another embodiment of the present invention, removing high frequency noise using the Mallat algorithm comprises the following steps:

[0109] B1, dividing the eye movement trajectory data into a plurality of eye movement trajectory segments according to a preset time interval;

[0110] B2. Processing any eye movement trajectory segment using the Mallat algorithm with preset adaptation parameters to decompose the eye movement trajectory segment into a high-frequency sub-band and a low-frequency sub-band;

[0111] B3. Identify the portion of the high frequency sub-band whose amplitude exceeds a preset high frequency noise amplitude threshold as high frequency noise, and remove it.

[0112] In the above technical solution, the eye movement trajectory data is processed by the Mallat algorithm, which can effectively remove high-frequency noise. In the process of collecting eye movement trajectory data, various high-frequency interference signals will inevitably be mixed in. These noises may come from electromagnetic interference of electronic equipment, high-frequency vibrations in the environment, etc. The Mallat algorithm divides the eye movement trajectory data into multiple segments according to preset time intervals, and performs fine processing on each segment, decomposing it into high-frequency sub-bands and low-frequency sub-bands, accurately identifying and removing the high-frequency noise part in the high-frequency sub-band whose amplitude exceeds the preset threshold. This makes the processed eye movement trajectory data purer, truly reflects the pilot's eye movement status, and provides a high-quality data foundation for subsequent data analysis and model training.

[0113] In the above technical solution, after the noise reduction processing of the Mallat algorithm, the useful features in the eye movement trajectory data are highlighted. After removing the high-frequency noise, the main trends and key features of the data are clearer, such as the movement trajectory of the gaze point, the key nodes of the eye saccade and other features can be extracted more accurately. When the eye movement trajectory data is subsequently fused with other data (such as electromyography feature vectors) and input into models such as recursive neural network variant structures and multi-layer convolutional neural networks for feature extraction, high-quality eye movement trajectory data helps the model to more effectively capture the spatial and temporal features in the data, improve the accuracy and completeness of feature extraction, and thus provide strong support for more accurate assessment of the flight crew's operating status. This helps to more accurately judge the pilot's concentration level, perception of the flight situation, etc., thereby improving the reliability of the assessment of the flight crew's operating status and providing a more reliable basis for ensuring flight safety.

[0114] In another embodiment of the present invention, the collection time of the eye movement trajectory data in B1 is 10-60s, and the preset time interval is 0.1-0.5s; the high-frequency noise amplitude threshold in B3 is 0.05-0.2 volts.

[0115] In the above technical solution, it is clearly stipulated that the collection time of eye movement trajectory data is 10-60s, and the preset time interval is 0.1-0.5s. Such a setting effectively controls the size of the data while ensuring data integrity. The appropriate collection time ensures that the eye movement trajectory data of the pilot in the representative flight phase can be obtained, avoiding data redundancy and excessive processing burden caused by too long collection time, and preventing the collection time from being too short to cover the key flight scenes. The reasonable preset time interval makes it possible to divide and process the eye movement trajectory data using the Mallat algorithm, without generating too many unnecessary fragments due to too short intervals, which increases the processing difficulty, and without missing important eye movement change information due to too long intervals. Through this precise data parameter setting, the efficiency of data processing is greatly improved, the speed of conversion from raw data to usable data is accelerated, and the time cost for subsequent model training and analysis is saved. In aviation flight scenarios, the eye movement patterns of pilots in different flight phases are quite different, and the collection time, preset time interval and high-frequency noise amplitude threshold of eye movement trajectory data can be adjusted appropriately.

[0116] In another embodiment of the present invention, the eye muscle electrical activity signal in S1 is converted into an electromyographic feature vector by the following steps:

[0117] C1. Use blind source separation algorithm to remove environmental electrical noise and common mode interference;

[0118] C2. The eye muscle electrical activity signal is extracted through the Hilbert-Huang transform to extract the characteristic frequency band of muscle contraction and relaxation, and converted into an electromyographic feature vector that can be directly used by the XGBoost model.

[0119] In the above technical scheme, a set of highly simulated flight simulation equipment is arranged in a professional flight simulation training center, including a simulated cockpit, instrument panel, joystick and projection equipment simulating dynamic scenes outside the window. Ensure that the scene can provide pilots with an experience close to real flight, so as to obtain representative eye muscle electrical activity signals in actual flight missions. A set of highly sensitive electromyographic sensor arrays is selected, and the sensors are calibrated and tested before the training begins to ensure their stable performance. A plurality of small, adhesive electromyographic sensors are carefully pasted on key muscle groups around the pilot's eyes, such as the extraocular rectus muscle, the internal rectus muscle, the superior oblique muscle and the inferior oblique muscle, according to a specific layout. The sensor is connected to a signal collector via a wire, and the signal collector can collect and digitize the eye muscle electrical activity signals in real time. The pilot enters the simulated cockpit and begins a series of flight mission training, including takeoff, cruising, landing and responding to various simulated emergencies. During the entire training process, the electromyographic sensor continuously collects electrical activity signals of the eye muscles under different visual tasks. For example, when the pilot frequently pays attention to multiple parameters on the instrument panel during takeoff, such as the airspeed indicator, altimeter, and attitude indicator, the sensor collects the electrical activity signals of the eye muscles when actively looking in different directions and quickly switching gaze points. When performing a specific target tracking visual task, such as simulating tracking another virtual aircraft, the sensor records the electrical activity signals generated by the eye muscles to maintain stable tracking of the target. When capturing flight parameter visual tasks in an instant, such as quickly checking the engine parameter instrument, the sensor also synchronously collects the corresponding eye muscle electrical activity signals.

[0120] In the above technical solution, the collected original eye muscle electrical activity signal is first input into a computer equipped with a blind source separation algorithm program. The algorithm separates environmental electrical noise and common mode interference from the original signal by analyzing and calculating the mixed model of the signal. For example, in a simulated flight training environment, the electromagnetic interference generated by the surrounding electronic equipment is mixed into the eye muscle electrical activity signal. The blind source separation algorithm can identify and remove these interference signals and output a relatively pure eye muscle electrical activity signal. The signal processed by the blind source separation algorithm is then further analyzed using the Hilbert-Huang transform. The transform first performs empirical mode decomposition on the signal and decomposes it into several intrinsic mode functions (IMFs). Then, each IMF is Hilbert transformed to obtain the corresponding instantaneous frequency and amplitude information. By analyzing this information, the characteristic frequency bands related to muscle contraction and relaxation are extracted. For example, when analyzing the electrical activity signal of the eye muscle in the process of tracking the target, the Hilbert-Huang transform can accurately extract the characteristic frequency bands that reflect the continuous tension and fine-tuning of the muscle. The extracted characteristic frequency bands are sorted and quantized, and the electromyographic feature vectors are formed according to certain rules. For example, the characteristic frequency band amplitude, frequency and other information extracted under different visual tasks are arranged in a specific order to form a multi-dimensional electromyographic feature vector. This vector can fully and accurately represent the electrical activity characteristics of eye muscles in different flight tasks, providing an effective data form for subsequent data analysis and model training.

[0121] In the above technical solution, a blind source separation algorithm is used to remove environmental electrical noise and common mode interference, effectively purifying the original eye muscle electrical activity signal. In a complex flight training environment, various interference signals may mask the real muscle electrical activity information. By removing these interferences, the subsequent extracted characteristic frequency bands are more accurate and reliable, greatly improving the quality of the data. High-quality data provides a solid foundation for accurately analyzing the eye muscle movement and operation status of flight personnel. The Hilbert-Huang transform can deeply explore the key physiological characteristics in the eye muscle electrical activity signal. Compared with traditional signal processing methods, it can more accurately capture the subtle changes in the process of muscle contraction and relaxation, and extract characteristic frequency bands that reflect the working state of muscles under different visual tasks. These characteristic frequency bands are of great significance for understanding the eye muscle force, fatigue level and operation intention of flight personnel when performing tasks, and provide key physiological data support for accurately evaluating the operation status of flight personnel.

[0122] In the above technical solution, the eye muscle electrical activity signal is converted into electromyographic feature vectors, which provides a suitable input data form for subsequent machine learning models (such as recursive neural network variant structures, multi-layer convolutional neural networks, and XGBoost models, etc.). These feature vectors can effectively express the activity characteristics of the eye muscles, so that the model can better learn and understand the relationship between the flight personnel's operating status and the electrical activity of the eye muscles. During the model training process, based on high-quality electromyographic feature vectors, the model can converge to the global optimal or approximate optimal solution more quickly, improving the training efficiency and accuracy of the model, and thus improving the reliability of the flight personnel's operating status assessment. Through the effective processing and feature extraction of the eye muscle electrical activity signal, a new perspective can be provided for the flight personnel's operating status assessment from a physiological level. Combined with other multimodal data such as eye movement trajectory data, the behavior and status of the flight personnel during the flight process can be more comprehensively analyzed.

[0123] In another embodiment of the present invention, the specific steps of training the fused data in the XGBoost model in S4 to obtain the situation awareness assessment model are:

[0124] D1, input the fused data obtained in S3 into the recursive neural network variant structure, use the memory unit of the recursive neural network variant structure to capture the long time series dependency, and output a dynamically updated feature representation with time series;

[0125] D2, then input the feature representation obtained in D1 into the multi-layer convolutional neural network connected by the dynamic routing algorithm, automatically adjust the network depth and convolution kernel parameters according to the feedback during the training process of the multi-layer convolutional neural network, optimize the extraction of spatial features, and finally connect the processed eye feature data to the XGBoost model;

[0126] D3. A feature filter driven by a quantum evolutionary algorithm using the superposition state characteristics of quantum bits is used to filter the eye feature data obtained in D2. Based on the contribution ranking of the eye feature data in improving the prediction accuracy of the model and a preset dynamic adaptive threshold, the current optimal feature subset is selected as a training set, and training is performed in an XGBoost model to obtain a situational awareness assessment model.

[0127] In the above technical solution, the fused data is input into the recursive neural network variant structure, and its memory unit can capture long-term series dependencies excellently. In the flight scene, the pilot's eye movement and muscle electrical activity data are closely related in the time dimension. For example, in the whole process from takeoff to cruising and then landing, the pilot's visual attention allocation and eye muscle movement are coherent and logical. The recursive neural network variant structure can effectively mine the potential patterns in this time series, output dynamically updated feature representations with time series, provide rich and time-related feature information for subsequent analysis, and greatly enhance the model's ability to understand the changes in the flight personnel's operating status over time. The above feature representation is then input into a multi-layer convolutional neural network connected by a dynamic routing algorithm. The network can automatically adjust the network depth and convolution kernel parameters according to the feedback during the training process to achieve optimal extraction of spatial features. In the analysis of the flight personnel's operating status, the distribution of eye movement trajectories in three-dimensional space and the spatial relationship with the surrounding flight environment elements are crucial. Multi-layer convolutional neural networks can keenly capture these spatial features. For example, by analyzing the spatial distribution of the pilot's gaze points on different instruments in the cockpit and the spatial change patterns of eye movements when tracking external targets, we can more comprehensively understand the flight personnel's visual behavior and perception of flight situations, and improve the model's ability to process complex spatial information.

[0128] In the above technical solution, the feature filter driven by the quantum evolutionary algorithm using the superposition state characteristics of quantum bits is used to select the current optimal feature subset as the training set based on the contribution ranking of the eye feature data in improving the prediction accuracy of the model and the preset dynamic adaptive threshold. This process can accurately remove redundant and irrelevant features, making the training data entering the XGBoost model more refined and representative. In this way, not only the computational amount and time cost of model training are reduced, but also the overfitting problem caused by too many useless features can be avoided, the generalization ability and prediction accuracy of the model are improved, and the model can more accurately evaluate the operating status of the flight personnel. The situational awareness assessment model constructed through the above series of steps combines the advantages of multiple advanced algorithms and model structures. The recursive neural network variant structure grasps the time series, the multi-layer convolutional neural network extracts the spatial features, and the feature filter driven by the quantum evolutionary algorithm optimizes the features, which work together on the XGBoost model. The model can analyze the operating status of the flight personnel from multiple dimensions and in an all-round way, comprehensively considering factors such as time, space and key features, greatly improving the performance and reliability of the model. The model finally constructed can more accurately and stably evaluate the operating status of flight crew members, providing strong technical support for flight safety in the aviation field.

[0129] In the above technical solution, various information in the fused data is fully utilized, from time series to spatial features, and then to the screening of key features, to fully tap the potential value of the data. Through in-depth processing and analysis of multimodal fusion data such as eye movement time series data, eye movement trajectory spatial position encoding data, and electromyography feature vectors, we can deeply understand the flight crew's behavior patterns, physiological states, and perception of flight situations during the flight. This in-depth analysis not only helps to accurately evaluate the current operating status, but also provides rich data support and theoretical basis for further studying the behavior patterns of flight crews, formulating personalized training programs, and optimizing flight operation processes.

[0130] In another embodiment of the present invention, the process of training the optimal feature subset as a training set in the XGBoost model further includes:

[0131] d1, dividing a validation set from the fused data obtained in S3;

[0132] d2. The XGBoost model is trained based on the data in the fusion data except the validation set, and uses the triple early stopping strategy and the adaptive learning rate power decay rule to make the situation awareness assessment model converge to the global optimal or approximate optimal solution;

[0133] Among them, in the triple early stopping strategy, whether to stop training early is determined based on the model loss function, the accuracy of the validation set, and the stability of the optimal feature subset during the XGBoost model training process; the model loss function is calculated based on the data during the model training process, the validation set accuracy is calculated based on the validation set data, and the feature subset stability is determined based on the feature subset data involved in the model training process.

[0134] In the above technical solution, the XGBoost model is trained based on the data in the fusion data except the validation set, and an adaptive learning rate power decay rule is adopted. In the early stage of training, a larger learning rate allows the model to quickly search for a rough optimal solution range in the parameter space; as the training progresses, the adaptive learning rate power decay rule makes the learning rate gradually decrease, and the model can adjust the parameters more finely to avoid oscillations near the optimal solution. This process greatly accelerates the convergence speed of the model and reduces the time and computing resources required for training. At the same time, faster convergence to the global optimal or approximate optimal solution also improves the final performance of the model, enabling it to more accurately evaluate the operating status of the flight crew.

[0135] In the above technical solution, the triple early stopping strategy decides whether to stop training early based on the model loss function, the accuracy of the validation set, and the stability of the optimal feature subset. The model loss function reflects the error of the model on the training data, the accuracy of the validation set reflects the generalization ability of the model, and the stability of the optimal feature subset ensures that the features on which the model depends will not fluctuate dramatically. When one or more of these three indicators meet the early stopping conditions, stopping training can prevent the model from overfitting in the later stages of training and ensure the stability of model training. For example, when it is found that the accuracy of the model on the validation set is no longer improving, but the loss function on the training set continues to decrease, it may mean that the model begins to overfit. At this time, the early stopping strategy will come into play to avoid the deterioration of model performance.

[0136] In the above technical solution, reasonable data division and effective training strategies avoid unnecessary model training time and waste of computing resources. Through the early stopping strategy, training is stopped in time when the model reaches a good performance, reducing invalid iterative calculations; the adaptive learning rate power decay rule improves computing efficiency while ensuring the quality of model convergence. This makes it possible to more efficiently build high-quality situational awareness assessment models with limited computing resources, facilitates the rapid application of advanced model technologies in the aviation field, and promotes the practical application and development of flight personnel operation status analysis technology.

[0137] In another embodiment of the present invention, the method of enhancing feature expression by a generative adversarial network to obtain the proliferated spatial position coding data comprises the following steps:

[0138] E1, input the eye movement trajectory spatial position coding obtained in S2 into the constructed adversarial generative network, use the generator in the adversarial generative network to extract and enhance the eye movement trajectory spatial position coding to generate intermediate feature data;

[0139] E2. According to a preset spatial position coding rule, the intermediate feature data is coded and converted to obtain intermediate spatial position coding data;

[0140] E3. Feed the intermediate spatial position coding data obtained in E2 back to the discriminator in the adversarial generative network. The discriminator judges its validity based on the preset judgment criteria. If it does not meet the criteria, it enters the generator to continue the optimization process until the amplified spatial position coding data that meets the criteria is obtained.

[0141] In the above technical solution, the spatial position encoding data of eye movement trajectories is innovatively processed through the adversarial generative network, which introduces new technical ideas and methods for the field of flight crew operation status analysis. The application and development of this technology not only helps to improve the accuracy and reliability of the current flight crew operation status assessment, but also provides a new direction for further research in this field. For example, subsequent research can build on this and explore how to more deeply integrate the adversarial generative network with other advanced technologies to further enhance the ability to analyze the flight crew's operation status and promote technological progress in this area in the entire aviation field.

[0142] The number of devices and processing scales described here are used to simplify the description of the present invention. Applications, modifications and variations of the flight personnel operation status analysis method of the present invention are obvious to those skilled in the art.

[0143] Although the embodiments of the present invention have been disclosed as above, they are not limited to the applications listed in the specification and the implementation modes, and they can be fully applied to various fields suitable for the present invention. For those familiar with the art, additional modifications can be easily implemented. Therefore, without departing from the general concept defined by the claims and the scope of equivalents, the present invention is not limited to the specific details and the illustrations shown and described herein.

Claims

1. A method for analyzing flight crew operation status, characterized in that: include: S1. Use an eye tracker to collect the pilot's eye movement trajectory data, and slice the normalized eye movement trajectory according to a preset picosecond fixed step size to generate eye movement time series data with a resolution of 1 picosecond; at the same time, use an electromyography sensor array to collect eye muscle electrical activity signals and convert them into electromyography feature vectors; S2, mapping the gaze point coordinates in the eye movement trajectory into a three-dimensional space, dividing the voxel region and encoding it, combining the voxel region encoding with the electromyographic feature vector, classifying and encoding according to the eye muscle coordinated movement group, and generating multi-modal deep fusion eye movement trajectory spatial position encoding data; S3, fusing the eye movement time series data after importance weight allocation with the eye movement trajectory spatial position encoding data after proliferation to obtain fused data; S4, training the fused data in the XGBoost model to obtain a situation awareness assessment model; S5. Analyzing the operating status of the flight crew according to the situation awareness assessment model; Among them, the eye movement time series data is based on a position-aware multi-head self-attention network, which automatically learns the importance weights of eye movement data in different time periods to situation awareness, and highlights key time segment information according to the importance weights to obtain the eye movement time series data after importance weight allocation; the eye movement trajectory spatial position encoding data is enhanced by feature expression through an adversarial generative network to obtain the proliferated eye movement trajectory spatial position encoding data.

2. The flight crew operation status analysis method according to claim 1, characterized in that: The eye movement trajectory data in S1 is the eye dynamic information of the pilot throughout the spatial orientation training, including the gaze point coordinates, eye saccade start and end positions and paths, and pupil scaling changes under different visual stimuli recorded by the nanosecond timestamp of the eye tracker; The eye muscle electrical activity signals include eye muscle electrical activity signals when actively looking at different directions, eye muscle electrical activity signals when performing a specific target gaze tracking visual task, and eye muscle electrical activity signals when performing a flight parameter visual task.

3. The flight crew operation status analysis method according to claim 1, characterized in that: The eye movement trajectory data described in S1 needs to be denoised before normalization, including: For the eye movement trajectory data, a Kalman filter fault-tolerant algorithm based on multimodal data fusion is used to eliminate abnormal data fragments caused by occasional jitter of the equipment, instantaneous head displacement of the pilot and strong light interference, and a Mallat algorithm is used to remove high-frequency noise.

4. The flight crew operating status analysis method according to claim 3, characterized in that: The specific steps of the Kalman filter fault-tolerant algorithm based on multimodal data fusion to remove abnormal data fragments caused by occasional equipment jitter, instantaneous head displacement of the pilot and strong light interference are as follows: A1. Use vibration sensors to collect equipment vibration data, motion capture sensors to collect pilot motion data, and light sensors to collect ambient light data; A2. Based on the Kalman filter fault-tolerant algorithm, the data from the vibration sensor, motion capture sensor and light sensor are integrated to process the eye movement trajectory data, and abnormal data fragments caused by occasional device jitter, instantaneous head displacement of the pilot and strong light interference are eliminated.

5. The flight crew operation status analysis method according to claim 3, characterized in that: Using the Mallat algorithm, removing high-frequency noise includes the following steps: B1, dividing the eye movement trajectory data into a plurality of eye movement trajectory segments according to a preset time interval; B2. Processing any eye movement trajectory segment using the Mallat algorithm with preset adaptation parameters to decompose the eye movement trajectory segment into a high-frequency sub-band and a low-frequency sub-band; B3. Identify the portion of the high frequency sub-band whose amplitude exceeds a preset high frequency noise amplitude threshold as high frequency noise, and remove it.

6. The flight crew operating status analysis method according to claim 5, characterized in that: The collection time of the eye movement trajectory data described in B1 is 10-60s, and the preset time interval is 0.1-0.5s; the high-frequency noise amplitude threshold in B3 is 0.05-0.2 volts.

7. The flight crew operation status analysis method according to claim 1, characterized in that: The eye muscle electrical activity signal described in S1 is converted into an electromyographic feature vector by the following steps: C1. Use blind source separation algorithm to remove environmental electrical noise and common mode interference; C2. The eye muscle electrical activity signal is extracted through the Hilbert-Huang transform to extract the characteristic frequency band of muscle contraction and relaxation, and converted into an electromyographic feature vector that can be directly used by the XGBoost model.

8. The flight crew operation status analysis method according to claim 1, characterized in that: In S4, the fusion data is trained in the XGBoost model to obtain the situation awareness assessment model in the following specific steps: D1, input the fused data obtained in S3 into the recursive neural network variant structure, use the memory unit of the recursive neural network variant structure to capture the long time series dependency, and output a dynamically updated feature representation with time series; D2, then input the feature representation obtained in D1 into the multi-layer convolutional neural network connected by the dynamic routing algorithm, automatically adjust the network depth and convolution kernel parameters according to the feedback during the training process of the multi-layer convolutional neural network, optimize the extraction of spatial features, and finally connect the processed eye feature data to the XGBoost model; D3. A feature filter driven by a quantum evolutionary algorithm using the superposition state characteristics of quantum bits is used to filter the eye feature data obtained in D2. Based on the contribution ranking of the eye feature data in improving the prediction accuracy of the model and a preset dynamic adaptive threshold, the current optimal feature subset is selected as a training set, and training is performed in an XGBoost model to obtain a situational awareness assessment model.

9. The flight crew operating status analysis method according to claim 8, characterized in that: The process of training the optimal feature subset as a training set in the XGBoost model also includes: d1, dividing a validation set from the fused data obtained in S3; d2. The XGBoost model is trained based on the data in the fusion data except the validation set, and uses the triple early stopping strategy and the adaptive learning rate power decay rule to make the situation awareness assessment model converge to the global optimal or approximate optimal solution; Among them, in the triple early stopping strategy, whether to stop training early is determined based on the model loss function, the accuracy of the validation set, and the stability of the optimal feature subset during the XGBoost model training process; the model loss function is calculated based on the data during the model training process, the validation set accuracy is calculated based on the validation set data, and the feature subset stability is determined based on the feature subset data involved in the model training process.

10. The flight crew operation status analysis method according to claim 1, characterized in that: The method of enhancing feature expression by a generative adversarial network to obtain the proliferated spatial position coding data comprises the following steps: E1, input the eye movement trajectory spatial position coding obtained in S2 into the constructed adversarial generative network, use the generator in the adversarial generative network to extract and enhance the eye movement trajectory spatial position coding to generate intermediate feature data; E2. According to a preset spatial position coding rule, the intermediate feature data is coded and converted to obtain intermediate spatial position coding data; E3. Feed the intermediate spatial position coding data obtained in E2 back to the discriminator in the adversarial generative network. The discriminator judges its validity based on the preset judgment criteria. If it does not meet the criteria, it enters the generator to continue the optimization process until the amplified spatial position coding data that meets the criteria is obtained.

Images