A pilot situation awareness ability evaluation method based on Ti-SA model

Abstract

The invention relates to a pilot situation awareness ability evaluation method based on a Ti-SA model. The Ti-SA model comprises an eye movement track multi-dimensional attention enhancement module, an AOI multi-scale comprehensive feature module and a fusion module. The evaluation method comprises: collecting eye movement track time series data and a flight training video of the pilot at continuous moments in real time, obtaining corresponding AOI gaze time series data based on the eye movement track time series data, the flight training video and a preset AOI subdivision area; preprocessing the eye movement track time series data and the AOI gaze time series data; inputting the preprocessed eye movement track time series data into the eye movement track multi-dimensional attention enhancement module to obtain a category probability value corresponding to the eye movement track time series data; inputting the preprocessed AOI gaze time series data into the AOI multi-scale comprehensive feature module to obtain a category probability value corresponding to the AOI fusion feature; and inputting the two category probability values ​​into the fusion module to obtain a predicted situation awareness level.

Description

Technical Field

[0001] The present invention relates to the field of aviation safety and situational awareness technology, and in particular to a pilot situational awareness capability evaluation method based on a Ti-SA model. Background Art

[0002] Aviation technology has made great progress, and the performance, functions and complexity of aircraft have been significantly improved. However, flight safety is still an urgent problem to be solved. The pilot's situational awareness (SA) ability is the pilot's attitude and motion perception of time, space and direction in a high-altitude closed cockpit environment. It is an important factor affecting the pilot's flight decision-making and flight safety. The higher the situational awareness ability, the more timely, accurate and comprehensive the pilot can grasp the flight situation, so as to make better flight decisions and operations. On the contrary, low or lost situational awareness will lead to cognitive bias, misjudgment or ignorance of the flight situation by the pilot, thereby increasing the risk of flight accidents. The pilot's situational awareness ability is crucial to flight safety. Studies have shown that 15%-20% of accidents are related to the pilot's lack of situational awareness. Spatial Disorientation (SD) is closely related to situational awareness SA. Pilots need to maintain good spatial orientation ability to ensure flight safety. SD is the inability or failure of directional judgment relative to the coordinate system composed of the ground and the vertical line of gravity. SD is closely related to SA ability. Specifically, the loss of SA ability by the pilot is inevitably accompanied by SD. Therefore, it is very important to build a system that can objectively evaluate the pilot's situational awareness during flight.

[0003] Pilot situational awareness assessment models constructed by existing technologies usually rely on traditional data analysis methods and subjective evaluation systems. These models are constructed based on questionnaire surveys, expert observations, or simple behavioral indicator analysis methods. The sample data is single and relies on the subjective judgment of experts, rather than on the real-time attitude and motion perception of pilots during flight training. The assessment is also conducted after the training is completed, which is lagging and inefficient. It is not suitable for large-scale motion capture and attitude capture, as well as real-time situational awareness assessment. Summary of the invention

[0004] In view of the above analysis, an embodiment of the present invention aims to provide a method for evaluating pilot situation awareness ability based on the Ti-SA model, so as to solve the technical problems of low real-time and low accuracy in evaluating pilot situation awareness ability by existing methods.

[0005] The present invention provides a pilot situation awareness capability evaluation method based on the Ti-SA model, comprising the following steps:

[0006] The Ti-SA model includes an eye tracking multi-dimensional attention enhancement module, an AOI multi-scale comprehensive feature module, and a fusion module; the evaluation method includes the following steps:

[0007] Step S1, collecting the pilot's eye movement tracking time series data and flight training video at consecutive moments in real time, and obtaining corresponding AOI gaze time series data based on the eye movement tracking time series data, the flight training video and the preset AOI subdivision area;

[0008] Step S2, preprocessing the eye movement tracking time series data and the AOI gaze time series data;

[0009] Step S3, inputting the preprocessed eye movement track time series data into the eye movement track multi-dimensional attention enhancement module to obtain the category probability value corresponding to the eye movement track time series data;

[0010] Inputting the preprocessed AOI gaze time series data into the AOI multi-scale comprehensive feature module to obtain the category probability value corresponding to the AOI fusion feature;

[0011] The category probability value corresponding to the eye movement tracking time series data and the category probability value corresponding to the AOI fusion feature are input into the fusion module to obtain the predicted situation awareness level.

[0012] Further, the eye tracking time series data and the AOI gaze time series data are preprocessed, including:

[0013] Unifying the timestamp formats of the eye tracking time series data and the AOI gaze time series data;

[0014] Identify missing values ​​of the eye tracking time series data and the AOI gaze time series data, mark the missing values, and further perform Z-score standardization to obtain standardized data;

[0015] The standardized data is input into an isolation forest model, an outlier score is generated for each data point using the isolation forest model, and based on a preset ratio, the data points with the highest score among all the outlier scores are taken as outliers to obtain outliers in the eye tracking time series data and the AOI gaze time series data;

[0016] Use the KNN algorithm to fill the missing values ​​and outliers, and find the mean of the K neighbors of the missing values ​​and outliers to fill them; use the KNN algorithm and the anytime forest algorithm to fill the missing values ​​and outliers in the eye tracking data and the AOI gaze time series data respectively;

[0017] The preprocessed eye tracking time series data and AOI gaze time series data are obtained.

[0018] Furthermore, the Ti-SA model is obtained by training through the following steps:

[0019] The first step is to obtain the eye movement tracking time series data and AOI gaze time series data of multiple pilots during simulated flight as sample data, generate corresponding category labels as a training data set, and divide the training data set into a training set and a validation set;

[0020] The second step is to determine the hyperparameter space and build the Ti-SA model; the hyperparameters include learning rate and training batch size;

[0021] Step 3: Use the cross entropy loss function to train the Ti-SA model using the training set until the loss function converges to obtain the trained Ti-SA model;

[0022] Step 4: Use the average of the accuracy of the validation set on the trained Ti-SA model and the inverse of the cross entropy loss function as the objective function;

[0023] Step 5: Select a Gaussian process as a priori, randomly select a portion of points in the hyperparameter space as initial points to construct a probability model of the objective function, and use the evaluation result of each point in the initial points to update the probability model; based on the evaluation results of the initial points, use the Gaussian process to construct the posterior distribution of the objective function;

[0024] Step 6: Continue to select the next part of points in the hyperparameter space and repeat step 5 until a preset number of iterations or a preset performance threshold is reached, and finally a set of hyperparameter configurations is obtained as the optimal hyperparameter configuration of the Ti-SA model;

[0025] Step 7: Based on the optimal hyperparameter configuration, continue to train the Ti-SA model using the training data set until the loss function converges to obtain a trained Ti-SA model.

[0026] Furthermore, the preset AOI subdivision area is determined in the following steps:

[0027] Identify the types of instrument panels and displays in aircraft;

[0028] Based on the instrument panel display area and the screen display area, the instrument panel and the screen display are subdivided into a plurality of logical AOI subdivision areas; the exterior scene is one AOI subdivision area;

[0029] Clarify the scope and meaning of each AOI subdivision area to ensure that each AOI subdivision area covers the information that pilots need to pay attention to in specific situations;

[0030] The AOI subdivision areas are numbered to obtain m AOI subdivision areas.

[0031] Furthermore, obtaining the training data set includes:

[0032] Obtaining the eye movement tracking time series data and flight training video during the pilot's simulated flight;

[0033] Detecting and obtaining all AOI subdivision areas in the flight training video in time sequence based on the flight training video;

[0034] Based on the two-dimensional and three-dimensional coordinates of the gaze focus in the eye tracking time series data, determine whether the coordinates are located in a certain AOI subdivision area, and record the AOI subdivision area number of the pilot's gaze in chronological order according to the matching result to obtain the AOI gaze time series data;

[0035] The eye tracking time series data and the AOI gaze time series data are used as sample data;

[0036] Execute the following process to generate the category label corresponding to the sample data, as follows:

[0037] Based on the situational awareness assessment items, measurement indicators, recovery results, planned results, and the full score of the situational awareness assessment items, the assessment result Score of each pilot's situational awareness assessment item is calculated as follows:

[0038]

[0039] Where n is the total number of indicators of situational awareness assessment items, D i M is the deviation between the pilot's recovery result and the expected result. i is the maximum deviation value in the indicator, fullscore i It is the full score obtained by the pilot when the recovery result of the indicator corresponding to the situational awareness assessment item is completely consistent with the expected result;

[0040] The level corresponding to the evaluation result is used as the corresponding category label of the eye tracking time series data and AOI gaze time series data sample data obtained during the flight simulation;

[0041] The sample data and the obtained corresponding category labels constitute a training data set.

[0042] Furthermore, the eye movement tracking multi-dimensional attention enhancement module includes:

[0043] A first linear projection layer, used to map the eye tracking time series data features to a high dimension;

[0044] The first random deactivation layer is used to perform random deactivation processing on the mapped high-dimensional time series data to enhance the generalization of the eye movement trace multi-dimensional attention enhancement module and prevent overfitting;

[0045] A first, a second and a third multi-layer attention structure connected in sequence, for obtaining an encoded representation of the eye tracking temporal data;

[0046] Bi-LSTM, which uses the bidirectional capability of LSTM to enhance the understanding of the time series of the encoded representation output after three multi-layer attention structures;

[0047] Adaptive average pooling layer to reduce the dimensionality of Bi-LSTM output features while retaining important information;

[0048] A second linear projection layer is used to match the output dimension to obtain three category probability values ​​corresponding to the eye movement trace;

[0049] Among them, the multi-head attention module in the three-layer attention structure divides the input features of the module into multiple attention heads. Each head processes the feature information independently and understands the features from different angles. Finally, the outputs of multiple attention heads are merged to obtain a comprehensive feature representation.

[0050] Further, the AOI multi-scale comprehensive feature module includes six sequentially connected first to sixth heuristic modules, a global average pooling layer, a third random inactivation layer and a third linear projection layer;

[0051] The first to sixth heuristic modules are used to perform multi-scale feature extraction on the AOI gaze time series data; wherein the AOI gaze time series data is connected to the output of the second heuristic module through a first residual; the output of the third heuristic module is connected to the output of the fifth heuristic module through a second residual;

[0052] The global average pooling layer is used to reduce the dimension of the output features of the first heuristic module and extract global features;

[0053] The third random deactivation layer is used to perform random deactivation processing on the extracted global features to enhance the generalization ability of the AOI multi-scale comprehensive feature module;

[0054] The third linear projection layer is used to reduce the dimension of the matching output dimension to obtain three category probability values ​​corresponding to the AOI fusion feature.

[0055] Further, the first to sixth heuristic modules each include a first component, a second component and an output layer, as follows:

[0056] The first component is the first bottleneck layer, which uses a filter with a length of 1 to reduce the dimension of the input multi-dimensional AOI gaze time series data;

[0057] The second component includes three parallel convolutions, a max pooling, and a second bottleneck layer;

[0058] The three parallel convolutions are used to perform convolution operations of three lengths on the output of the first bottleneck layer to capture temporal features of different scales;

[0059] The maximum pooling is used to perform a maximum pooling operation on the input multi-dimensional AOI gaze time series data;

[0060] The second bottleneck layer is used to reduce the dimension of the maximum pooling output;

[0061] The output layer is used to concatenate the time series features of different scales captured by the three parallel convolutions and the output features of the maximum pooling to form the final output multi-dimensional time series.

[0062] Further, the fusion module averages the probability values ​​of the three categories corresponding to the eye tracking time series data and the three category probability values ​​corresponding to the AOI fusion feature to obtain an averaged probability value;

[0063] Comparing the averaged probability values ​​of the three categories to determine the category with the highest probability;

[0064] The category with the largest average probability value is taken as the pilot's situation awareness level.

[0065] Furthermore, it is characterized in that the eye movement tracking time series data includes the two-dimensional coordinates of the gaze focus, the three-dimensional coordinates of the gaze focus, the three-dimensional coordinates of the original position of the left eye, the three-dimensional coordinates of the moving direction of the left eye, the pupil diameter of the left eye, the three-dimensional coordinates of the original position of the right eye, the three-dimensional coordinates of the moving direction of the right eye and the pupil diameter of the right eye;

[0066] The AOI gaze timing data includes a number sequence of AOI subdivision areas that the pilot gazes at in sequence based on time during flight; the AOI subdivision areas include A sub-areas divided based on the instrument panel display area and the screen display area, and an exterior view area.

[0067] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:

[0068] 1. The present invention collects the pilot's eye tracking time series data and AOI gaze time series data in real time, and uses the Ti-SA model to process and analyze them, so as to achieve real-time evaluation of the pilot's situational awareness ability. This method has higher real-time and accuracy than the traditional method based on questionnaire survey, expert observation or post-training evaluation, and can be directly used to evaluate the actual performance of pilots during flight training;

[0069] 2. By combining eye tracking data and AOI gaze data, the present invention provides a more comprehensive assessment of pilot situational awareness, which does not rely on the subjective judgment of experts, but is based on objective data-driven analysis. The method of the present invention can more accurately capture the pilot's cognitive state of the flight environment, thereby improving the reliability and effectiveness of the assessment results;

[0070] 3. The evaluation method of the present invention can identify possible cognitive deficiencies or errors of pilots at an early stage, so as to take preventive measures and reduce flight accidents caused by insufficient situational awareness. By analyzing the situational awareness level during training, customized training suggestions can be provided to pilots to help them improve their flying skills and ability to deal with complex situations, thereby improving overall flight safety;

[0071] 4. The Ti-SA model in the present invention can adapt to the cognitive characteristics of different pilots and different flight stages, provide personalized evaluation results, and enhance the adaptability and flexibility of the evaluation method; based on real-time evaluation of the pilot's situational awareness ability, the pilot training system can adjust the training strategy in time and optimize the pilot's training effect, so as to help the pilot improve his flying skills and ability to cope with complex flight situations.

[0072] In the present invention, the above-mentioned technical solutions can also be combined with each other to achieve more preferred combination solutions. Other features and advantages of the present invention will be described in the subsequent description, and some advantages can become obvious from the description, or can be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained through the contents particularly pointed out in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] The drawings are only for the purpose of illustrating specific embodiments and are not to be considered limiting of the present invention. Like reference symbols denote like components throughout the drawings.

[0074] Figure 1 This is a flow chart of a method for evaluating pilot situation awareness capability based on the Ti-SA model in an embodiment of the present invention;

[0075] Figure 2 Schematic diagram of the overall structure of the Ti-SA model in an embodiment of the present invention;

[0076] Figure 3 is a flow chart of a method for acquiring eye movement tracking time series data and AOI gaze time series data in an embodiment of the present invention;

[0077] Figure 4 This is a schematic diagram of the pilot's eye movement traces in a spatial directional flight simulation state in an embodiment of the present invention - a schematic diagram of the pilot's gaze focus above the runway;

[0078] Figure 5 This is a schematic diagram of the pilot's eye movement traces in a spatial directional flight simulation state in an embodiment of the present invention - a schematic diagram of the pilot's gaze focus on the instrument panel area;

[0079] Figure 6 Schematic diagram of AOI area division of real scene in the cabin in an embodiment of the present invention;

[0080] Figure 7 Schematic diagram of the heuristic module structure in an embodiment of the present invention. DETAILED DESCRIPTION

[0081] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.

[0082] In response to the problems of the current existing technology, the present invention constructs a data set based on deep learning, and uses a deep learning method to perform SA in a pilot's spatial directional flight simulation state, and proposes a high-accuracy Ti-SA model. The evaluation accuracy is improved by fusing eye movement tracking time series data and AOI (Area of ​​Interest) gaze time series data, which is used for evaluating the pilot's situational awareness ability in a spatial directional flight simulation state, as a preliminary exploration in this field.

[0083] A specific embodiment of the present invention discloses a method for evaluating pilot situation awareness based on the Ti-SA model, such as Figure 1 and Figure 2 As shown, the Ti-SA model includes an eye tracking multi-dimensional attention enhancement module, an AOI multi-scale comprehensive feature module, and a fusion module; the evaluation method includes the following steps:

[0084] Step S1, collecting the pilot's eye movement tracking time series data and flight training video at consecutive moments in real time, and obtaining corresponding AOI gaze time series data based on the eye movement tracking time series data, the flight training video and the preset AOI subdivision area;

[0085] Step S2, preprocessing the eye movement tracking time series data and the AOI gaze time series data;

[0086] Step S3, inputting the preprocessed eye movement track time series data into the eye movement track multi-dimensional attention enhancement module to obtain the category probability value corresponding to the eye movement track time series data;

[0087] Inputting the preprocessed AOI gaze time series data into the AOI multi-scale comprehensive feature module to obtain the category probability value corresponding to the AOI fusion feature;

[0088] The category probability value corresponding to the eye movement tracking time series data and the category probability value corresponding to the AOI fusion feature are input into the fusion module to obtain the predicted situation awareness level.

[0089] The test equipment involved in the present invention is as follows:

[0090] (1) Flight simulator: The DISO space orientation flight simulator produced by AMST of Austria is used. It uses a three-dimensional 6-DOF hydraulic platform and a highly simulated cockpit vision system. It can simulate more than 10 kinds of vestibular and visual illusions to intuitively experience the various SDs generated by the movement of the human body's orientation organs when facing an abnormal three-dimensional space environment.

[0091] (2) Wearable eye tracker: The Tobii Pro Glasses 3 eye tracker produced by Swedish company Tobii is used. It has a sampling rate of 100 Hz, integrates 16 light sources and 4 eye tracking sensors, and is embedded in scratch-resistant lenses. The eye tracker has a 106° wide-angle scene camera that can record behavioral details from a first-person perspective.

[0092] The test conditions involved in the present invention are as follows: the test sequentially performs the following simulated flight links: at an altitude of 914m and a speed of 400 knots, the pitch illusion is experienced at an elevation angle of 10°, the Coriolis illusion is experienced, and the tilt illusion is experienced at a slope of 45°; at a speed of 400 knots, the cloud base is 914m, and the cloud top is 3657m to experience the false horizon illusion; at an altitude of 1219m, a speed of 250 knots, and a heading of 163°, the black hole approach illusion is experienced.

[0093] Step S1 includes steps S11-S13.

[0094] Step S11: collect the pilot's eye movement tracking data and flight training videos in real time. Figure 3 shown.

[0095] The eye movement tracking time series data includes the two-dimensional coordinates of the gaze focus, the three-dimensional coordinates of the gaze focus, the three-dimensional coordinates of the original position of the left eye, the three-dimensional coordinates of the moving direction of the left eye, the pupil diameter of the left eye, the three-dimensional coordinates of the original position of the right eye, the three-dimensional coordinates of the moving direction of the right eye and the pupil diameter of the right eye;

[0096] The pilot's eye movement time series data and flight training video are obtained through a spatial orientation flight simulator and a wearable eye tracker. When the pilot's eyes look at the spatial orientation flight simulator screen through the eye tracker, the wearable eye tracker records the trajectory of his eye movement.

[0097] The basic working principle of the eye tracker: First, use infrared or other light sources to illuminate the eyes to produce two bright spots: corneal reflection and pupil reflection. Then use a camera or other image sensor to capture the image of the eye and extract the position information of the corneal reflection and pupil reflection. Then use advanced image processing algorithms and 3D eyeball models to calculate the direction of the eye and the coordinate position of the gaze point relative to the screen. Finally, there is a calibration process to map the eye's gaze point coordinates to the visual gaze coordinates on the screen, thereby obtaining the eye movement tracking time series data and flight training video of the test pilot.

[0098] Figure 4 The white circular outline area is the pilot's visual focus, and the red curve is the historical trajectory of the pilot's previous gaze movement. Figure 4 to Figure 5 It shows that the pilot's gaze focus shifts from the runway to the instrument panel. The characteristics of the acquired eye tracking time series data are shown in Table 1.

[0099] Table 1: Eye tracking time series data characteristics

[0100]

[0101] The two-dimensional coordinates of visual gaze are split into two-dimensional features of X and Y coordinates, and the three-dimensional coordinates are split into three-dimensional features of X, Y and Z coordinates.

[0102] The eye movement coordinate position is an important feature that reflects the pilot's visual information. In addition, the pupil diameter feature also plays a pivotal role. The pilot's left and right pupil diameter feature is an indicator that reflects the pilot's physiological and psychological state. The pilot's pupil diameter will change with factors such as lighting, visual stimulation, cognitive load, emotions, and fatigue. Generally speaking, the larger the pupil diameter, the higher the pilot's cognitive load; conversely, the smaller the pupil diameter, the lower the pilot's cognitive load. In addition, the difference in the pilot's left and right pupil diameters can also reflect the pilot's spatial orientation ability. If the difference in the left and right pupil diameters is large, it means that the pilot's spatial orientation ability is poor and is prone to spatial orientation disorders.

[0103] Step S12: obtaining corresponding AOI gaze timing data based on the eye movement tracking timing data, the flight training video and the preset AOI subdivision area.

[0104] Detecting and obtaining all AOI subdivision areas in the flight training video in time sequence based on the flight training video;

[0105] Based on the two-dimensional and three-dimensional coordinates of the gaze focus in the eye tracking time series data, it is determined whether the coordinates are located in a certain AOI subdivision area. According to the matching results, the AOI subdivision area numbers of the pilot's gaze are recorded in chronological order to obtain the AOI gaze time series data.

[0106] The AOI gaze timing data includes a number sequence of AOI subdivision areas that the pilot gazes at in sequence based on time during flight; the AOI subdivision areas include A sub-areas divided based on the instrument panel display area and the screen display area, and an exterior view area.

[0107] AOI is an important source of information for pilots to obtain and maintain situational awareness. Pilots form a comprehensive understanding and prediction of current and future situations by observing and analyzing information on aircraft instruments, radar, navigation, communication and other equipment, as well as external weather, terrain and other information. Therefore, pilots need to reasonably allocate and adjust their attention to AOI according to mission requirements and environmental changes. Studies have shown that the longer the time spent paying attention to the AOI subdivision area, the more attention they pay to situational awareness information.

[0108] There are three main areas that pilots observe during flight: Figure 6 Number 1 is the instrument panel, number 2 is the screen display and number 3 is the exterior view.

[0109] For example, in the present invention, the three areas of instrument panel, screen display and exterior view are divided into 13 preset AOI subdivision areas, such as Figure 6 And as shown in Table 2.

[0110] The AOI subdivision area is mainly based on the instrument type. Pilots will use and pay attention to different instruments at different stages. The specific meanings of the preset AOI subdivision areas are shown in Table 2.

[0111] Table 2: Meaning of AOI subdivision areas in the real scene inside the space directional flight simulator and the outdoor location

[0112]

[0113] The preset AOI subdivision area is determined in the following steps:

[0114] Step 1: Identify the types of instrument panels and screen displays in the flight simulator;

[0115] Step 2: Based on the instrument panel function module and the screen information display area, the instrument panel and the screen display are subdivided into a plurality of logical AOI subdivision areas; the exterior scene is one AOI subdivision area;

[0116] Step 3: Clarify the scope and meaning of each AOI segment to ensure that each AOI segment covers the information that pilots need to pay attention to in a specific situation.

[0117] Step 4: number the AOI subdivision areas to obtain m AOI subdivision areas.

[0118] For example, based on the instrument categories of the space orientation simulation aircraft in the present invention, 13 AOI subdivision areas are determined, wherein the exterior scene is counted as one AOI subdivision area.

[0119] Since the 6-DOF spatial orientation flight simulator has a non-fixed viewing angle, it is necessary to directly Figure 6 Transforming to a regional template for matching does not scale well to all pilot perspectives.

[0120] If you directly Figure 6 The displayed flight simulator instrument panel is considered as a regional template and then matched with the pilot's eye tracking time series data, the following problems will occur:

[0121] When the pilot's view rotates or tilts, Figure 6 The regional template also needs to be rotated or tilted accordingly to align with the pilot's eye tracking time series data, which increases the complexity and error of the calculation; when the pilot's perspective is translated or scaled, Figure 6 The area template also needs to be translated or scaled accordingly to align with the pilot's eye tracking time series data, which will cause the position and shape of the AOI to change, thereby affecting the accuracy of AOI recognition and matching; when the pilot's perspective is blocked or distorted, Figure 6 The area template may not completely cover the pilot's eye tracking timing data, or may be inconsistent with the pilot's eye tracking timing data, resulting in loss or error of AOI.

[0122] In order to solve the above problems, the present invention uses a method of AOI recognition and matching based on deep learning to train the YOLOv8n model to recognize the AOI area, as follows:

[0123] The first step is to use the lightweight YOLOv8n model, input the image data of 13 AOI subdivision areas for training, and obtain a neural network model that can detect AOI. The YOLOv8n model is an improved target detection model based on the YOLO series. YOLOv8n replaces the C3 structure of YOLOv8 with a C2f structure with richer gradient flow, and adjusts the number of channels for models of different scales, thereby improving the performance and efficiency of the model; uses a new detection head to separate the classification and regression branches, and changes from Anchor-Based to Anchor-Free, thereby improving the accuracy and robustness of detection; uses a new loss function to balance the contributions of different targets, thereby improving the stability and generalization of detection;

[0124] The second step is to input the flight training video recorded by the eye tracker into the trained network for inference, which can detect the AOI subdivision area in the video, as well as the position and shape of the AOI subdivision area. No matter how the pilot's perspective changes, the pilot's visual gaze area can be identified in real time, thus adapting to all perspectives;

[0125] The third step is to use the focus coordinates in the eye tracking time series data to calculate whether it is located in a certain AOI subdivision area, and then get the AOI gaze sequence. The AOI gaze sequence refers to the number of AOIs that the pilot gazes in chronological order during the flight. It can reflect the pilot's visual attention and cognitive process, as well as the pilot's situational awareness level.

[0126] Since the AOI area features are relatively simple, we use the lightweight YOLOv8n model to input 13 ( Figure 6 In the training, the AOI subdivision area is trained, and then the flight training video recorded by the eye tracker is input into the trained YOLOv8n network for inference to detect the AOI subdivision area. The AOI gaze sequence data can be obtained by calculating whether the focus coordinates in the eye movement tracking time series data are located in a certain AOI subdivision area.

[0127] Step S1 acquires the eye tracking data and AOI gaze time series data of the pilots in real time during the flight training. The two types of data are the basic data for evaluating the pilots' situational awareness ability.

[0128] Step S2: preprocessing the eye movement tracking data and AOI gaze timing data obtained in step S1 to obtain preprocessed eye movement tracking data and AOI gaze timing data.

[0129] Preprocessing the eye tracking time series data and the AOI gaze time series data includes:

[0130] Unifying the timestamp formats of the eye tracking time series data and the AOI gaze time series data;

[0131] Identify missing values ​​of the eye tracking time series data and the AOI gaze time series data, mark the missing values, and further perform Z-score standardization to obtain standardized data;

[0132] The standardized data is input into an isolation forest model, an outlier score is generated for each data point using the isolation forest model, and based on a preset ratio, the data points with the highest score among all the outlier scores are taken as outliers to obtain outliers in the eye tracking time series data and the AOI gaze time series data;

[0133] Use the KNN algorithm to fill the missing values ​​and outliers, and find the mean of the K neighbors of the missing values ​​and outliers to fill them; use the KNN algorithm and the anytime forest algorithm to fill the missing values ​​and outliers in the eye tracking data and the AOI gaze time series data respectively;

[0134] The preprocessed eye tracking time series data and AOI gaze time series data are obtained.

[0135] (1) Unify the timestamps. Unify the timestamp formats of the eye tracking data and the AOI gaze timing data to ensure that the timestamp formats of the two are consistent, which facilitates the alignment of timestamps in subsequent steps.

[0136] (2) Data standardization: Identify missing values ​​in the eye tracking data and AOI gaze timing data, and annotate the missing values. Currently, this step does not delete or fill missing values. Then, the eye tracking data and AOI gaze timing data are further Z-score standardized to obtain standardized data.

[0137] The eye tracker's sensor may malfunction or perform erratically, resulting in inaccurate eye movement data. Data may be lost due to technical glitches during data transmission or storage. Z-score normalization is used to ensure that all features have similar scales so that the algorithm can more effectively identify outliers.

[0138] (3) Identify outliers:

[0139] Based on the standardized data, the isolation forest model is used for outlier identification.

[0140] Exemplarily, an isolation forest model with 100 trees is constructed, the standardized data is input into the isolation forest model, and an outlier score is performed on each data point to obtain an outlier score for each data point.

[0141] Each outlier score is assigned, and the higher the score, the more likely it is to be an outlier. According to the distribution of the scores, a suitable preset ratio is selected. For example, 1% is used as the preset ratio limit, that is, the data points with the highest outlier scores, which account for 1% of the total data volume, are identified as outliers.

[0142] (4) Fill in missing values ​​and outliers.

[0143] Since the eye tracking time series data has a historical continuous curve, the KNN algorithm is used to fill missing values ​​and outliers. The missing values ​​and outliers are estimated by finding the K neighbors that are most similar to the missing values ​​or outliers, and the distance metric is Euclidean distance. The mean of the K neighbors is used for filling. For example, K is selected as 10.

[0144] AOI time series data usually contains nonlinear relationships, and the random forest method is used to fill missing values ​​and outliers. Random forest is an integrated learning method that can effectively capture nonlinear relationships in data and process high-dimensional data. During the filling process, random forest automatically identifies important features and uses related features for filling, which improves the effectiveness of filling and has strong robustness to noise and outliers.

[0145] For abnormal values ​​of eye tracking data and AOI gaze timing data, fill values ​​are directly used to cover the data; for missing values, fill values ​​are directly used to fill them.

[0146] Step S3 includes steps S31-S33.

[0147] Step S31, constructing a training data set.

[0148] (1) Obtaining sample data. Obtaining the eye movement tracking time series data and flight training videos of multiple pilots during simulated flight;

[0149] Detecting and obtaining all AOI subdivision areas in the flight training video in time sequence based on the flight training video;

[0150] Based on the two-dimensional and three-dimensional coordinates of the gaze focus in the eye tracking time series data, determine whether the coordinates are located in a certain AOI subdivision area, and record the AOI subdivision area number of the pilot's gaze in chronological order according to the matching result to obtain the AOI gaze time series data;

[0151] The eye tracking time series data and the AOI gaze time series data are used as sample data;

[0152] Obtain eye movement tracking time series data and flight training videos of multiple pilots during simulated flight;

[0153] Based on the flight training video detection, all AOI subdivision areas in the flight training video in time sequence are obtained; based on the two-dimensional and three-dimensional coordinates of the gaze focus in the eye tracking time series data, it is determined whether the coordinates are located in a certain AOI subdivision area, and according to the matching results, the AOI subdivision area numbers that the pilot is fixated on are recorded in time sequence to obtain the AOI gaze time series data.

[0154] (2) Generate category labels corresponding to sample data.

[0155] Generate category labels corresponding to eye tracking time series data and AOI gaze time series data sample data.

[0156] For example, the situational awareness assessment items of the simulation experiment training include: pitch illusion, Coriolis illusion, tilt illusion, false horizon illusion, runway width and depth illusion, and black hole approach illusion, as described in Table 3.

[0157] Table 3: Situational awareness assessment items

[0158]

[0159] If the corrected result of each situation awareness assessment item is completely consistent with the expected result, the full score can be obtained. The specific situation awareness assessment item and the corresponding corrected full score are fullscore. i Definition, based on the specific needs of each assessment.

[0160] Execute the following process to generate the category label corresponding to the sample data, as follows:

[0161] Based on the situation awareness assessment items, measurement indicators, recovery results, planned results, and the full score of the situation awareness assessment items, the assessment result Score of each pilot's situation awareness assessment item is calculated, as shown in formula (1):

[0162]

[0163] Where n is the total number of indicators of situational awareness assessment items, D i M is the deviation between the pilot's recovery result and the expected result. i is the maximum deviation value in the indicator, fullscore i It is the full score obtained by the pilot when the recovery result of the indicator corresponding to the situational awareness assessment item is completely consistent with the expected result;

[0164] The level corresponding to the evaluation result is used as the corresponding category label of the eye tracking time series data and AOI gaze time series data obtained during the flight simulation.

[0165] For each metric of the situational awareness assessment project, exemplarily, as shown in Table 4, for a pitch illusion dive situation, the pilot may recover successfully, or fail to recover, or be in the middle stage between successful recovery and failure to recover. Exemplarily, complex situation recovery is shown in Table 4.

[0166] Table 4: Complex situation recovery items for situation awareness assessment

[0167]

[0168] The eye tracking time series data and AOI gaze sequence data in the sample data are both 1 matrix, that is, the data itself is a three-dimensional tensor of multi-dimensional time series data, which is expanded and spliced ​​into two dimensions according to the samples. Each sample has only one category label.

[0169] During the 60-minute training course, the pilots were given the background knowledge of the SA problem, and the test personnel guided them on how to use the space orientation flight simulator to complete the situational awareness assessment project. Each pilot had a 30-minute space orientation flight simulator operation experience to ensure that the pilots could successfully operate the space orientation flight simulator to collect data.

[0170] During flight, pilots face various complex situations. Complex situation recovery for pilots refers to the process of the pilot restoring the aircraft from an abnormal flight state to a normal controllable state, which is an important part of pilot training.

[0171] The level corresponding to the scoring result is as shown in formula (2):

[0172]

[0173] The corresponding level of the scoring result is used as the corresponding category label of the eye tracking time series data and AOI gaze time series data obtained during the flight simulation. If the scoring result is less than 60 points, it is considered a simulated flight failure, and it is not counted as a sample, and the spatial orientation flight simulation test is repeated.

[0174] Acquire the pre-processed eye tracking time series data and AOI gaze time series data of multiple pilots during simulated flight as sample data, wherein the sample data and the generated corresponding category labels constitute a training data set; wherein the category label is a pilot situation awareness level label;

[0175] For example, 80% of the training data set is used as a training set, and 20% is used as a validation set.

[0176] Step S32: construct and train a Ti-SA model for pilot situation awareness assessment.

[0177] The Ti-SA model is trained by the following steps:

[0178] The first step is to obtain the eye movement tracking time series data and AOI gaze time series data of multiple pilots during simulated flight as sample data, generate corresponding category labels as a training data set, and divide the training data set into a training set and a validation set;

[0179] The second step is to determine the hyperparameter space and build the Ti-SA model; the hyperparameters include learning rate and training batch size;

[0180] Step 3: Use the cross entropy loss function to train the Ti-SA model using the training set until the loss function converges to obtain the trained Ti-SA model;

[0181] Step 4: Use the average of the accuracy of the validation set on the trained Ti-SA model and the inverse of the cross entropy loss function as the objective function;

[0182] Step 5: Select a Gaussian process as a priori, randomly select a portion of points in the hyperparameter space as initial points to construct a probability model of the objective function, and use the evaluation result of each point in the initial points to update the probability model; based on the evaluation results of the initial points, use the Gaussian process to construct the posterior distribution of the objective function;

[0183] Step 6: Continue to select the next part of points in the hyperparameter space and repeat step 4 until a preset number of iterations or a preset performance threshold is reached, and finally a set of hyperparameter configurations is obtained as the optimal hyperparameter configuration of the Ti-SA model;

[0184] Step 7: Based on the optimal hyperparameter configuration, continue to train the Ti-SA model using the training data set until the loss function converges to obtain a trained Ti-SA model.

[0185] Among them, the first step is to construct the training data set part, and its specific technical content is in step S31.

[0186] The second step is to determine the hyperparameter space and construct the Ti-SA model.

[0187] (1) Determine the hyperparameter space.

[0188] During the training process, choosing the right hyperparameters (e.g., learning rate, training batch size) is crucial to the performance of the model. Bayesian parameter tuning is a very efficient hyperparameter optimization method that aims to guide the selection of hyperparameters by building a probabilistic model. Determine the hyperparameter space of learning rate and batch size.

[0189] For example, the learning rate range is [1e-5, 1e-1], the training batch size is [8, 128], and the batch range size must be an integer.

[0190] (2) Construct the Ti-SA model.

[0191] like Figure 2 As shown in the overall structure diagram of the Ti-SA model, the Ti-SA model for pilot situation awareness assessment includes an eye tracking multi-dimensional attention enhancement module, an AOI multi-scale comprehensive feature module, a fusion module and a SA level output module;

[0192] The eye movement trace multi-dimensional attention enhancement module is used to receive the eye movement trace time series data, perform multi-dimensional attention feature extraction, and output the category probability value corresponding to the eye movement trace time series data;

[0193] The AOI multi-scale comprehensive feature module is used to receive the AOI gaze time series data, perform multi-scale feature fusion, and output the category probability value corresponding to the AOI fusion feature;

[0194] The fusion module fuses the eye tracking depth feature and the AOI fusion feature to obtain a predicted situation awareness level result;

[0195] The SA level output module is used to output the predicted situation awareness level result.

[0196] The present invention transforms the pilot situation awareness assessment problem into a multi-dimensional time series data classification problem. Since the amount of experimental data is huge and the sequence size is particularly long, it is particularly necessary to consider the model selection in this case. The overall structure of the proposed Ti-SA model is as follows: Figure 2 shown.

[0197] The eye tracking multi-dimensional attention enhancement module includes:

[0198] A first linear projection layer, used to map the eye tracking time series data features to a high dimension;

[0199] The first random deactivation layer is used to perform random deactivation processing on the mapped high-dimensional time series data to enhance the generalization of the eye movement trace multi-dimensional attention enhancement module and prevent overfitting;

[0200] A first, a second and a third multi-layer attention structure connected in sequence, for obtaining an encoded representation of the eye tracking temporal data;

[0201] Bi-LSTM, which uses the bidirectional capability of LSTM to enhance the understanding of the time series of the encoded representation output after three multi-layer attention structures;

[0202] Adaptive average pooling layer to reduce the dimensionality of Bi-LSTM output features while retaining important information;

[0203] A second linear projection layer is used to match the output dimension to obtain three category probability values ​​corresponding to the eye movement trace;

[0204] Among them, the multi-head attention module in the three-layer attention structure divides the input features of the module into multiple attention heads. Each head processes the feature information independently and understands the features from different angles. Finally, the outputs of multiple attention heads are merged to obtain a comprehensive feature representation.

[0205] Wherein, the multi-layer attention structure includes:

[0206] The multi-head attention module in the first multi-layer attention structure is used to obtain the output features of the first random inactivation layer, and use multiple attention heads to obtain comprehensive features; the multi-head attention module in the second multi-layer attention structure obtains the output features of the first multi-layer attention structure; the multi-head attention module in the third multi-layer attention structure obtains the output features of the second multi-layer attention structure;

[0207] The first residual connection and normalization layer in the first multi-layer attention structure is used to connect the data residual of the first random inactivation layer to the comprehensive features output by the multi-head attention module and then normalize them to obtain a normalized result 1A;

[0208] The first residual connection and normalization layer in the second multi-layer attention structure are used to connect the output feature residuals of the second multi-layer attention structure and then normalize them to obtain a normalized result 1B;

[0209] The first residual connection and normalization layer in the third multi-layer attention structure are used to residually connect the output features of the third multi-layer attention structure and then normalize them to obtain a normalized result 1C;

[0210] The feedforward network in each multi-layer attention structure is used to learn the nonlinear representation of the normalized results 1A, 1B and 1C respectively;

[0211] The second residual connection and normalization layer in each multi-layer attention structure are respectively used to perform residual connection on the normalized results 1A, 1B and 1C and the output of the feedforward network and then normalize them to obtain normalized results 2A, 2B and 2C;

[0212] The second random deactivation layer in each multi-layer attention structure is used to perform random deactivation processing on the normalized results 2A, 2B and 2C respectively to enhance the generalization ability of the multi-layer attention structure.

[0213] The core idea of ​​the attention mechanism is to use query vectors (Query), key vectors (Key) and value vectors (Value) to represent different parts of the input sequence, and obtain the part most relevant to the query vector by calculating the similarity between them. Then, by normalizing and weighted summing the similarity, an output vector is obtained, which is the output result of the multi-head attention module.

[0214] The attention mechanism imitates the principle of human attention, allowing the model to focus on key information in the input data. This mechanism can assign weights to the input data by calculating the importance of different parts, thereby more effectively capturing the key features of the data. Therefore, for eye tracking time series data, the attention mechanism is used to construct a network for training. Adding an LSTM module can use the implicit state in the LSTM to alleviate errors or noise in the past sequence and improve the robustness of the model.

[0215] The input of the eye movement track multi-dimensional attention enhancement module is the eye movement track time series data in the training data set, and the output is the category probability value corresponding to the eye movement track time series data.

[0216] The AOI multi-scale comprehensive feature module includes six sequentially connected first to sixth heuristic modules, a global average pooling layer, a third random inactivation layer and a third linear projection layer;

[0217] The first to sixth heuristic modules are used to perform multi-scale feature extraction on the AOI gaze time series data; wherein the AOI gaze time series data is connected to the output of the second heuristic module through a first residual; the output of the third heuristic module is connected to the output of the fifth heuristic module through a second residual;

[0218] The global average pooling layer is used to reduce the dimension of the output features of the first heuristic module and extract global features;

[0219] The third random deactivation layer is used to perform random deactivation processing on the extracted global features to enhance the generalization ability of the AOI multi-scale comprehensive feature module;

[0220] The third linear projection layer is used to reduce the dimension of the matching output dimension to obtain three category probability values ​​corresponding to the AOI fusion feature.

[0221] like Figure 6 As shown, the heuristic module is not a traditional convolutional layer.

[0222] The first to sixth heuristic modules each include a first component, a second component and an output layer, as follows:

[0223] The first component is the first bottleneck layer, which uses a filter with a length of 1 to reduce the dimension of the input multi-dimensional AOI gaze time series data;

[0224] The second component includes three parallel convolutions, a max pooling, and a second bottleneck layer;

[0225] The three parallel convolutions are used to perform convolution operations of three lengths on the output of the first bottleneck layer to capture temporal features of different scales;

[0226] The maximum pooling is used to perform a maximum pooling operation on the input multi-dimensional AOI gaze time series data;

[0227] The second bottleneck layer is used to reduce the dimension of the maximum pooling output;

[0228] The output layer is used to concatenate the time series features of different scales captured by the three parallel convolutions and the output features of the maximum pooling to form the final output multi-dimensional time series.

[0229] Exemplarily, the three convolution lengths are one-dimensional convolutions of different lengths, and the convolution kernels are 15, 25, and 50 respectively.

[0230] The first bottleneck layer performs dimensionality reduction, thereby reducing the complexity of the model and the risk of overfitting. The heuristic module is a structure that can simultaneously apply convolution kernels and pooling layers of different sizes, thereby improving the expressiveness and efficiency of the model.

[0231] The eye movement data category probability value output by the eye movement multi-dimensional attention enhancement module and the AOI gaze time series data category probability value output by the AOI multi-scale comprehensive feature module are fused through the fusion module to obtain the situation awareness level state, which is output through the situation awareness level output module.

[0232] The fusion module averages the probability values ​​of the three categories corresponding to the eye tracking time series data and the three category probability values ​​corresponding to the AOI fusion features to obtain an averaged probability value;

[0233] Comparing the averaged probability values ​​of the three categories to determine the category with the highest probability;

[0234] The category with the largest average probability value is taken as the pilot's situation awareness level.

[0235] Step 3: Use the cross entropy loss function to train the Ti-SA model using the training set until the loss function converges to obtain the trained Ti-SA model;

[0236] For the training feature data input into the Ti-SA model (including eye tracking time series data and AOI gaze time series data), Spearman correlation coefficient analysis was used, p < 0.05, which was significantly correlated, so the input feature data did not need to be screened again.

[0237] The loss function of the Ti-SA model training is the cross entropy function. Until the loss function converges, the trained Ti-SA model is obtained. The cross entropy function E is shown in formula (3):

[0238]

[0239] Among them, n is the number of category labels, y i and p i are the true label and predicted category probability value of the i-th category label respectively.

[0240] Until the cross entropy loss function converges, the trained Ti-SA model is obtained.

[0241] The fourth step is to use the validation set to validate the Ti-SA model and construct the objective function based on the accuracy of the validation set in the validation model and the cross entropy function.

[0242] The validation set in the training data set is used to verify the trained Ti-SA model, and the average of the accuracy and the inverse of the cross entropy loss function is used as the objective function.

[0243] Step 5. Select Gaussian process as the prior, randomly select a part of points in the hyperparameter space as initial points to construct the probability model of the objective function, and use the evaluation results of each point in the initial points to update the probability model; based on the evaluation results of the initial points, use Gaussian process to construct the posterior distribution of the objective function; the probability model of the objective function will be used to predict the performance at unevaluated points.

[0244] Step 6. Continue to select the next part of the hyperparameter space and repeat step 5 until the preset number of iterations or the preset performance threshold is reached.

[0245] Based on the current probability model and objective function, select the next part of the point combination in the hyperparameter space for evaluation; evaluate the objective function and update the objective function probability model.

[0246] After multiple iterations, a set of hyperparameter configurations is finally determined, which performs best in all evaluations. The best hyperparameter configuration found is used as the optimal hyperparameter configuration for the Ti-SA model.

[0247] Step 7. Based on the optimal hyperparameters, continue to train the Ti-SA model using the training data set until the loss function converges, obtain the weights and bias of the optimal model, and finally obtain the trained Ti-SA model.

[0248] Step S33, inputting the preprocessed eye movement track time series data into the eye movement track multi-dimensional attention enhancement module to obtain the category probability value corresponding to the eye movement track time series data;

[0249] Inputting the preprocessed AOI gaze time series data into the AOI multi-scale comprehensive feature module to obtain the category probability value corresponding to the AOI fusion feature;

[0250] The category probability value corresponding to the eye movement tracking time series data and the category probability value corresponding to the AOI fusion feature are input into the fusion module to obtain the predicted situation awareness level.

[0251] The pilot data of the eye tracking time series data and AOI gaze time series data obtained in the present invention specially select professional pilots as the subjects of this test group. Through simple random sampling, fighter pilots who meet the standards are randomly selected from the professional pilot group participating in flight training by lottery. Inclusion criteria: The pilots are in good physical condition, the Minnesota Multiphasic Personality Inventory (MMPI) meets the standards, they all have the necessary flying skills and experience, and the pilots' physical and cognitive abilities are normal during the test. All professional pilots participating in the experiment understand the test process and precautions, and voluntarily sign a written informed consent. A total of 30 pilots were included in the simulated flight process. They were in good physical and mental condition during the experiment. They were all male, aged 23-38 years old, with an average age of (25.15±4.52) years old, and had a flight time of 300-2200 hours, with an average flight time of 350.00 (340.75,912.50) hours. The performance comparison of the Ti-SA model is as follows:

[0252] For example, the Ti-SA model training is performed under the experimental conditions of Python 3.8 and the deep learning framework Pytorch 2.1.2. In order to evaluate the performance of the Ti-SA model, the accuracy, precision, recall, and F1-score are used for measurement, as shown in formulas (4)-(7):

[0253]

[0254] Wherein, Acc, Pre, Rec and F1 represent accuracy, precision, recall and F1 score respectively. TP represents the number of samples that are actually positive and correctly predicted as positive by the Ti-SA model. TN represents the number of samples that are actually negative and correctly predicted as negative by the Ti-SA model. FP represents the number of samples that are actually negative but incorrectly predicted as positive by the Ti-SA model. FN represents the number of samples that are actually positive but incorrectly predicted as negative by the Ti-SA model. The performance comparison of the Ti-SA model of the present invention and other multidimensional time series classification domain models is shown in Table 5:

[0255] Table 5: Performance comparison between each model and the proposed model

[0256]

[0257]

[0258] (1) MLSTM-FCN: Convert the long short-term memory fully convolutional network (LSTM-FCN) to a multivariate model and expand the fully convolutional block;

[0259] (2) Res-CNN: Integrates residual networks and convolutional neural networks for time series classification;

[0260] (3) MiniRocket: A fast implementation that uses random convolution kernels to transform input time series and uses the transformed features to train the model;

[0261] (4) TST: A multi-dimensional temporal learning framework based on Transformer encoder.

[0262] (5) The Ti-SA model proposed in this invention is a multi-dimensional situational awareness data learning model that integrates multi-head attention mechanism and heuristic module.

[0263] The performance of different models on the training dataset is shown in Table 4. The Ti-SA model achieved the best performance when used on the obtained dataset. The accuracy rate reached 92.18%, which is better than other commonly used models in the field of multidimensional time series classification.

[0264] Situational awareness affects the pilot's decision-making process and information acquisition, and is related to flight safety, efficiency and the smooth operation of the entire aviation system. Therefore, evaluating the pilot's situational awareness level has always been a focus of concern in the aviation industry. The present invention constructs a training data set under spatial directional flight simulation conditions and evaluates the pilot's situational awareness level through a deep learning method, and obtains a Ti-SA model with high accuracy.

[0265] During the flight simulation test, each pilot was randomly frozen in the simulation scene at least five times. The purpose of this is to simulate the emergencies and challenges in real flight. Freezing the simulation scene helps to assess their situational awareness level. Such comprehensive training can enhance the pilot's physiological adaptability and coping ability in flight missions, thereby improving flight safety and operational effectiveness.

[0266] The Ti-SA model combines the multi-head attention mechanism and the heuristic module to extract and learn the features of the eye tracking time series data and the AOI gaze time series data. Compared with the model that only learns unilateral features, the Ti-SA model of the present invention can comprehensively consider the features of the eye tracking time series data and the AOI gaze sequence to achieve a more comprehensive pilot situation awareness assessment.

[0267] The Ti-SA model uses a multi-head attention mechanism to capture long-term dependencies in eye movement time series data and mitigates errors or noise in past sequences through a Bi-LSTM model.

[0268] At the same time, the local short-term dependency features in the AOI gaze sequence are captured through the heuristic module to adapt to the situation where the pilot focuses on a specific AOI segment within a specific time.

[0269] The results in Table 5 show that the proposed situation awareness assessment method that integrates eye tracking time series data and AOI gaze sequence is effective and can accurately judge the pilot's situation awareness level. The model in this study has a comprehensive feature learning mechanism, robustness, and a specific design for pilot SA tasks, which has obvious advantages in pilot situation awareness assessment.

[0270] In summary, the pilot situation awareness ability assessment method based on the Ti-SA model according to the embodiment of the present invention has the following beneficial effects:

[0271] 1. The present invention collects the pilot's eye tracking time series data and AOI gaze time series data in real time, and uses the Ti-SA model to process and analyze them, thereby achieving real-time evaluation of the pilot's situational awareness ability. This method has higher real-time and accuracy than traditional methods based on questionnaires, expert observations or post-training evaluations, so it can be directly used to evaluate the actual performance of pilots during flight training;

[0272] 2. By combining eye tracking data and AOI gaze data, the present invention provides a more comprehensive assessment of pilot situational awareness, which does not rely on the subjective judgment of experts, but is based on objective data-driven analysis. The method of the present invention can more accurately capture the pilot's cognitive state of the flight environment, thereby improving the reliability and effectiveness of the assessment results;

[0273] 3. The evaluation method of the present invention can identify possible cognitive deficiencies or errors of pilots at an early stage, so as to take preventive measures and reduce flight accidents caused by insufficient situational awareness. By analyzing the situational awareness level during training, customized training suggestions can be provided to pilots to help them improve their flying skills and ability to deal with complex situations, thereby improving overall flight safety;

[0274] 4. The Ti-SA model in the present invention can adapt to the cognitive characteristics of different pilots and different flight stages, provide personalized evaluation results, and enhance the adaptability and flexibility of the evaluation method; based on real-time evaluation of the pilot's situational awareness ability, the pilot training system can adjust the training strategy in time, optimize the pilot's training effect, and enhance the pilot's ability to improve his flying skills and cope with complex flight situations.

[0275] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. A method for evaluating pilot situational awareness based on the Ti-SA model, characterized in that: The Ti-SA model includes an eye tracking multi-dimensional attention enhancement module, an AOI multi-scale comprehensive feature module, and a fusion module; the evaluation method includes the following steps: Step S1, collecting the pilot's eye movement tracking time series data and flight training video at consecutive moments in real time, and obtaining corresponding AOI gaze time series data based on the eye movement tracking time series data, the flight training video and the preset AOI subdivision area; Step S2, preprocessing the eye movement tracking time series data and the AOI gaze time series data; Step S3, inputting the preprocessed eye movement track time series data into the eye movement track multi-dimensional attention enhancement module to obtain the category probability value corresponding to the eye movement track time series data; Inputting the preprocessed AOI gaze time series data into the AOI multi-scale comprehensive feature module to obtain the category probability value corresponding to the AOI fusion feature; The category probability value corresponding to the eye tracking time series data and the category probability value corresponding to the AOI fusion feature are input into the fusion module to obtain a predicted situation awareness level; The eye tracking multi-dimensional attention enhancement module includes: A first linear projection layer, used to map the eye movement tracking time series data features to a high dimension; The first random deactivation layer is used to perform random deactivation processing on the mapped high-dimensional time series data to enhance the generalization of the eye movement trace multi-dimensional attention enhancement module and prevent overfitting; A first, a second and a third multi-layer attention structure connected in sequence, for obtaining an encoded representation of the eye tracking temporal data; Bi-LSTM, which uses the bidirectional capability of LSTM to enhance the understanding of the time series of the encoded representation output after three multi-layer attention structures; Adaptive average pooling layer to reduce the dimensionality of Bi-LSTM output features while retaining important information; A second linear projection layer is used to match the output dimension to obtain three category probability values ​​corresponding to the eye movement trace; Among them, the multi-head attention module in the three-layer attention structure divides the input features of the module into multiple attention heads. Each head processes the feature information independently and understands the features from different angles. Finally, the outputs of multiple attention heads are merged to obtain a comprehensive feature representation.

2. The method according to claim 1, characterized in that: Preprocessing the eye tracking time series data and the AOI gaze time series data includes: Unifying the timestamp formats of the eye tracking time series data and the AOI gaze time series data; Identify missing values ​​of the eye tracking time series data and the AOI gaze time series data, mark the missing values, and further perform Z-score standardization to obtain standardized data; The standardized data is input into an isolation forest model, an outlier score is generated for each data point using the isolation forest model, and based on a preset ratio, the data points with the highest score among all the outlier scores are taken as outliers to obtain outliers in the eye tracking time series data and the AOI gaze time series data; Use the KNN algorithm to fill the missing values ​​and outliers in the eye tracking time series data, and find the mean of the K neighbors of the missing values ​​and outliers to fill them; use the random forest algorithm to fill the missing values ​​and outliers in the AOI gaze time series data; The preprocessed eye tracking time series data and AOI gaze time series data are obtained.

3. The method according to claim 1, characterized in that: The Ti-SA model is trained by the following steps: The first step is to obtain the eye movement tracking time series data and AOI gaze time series data of multiple pilots during simulated flight as sample data, generate corresponding category labels as a training data set, and divide the training data set into a training set and a validation set; The second step is to determine the hyperparameter space and build the Ti-SA model; the hyperparameters include learning rate and training batch size; Step 3: Use the cross entropy loss function to train the Ti-SA model using the training set until the loss function converges to obtain the trained Ti-SA model; Step 4: Use the average of the accuracy of the validation set on the trained Ti-SA model and the inverse of the cross entropy loss function as the objective function; Step 5: Select a Gaussian process as a priori, randomly select a portion of points in the hyperparameter space as initial points to construct a probability model of the objective function, and use the evaluation result of each point in the initial points to update the probability model; based on the evaluation results of the initial points, use the Gaussian process to construct the posterior distribution of the objective function; Step 6: Continue to select the next part of points in the hyperparameter space and repeat step 5 until a preset number of iterations or a preset performance threshold is reached, and finally a set of hyperparameter configurations is obtained as the optimal hyperparameter configuration of the Ti-SA model; Step 7: Based on the optimal hyperparameter configuration, continue to train the Ti-SA model using the training data set until the loss function converges to obtain a trained Ti-SA model.

4. The method according to claim 1, characterized in that: The preset AOI subdivision area is determined in the following steps: Identify the types of instrument panels and displays in aircraft; Based on the instrument panel display area and the screen display area, the instrument panel and the screen display are subdivided into a plurality of logical AOI subdivision areas; the exterior scene is one AOI subdivision area; Clarify the scope and meaning of each AOI subdivision area to ensure that each AOI subdivision area covers the information that pilots need to pay attention to in specific situations; The AOI subdivision areas are numbered to obtain m AOI subdivision areas.

5. The method according to claim 3, characterized in that: Acquiring the training data set includes: Obtaining the eye movement tracking time series data and flight training video during the pilot's simulated flight; Detecting and obtaining all AOI subdivision areas in the flight training video in time sequence based on the flight training video; Based on the two-dimensional and three-dimensional coordinates of the gaze focus in the eye tracking time series data, determine whether the coordinates are located in a certain AOI subdivision area, and record the AOI subdivision area number of the pilot's gaze in chronological order according to the matching result to obtain the AOI gaze time series data; The eye tracking time series data and the AOI gaze time series data are used as sample data; Execute the following process to generate the category label corresponding to the sample data, as follows: Based on the situational awareness assessment items, measurement indicators, recovery results, planned results, and the full score of the situational awareness assessment items, the assessment result Score of each pilot's situational awareness assessment item is calculated as follows: Where n is the total number of indicators of situational awareness assessment items, D i M is the deviation between the pilot's recovery result and the expected result. i is the maximum deviation value in the indicator, fullscore i It is the full score obtained by the pilot when the recovery result of the indicator corresponding to the situational awareness assessment item is completely consistent with the expected result; The level corresponding to the evaluation result is used as the corresponding category label of the eye tracking time series data and AOI gaze time series data sample data obtained during the simulated flight; The sample data and the obtained corresponding category labels constitute a training data set.

6. The method according to claim 1, characterized in that: The AOI multi-scale comprehensive feature module includes six sequentially connected first to sixth heuristic modules, a global average pooling layer, a third random inactivation layer and a third linear projection layer; The first to sixth heuristic modules are used to perform multi-scale feature extraction on the AOI gaze time series data; wherein the AOI gaze time series data is connected to the output of the second heuristic module through a first residual; the output of the third heuristic module is connected to the output of the fifth heuristic module through a second residual; The global average pooling layer is used to reduce the dimension of the output features of the sixth heuristic module and extract global features; The third random deactivation layer is used to perform random deactivation processing on the extracted global features to enhance the generalization ability of the AOI multi-scale comprehensive feature module; The third linear projection layer is used to reduce the dimension of the matching output dimension to obtain three category probability values ​​corresponding to the AOI fusion feature.

7. The method according to claim 6, characterized in that: The first to sixth heuristic modules each include a first component, a second component and an output layer, as follows: The first component is the first bottleneck layer, which uses a filter with a length of 1 to reduce the dimension of the input multi-dimensional AOI gaze time series data; The second component includes three parallel convolutions, a max pooling, and a second bottleneck layer; The three parallel convolutions are used to perform convolution operations of three lengths on the output of the first bottleneck layer to capture temporal features of different scales; The maximum pooling is used to perform a maximum pooling operation on the input multi-dimensional AOI gaze time series data; The second bottleneck layer is used to reduce the dimension of the maximum pooling output; The output layer is used to concatenate the time series features of different scales captured by the three parallel convolutions and the output features of the maximum pooling to form the final output multi-dimensional time series.

8. The method according to claim 1, characterized in that: The fusion module averages the probability values ​​of the three categories corresponding to the eye tracking time series data and the three category probability values ​​corresponding to the AOI fusion features to obtain an averaged probability value; Comparing the averaged probability values ​​of the three categories to determine the category with the highest probability; The category with the largest average probability value is taken as the pilot's situation awareness level.

9. The method according to any one of claims 1 to 8, characterized in that: The eye movement tracking time series data includes the two-dimensional coordinates of the gaze focus, the three-dimensional coordinates of the gaze focus, the three-dimensional coordinates of the original position of the left eye, the three-dimensional coordinates of the moving direction of the left eye, the pupil diameter of the left eye, the three-dimensional coordinates of the original position of the right eye, the three-dimensional coordinates of the moving direction of the right eye and the pupil diameter of the right eye; The AOI gaze timing data includes a number sequence of AOI subdivision areas that the pilot gazes at in sequence based on time during flight; the AOI subdivision areas include A sub-areas divided based on the instrument panel display area and the screen display area, and an exterior view area.

Images