Pilot SA ability assessment system based on eye tracking and spatial orientation flight simulation

Abstract

The invention relates to a pilot SA ability evaluation system based on eye tracking and spatial directional flight simulation, comprising a spatial directional flight simulator for simulating a real flight environment; an eye tracker for recording the eye movement trajectory of a pilot in real time in the simulated real flight environment to obtain eye tracking time series data and shoot a flight training video; a microprocessor evaluation module comprising a time series data processing module and a Ti-SA model; the time series data processing module obtains corresponding AOI gaze time series data based on the eye tracking time series data, the flight training video and a preset AOI subdivision area; the Ti-SA model predicts the pilot's current SA ability value based on the eye tracking time series data and the AOI gaze time series data, and determines whether the predicted SA ability value is lower than a preset SA threshold value, and if so, sends an early warning instruction to an early warning and voice broadcast module; and the early warning and voice broadcast module is used to issue an early warning when the SA ability value is lower than the preset SA threshold value.

Description

Technical Field

[0001] The present invention relates to the field of aviation safety and situational awareness technology, and in particular to a pilot SA capability assessment system based on eye tracking and spatial orientation flight simulation. Background Art

[0002] Aviation technology has made great progress, and the performance, functions and complexity of aircraft have been significantly improved. However, flight safety is still an urgent problem to be solved. SA (Situation Awareness) is one of the key factors for pilots to fly safely in the aviation field. The pilot's situational awareness (SA) ability is the pilot's attitude perception and motion perception of time, space and direction in a high-altitude closed cockpit environment, which is an important factor affecting the pilot's flight decision-making and flight safety. For a long time, the pilot's SA ability has been mainly evaluated through subjective evaluation and post-flight behavior analysis, which lacks real-time and accuracy. The higher the situational awareness ability, the more the pilot can grasp the flight situation in a timely, accurate and comprehensive manner, so as to make better flight decisions and operations. On the contrary, low or lost situational awareness will lead to cognitive bias, misjudgment or ignorance of the pilot's flight situation, thereby increasing the risk of flight accidents. The pilot's situational awareness ability is crucial to flight safety. Studies have shown that 15%-20% of accidents are related to insufficient pilot situational awareness. Spatial disorientation (SD) is closely related to situational awareness (SA). Pilots need to maintain good spatial orientation ability to ensure flight safety. SD is the inability or failure of orientation judgment relative to the coordinate system formed by the ground and the vertical line of gravity. SD is closely related to SA ability. Specifically, the loss of SA ability by pilots is inevitably accompanied by SD. Therefore, it is very important to build a system that can accurately and objectively evaluate SA ability in real time during the flight process.

[0003] Traditional methods for evaluating pilot SA capabilities mainly rely on questionnaires, manual observations in flight simulators, and post-flight data analysis. These methods are obviously subjective and time-delayed, rely on the subjective judgment of experts, cannot achieve real-time evaluation, and are difficult to quantify pilot SA capabilities. With the rapid development of aviation technology, the need for real-time, accurate, and objective evaluation of pilot SA capabilities is becoming increasingly urgent. Summary of the invention

[0004] In view of the above analysis, an embodiment of the present invention aims to provide a pilot SA ability assessment system based on eye tracking and spatial orientation flight simulation, so as to solve the technical problem that the existing method cannot perform real-time and accurate assessment of the pilot's situational awareness ability.

[0005] The present invention provides a pilot SA ability assessment system based on eye tracking and spatial orientation flight simulation, comprising the following steps:

[0006] Space directional flight simulator, used to simulate real flight environment;

[0007] An eye tracker is used to record the pilot's eye movement trajectory in real time in the simulated real flight environment to obtain eye movement track time series data, and to shoot a flight training video;

[0008] A microprocessor evaluation module includes a timing data processing unit and a Ti-SA model; the timing data processing unit is used to obtain corresponding AOI gaze timing data based on the eye movement tracking timing data, the flight training video and the preset AOI subdivision area; the Ti-SA model is used to predict the pilot's current SA ability value based on the eye movement tracking timing data and the AOI gaze timing data, and to determine whether the predicted pilot's current SA ability value is lower than a preset SA threshold value, and if so, send a warning instruction to the warning and voice broadcast module;

[0009] The early warning and voice broadcast module is used to issue an early warning when the SA capability value is lower than the preset SA threshold.

[0010] Furthermore, the system further comprises:

[0011] A data storage module, used to store the eye tracking time series data, flight training videos and AOI gaze time series data, corresponding SA ability levels, data collection logs and warning logs;

[0012] The visualization module is used to display the eye tracking data and AOI gaze timing data in real time, as well as the corresponding SA capability value; and provide an editing interface for the preset AOI subdivision area and the preset SA threshold.

[0013] Further, the timing data processing unit performs the following steps to obtain the corresponding AOI gaze timing data, as follows:

[0014] Detect and obtain all AOI subdivision areas in the flight training video that are arranged in time sequence;

[0015] Based on the two-dimensional and three-dimensional coordinates of the gaze focus in the eye tracking time series data, determine whether the coordinates are located in a certain AOI subdivision area, and record the AOI subdivision area number of the pilot's gaze in chronological order according to the matching result to obtain the AOI gaze time series data;

[0016] The steps for presetting the AOI subdivision area are as follows:

[0017] Identify the types of instrument panels and displays in a space-oriented flight simulator:

[0018] Based on the instrument panel display area and the screen display area, the instrument panel and the screen display are subdivided into a plurality of logical AOI subdivision areas; the exterior scene is one AOI subdivision area;

[0019] The AOI subdivision areas are numbered to obtain k AOI subdivision areas.

[0020] Furthermore, the Ti-SA model includes an eye tracking multi-dimensional attention enhancement module, an AOI multi-scale comprehensive feature module, a fusion module and a SA level output module;

[0021] The eye movement trace multi-dimensional attention enhancement module is used to receive the eye movement trace time series data, perform multi-dimensional attention feature extraction, and output the category probability value corresponding to the eye movement trace time series data;

[0022] The AOI multi-scale comprehensive feature module is used to receive the AOI gaze time series data, perform multi-scale feature fusion, and output the category probability value corresponding to the AOI fusion feature;

[0023] The fusion module fuses the eye tracking depth feature and the AOI fusion feature to obtain the pilot's current SA capability value;

[0024] The SA level output module is used to output the pilot's current SA capability value.

[0025] Furthermore, the eye movement tracking multi-dimensional attention enhancement module includes:

[0026] A first linear projection layer, used to map the eye tracking time series data features to a high dimension;

[0027] The first random dropout layer is used to perform random dropout processing on the mapped high-dimensional time series data;

[0028] A first, a second and a third multi-layer attention structure connected in sequence, for obtaining an encoded representation of the eye tracking temporal data;

[0029] Bi-LSTM, which uses the bidirectional capability of LSTM to enhance the understanding of the time series of the encoded representation output after three multi-layer attention structures;

[0030] Adaptive average pooling layer to reduce the dimensionality of Bi-LSTM output features while retaining important information;

[0031] The second linear projection layer is used to match the output dimension and obtain three category probability values ​​corresponding to the eye movement trace.

[0032] Further, the AOI multi-scale comprehensive feature module includes six sequentially connected first to sixth heuristic modules, a global average pooling layer, a third random inactivation layer and a third linear projection layer;

[0033] The first to sixth heuristic modules are used to perform multi-scale feature extraction on the AOI gaze time series data; wherein the AOI gaze time series data is connected to the output of the second heuristic module through a first residual; the output of the third heuristic module is connected to the output of the fifth heuristic module through a second residual;

[0034] The global average pooling layer is used to reduce the dimension of the output features of the sixth heuristic module and extract global features;

[0035] The third random deactivation layer is used to perform random deactivation processing on the extracted global features to enhance the generalization ability of the AOI multi-scale comprehensive feature module;

[0036] The third linear projection layer is used to reduce the dimension of the matching output dimension to obtain three category probability values ​​corresponding to the AOI fusion feature.

[0037] Furthermore, the first to sixth heuristic modules each include a first component, a second component and an output layer, wherein:

[0038] The first component is the first bottleneck layer, which uses a filter with a length of 1 to reduce the dimension of the input multi-dimensional AOI gaze time series data;

[0039] The second component includes three parallel convolutions, a max pooling, and a second bottleneck layer;

[0040] The three parallel convolutions are used to perform convolution operations of three different lengths on the output of the first bottleneck layer respectively, so as to capture temporal features of different scales;

[0041] The maximum pooling is used to perform a maximum pooling operation on the input multi-dimensional AOI gaze time series data;

[0042] The second bottleneck layer is used to reduce the dimension of the maximum pooling output;

[0043] The output layer is used to concatenate the time series features of different scales captured by the three parallel convolutions and the output features of the maximum pooling to form a final multi-dimensional time series output.

[0044] Further, the fusion module averages the probability values ​​of the three categories corresponding to the eye tracking time series data and the three category probability values ​​corresponding to the AOI fusion feature to obtain an averaged probability value;

[0045] Comparing the averaged probability values ​​of the three categories to determine the category with the highest probability;

[0046] The SA value corresponding to the category with the largest average probability value is used as the pilot's current SA capability value.

[0047] Further, the eye movement tracking time series data includes two-dimensional coordinates of gaze focus, three-dimensional coordinates of gaze focus, three-dimensional coordinates of left eye original position, three-dimensional coordinates of left eye moving direction, left eye pupil diameter, three-dimensional coordinates of right eye original position, three-dimensional coordinates of right eye moving direction and right eye pupil diameter;

[0048] The AOI gaze timing data includes a number sequence of AOI subdivision areas that the pilot gazes at in sequence based on time during flight; the AOI subdivision areas include A sub-areas divided based on instrument display areas and screen display areas, and exterior scenery areas.

[0049] Furthermore, in the real flight environment simulated by the spatial orientation simulator, pilot training is performed based on preset SA assessment items to obtain corresponding eye tracking time series data and AOI gaze time series data;

[0050] Among them, the SA assessment items include pitch illusion, Coriolis illusion, tilt illusion, false horizon illusion, runway width and depth illusion and black hole approach illusion.

[0051] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:

[0052] 1. The present invention obtains eye movement tracking time series data by real-time recording of the eye movement trajectory of the pilot in the space orientation flight simulator during simulated flight. The eye movement tracking time series data is combined with the AOI (Area of ​​Interest) gaze time series data obtained in a simulated real flight environment to provide instant and accurate SA ability assessment, which significantly improves the timeliness and accuracy of the assessment;

[0053] 2. The system of the present invention has an early warning function. When the pilot's SA ability is lower than the safety threshold, the early warning and voice broadcast module can issue a real-time early warning, effectively preventing flight risks caused by insufficient situational awareness, thereby enhancing overall flight safety;

[0054] 3. The present invention utilizes eye tracking time series data and AOI gaze time series data, and realizes quantitative analysis of pilot situational awareness through the Ti-SA evaluation model built into the microprocessor, thereby reducing the interference of human subjective factors, and the data-driven objective quantification improves the objectivity of the evaluation;

[0055] 4. The present invention integrates a spatial orientation flight simulator, eye tracking and artificial intelligence deep learning. The present invention represents an innovative application of technology integration in the aviation field and provides a new scientific tool for pilot training and evaluation.

[0056] 5. The present invention uses a visualization module to personalize the AOI and preset SA thresholds, and conducts evaluation based on multi-dimensional features. The system can provide customized evaluation results for each pilot;

[0057] 6. The data storage module in the present invention can be local storage or cloud storage according to the specific needs of pilot training, providing data support for pilot flight training and decision-making.

[0058] In the present invention, the above-mentioned technical solutions can also be combined with each other to achieve more preferred combination solutions. Other features and advantages of the present invention will be described in the subsequent description, and some advantages can become obvious from the description, or can be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained through the contents particularly pointed out in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The drawings are only for the purpose of illustrating particular embodiments and are not to be considered limiting of the present invention. Like reference symbols denote like components throughout the drawings.

[0060] Figure 1 Schematic diagram of a pilot SA ability assessment system based on eye tracking and spatial orientation flight simulation in an embodiment of the present invention;

[0061] Figure 2 The following is a flowchart of the construction and training of the Ti-SA model in an embodiment of the present invention;

[0062] Figure 3 Schematic diagram of the overall structure of the Ti-SA model in an embodiment of the present invention;

[0063] Figure 4 is a flow chart of a method for acquiring eye movement tracking time series data and AOI gaze time series data in an embodiment of the present invention;

[0064] Figure 5 This is a schematic diagram of the pilot's eye movement traces in a spatial directional flight simulation state in an embodiment of the present invention - a schematic diagram of the pilot's gaze focus above the runway;

[0065] Figure 6 This is a schematic diagram of the pilot's eye movement traces in a spatial directional flight simulation state in an embodiment of the present invention - a schematic diagram of the pilot's gaze focus on the instrument panel area;

[0066] Figure 7Schematic diagram of AOI area division of real scene in the cabin in an embodiment of the present invention;

[0067] Figure 8 Schematic diagram of the heuristic module structure in an embodiment of the present invention. DETAILED DESCRIPTION

[0068] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.

[0069] In view of the problems of the current existing technology, the present invention constructs a pilot SA ability assessment system based on eye tracking and spatial directional flight simulation, which is used for pilot SA assessment in spatial directional flight simulation flight state, as a preliminary exploration in this field.

[0070] A specific embodiment of the present invention discloses a pilot SA ability assessment system based on eye tracking and spatial orientation flight simulation, such as Figure 1 As shown, including:

[0071] Space directional flight simulator, used to simulate real flight environment;

[0072] An eye tracker is used to record the pilot's eye movement trajectory in real time in the simulated real flight environment to obtain eye movement track time series data, and to shoot a flight training video;

[0073] A microprocessor evaluation module includes a timing data processing module and a Ti-SA model; the timing data processing module is used to obtain corresponding AOI gaze timing data based on the eye movement tracking timing data, the flight training video and the preset AOI subdivision area; the Ti-SA model is used to predict the pilot's current SA ability value based on the eye movement tracking timing data and the AOI gaze timing data, and to determine whether the predicted pilot's current SA ability value is lower than a preset SA threshold value, and if so, send a warning instruction to the warning and voice broadcast module;

[0074] The early warning and voice broadcast module is used to issue an early warning when the SA capability value is lower than the preset SA threshold.

[0075] The equipment involved in the present invention is as follows:

[0076] (1) Space Orienteering Flight Simulator: The DISO space orientation flight simulator produced by AMST of Austria is used. It uses a three-dimensional 6-DOF hydraulic platform and a highly simulated cockpit vision system. It can simulate more than 10 kinds of vestibular and visual illusions to intuitively experience the various SDs generated by the movement of the human body's orientation organs when facing an abnormal three-dimensional space environment.

[0077] The spatial orientation flight simulator is the main component of the pilot's simulated flight. It simulates the driving of the aircraft cabin, which is equivalent to the role of the aircraft. Based on this simulator, the pilot can use the eye tracker for simulation training, collect eye movement trajectory data and shoot flight training videos;

[0078] The space directional flight simulator is mainly used for training and solving flight illusion problems. It is an advanced simulation device that can simulate the real flight environment of aircraft and air, and simulate 16 major flight illusion environments such as Coriolis illusion, tilt illusion, black hole approach illusion, etc.

[0079] (2) Eye tracker: A wearable eye tracker, Tobii Pro Glasses 3 produced by Swedish company Tobii, with a sampling rate of 100 Hz, integrates 16 light sources and 4 eye tracking sensors, embedded in scratch-resistant lenses. The eye tracker has a 106° wide-angle scene camera that can record behavioral details from a first-person perspective.

[0080] The eye tracker records the pilot's eye movement trajectory during simulated flight in real time while shooting flight training videos.

[0081] Flight training videos are visual records of pilots during simulated flight. When combined with eye tracking data recorded by an eye tracker, AOI gaze timing data can be obtained based on flight training videos. In addition, flight training data has strong practical significance, such as flight training scenario reconstruction analysis, pilot training behavior analysis, flight simulation environment verification, and case teaching materials to guide new pilots or conduct case studies to improve the design of flight simulators.

[0082] The microprocessor evaluation module is the core component of the system in the present invention, which is responsible for processing and analyzing the eye tracking time series data and the flight training video. The time series data processing unit in the module is specifically based on the eye tracking time series data, and matches it with the preset AOI subdivision area preset in the flight simulator to determine the pilot's visual attention allocation and obtain the AOI gaze time series data. The microprocessor evaluation module uses the Ti-SA model to analyze the eye tracking data and the AOI gaze time series data, so as to evaluate the pilot's SA ability.

[0083] The early warning and voice broadcast module has two functions: early warning and voice broadcast. When the microprocessor evaluation module predicts that the pilot's current SA ability value is lower than the preset SA threshold, a warning instruction is sent to the module to issue an early warning and voice broadcast to attract the attention of the pilot and auxiliary simulation experiment personnel, stop the simulated flight training or change the training strategy.

[0084] The test conditions involved in the present invention are as follows: the test sequentially conducts the following simulated flight links: at an altitude of 914m and a speed of 400 knots, the pitch illusion is experienced at an elevation angle of 10°, the Coriolis illusion is experienced, and the tilt illusion is experienced at a slope of 45°; at a speed of 400 knots, the cloud base is 914m, and the cloud top is 3657m to experience the false horizon illusion; at an altitude of 1219m, a speed of 250 knots, and a heading of 163°, the black hole approach illusion is experienced.

[0085] The system further comprises:

[0086] A data storage module, used to store the eye tracking time series data, flight training videos and AOI gaze time series data, corresponding SA ability levels, data collection logs and warning logs;

[0087] The visualization module is used to display the eye tracking data and AOI gaze timing data in real time, as well as the corresponding SA capability value; and provide an editing interface for the preset AOI subdivision area and the preset SA threshold.

[0088] Exemplarily, the data storage module can be local storage or cloud storage, which is determined according to the actual situation of the simulated flight training.

[0089] The visualization module provides the pilot with a UI (User Interface) interface, which displays the real-time data acquired in the system and the SA capability value predicted by the microprocessor evaluation module corresponding to the real-time acquired data; and provides an editing (including adding, updating and deleting operations) interface for the preset AOI subdivision areas and preset SA thresholds of the instrument category based on the space directional flight simulator.

[0090] The timing data processing unit performs the following steps to obtain the corresponding AOI gaze timing data, as follows:

[0091] Detect and obtain all AOI subdivision areas in the flight training video that are arranged in time sequence;

[0092] Based on the two-dimensional and three-dimensional coordinates of the gaze focus in the eye tracking time series data, determine whether the coordinates are located in a certain AOI subdivision area, and record the AOI subdivision area number of the pilot's gaze in chronological order according to the matching result to obtain the AOI gaze time series data;

[0093] The steps for presetting the AOI subdivision area are as follows:

[0094] Identify the types of instrument panels and displays in a space-oriented flight simulator:

[0095] Based on the instrument panel display area and the screen display area, the instrument panel and the screen display are subdivided into a plurality of logical AOI subdivision areas; the exterior scene is one AOI subdivision area;

[0096] The AOI subdivision areas are numbered to obtain k AOI subdivision areas.

[0097] By analyzing the AOI subdivision areas and eye movement tracking timing data in the flight training video, the pilot's visual attention allocation during the simulated flight can be accurately tracked and recorded, thereby achieving real-time evaluation and analysis of the pilot's situational awareness ability.

[0098] Training the Ti-SA model includes:

[0099] Step S1, obtaining the eye tracking time series data and AOI gaze time series data of the pilot during the simulated flight as sample data, wherein the sample data and the generated corresponding category labels constitute a training data set; wherein the category label is an SA ability level label;

[0100] Step S2: Use the cross entropy loss function to train the Ti-SA model using the training data set until the loss function converges to obtain a trained Ti-SA model.

[0101] Train the Ti-SA model, such as Figure 2 As shown, steps S1-S2 are included.

[0102] Step S1 includes steps S11-S13.

[0103] Step S11, obtaining the eye movement tracking time series data and AOI gaze time series data of each pilot during the simulated flight as sample data. Figure 4 shown.

[0104] The eye movement tracking time series data includes the two-dimensional coordinates of the gaze focus, the three-dimensional coordinates of the gaze focus, the three-dimensional coordinates of the original position of the left eye, the three-dimensional coordinates of the moving direction of the left eye, the pupil diameter of the left eye, the three-dimensional coordinates of the original position of the right eye, the three-dimensional coordinates of the moving direction of the right eye and the pupil diameter of the right eye;

[0105] The AOI gaze timing data includes a number sequence of AOI subdivision areas that the pilot gazes at in sequence based on time during flight; the AOI subdivision areas include A sub-areas divided based on instrument display areas and screen display areas, and exterior scenery areas.

[0106] The eye tracking time series data and AOI gaze time series data of the pilot during the simulated flight are obtained as sample data, including:

[0107] Acquiring the eye movement tracking time series data and the flight training video;

[0108] Based on the flight experiment video detection, all AOI subdivision areas in the flight training video in time sequence are obtained;

[0109] Based on the two-dimensional and three-dimensional coordinates of the gaze focus in the eye tracking time series data, it is determined whether the coordinates are located in a certain AOI subdivision area. According to the matching results, the AOI subdivision area numbers of the pilot's gaze are recorded in chronological order to obtain the AOI gaze time series data.

[0110] (1) Obtain eye movement tracking time series data and flight training videos.

[0111] The eye movement trajectory of the pilot is obtained through the spatial orientation flight simulator and the wearable eye tracker to obtain the eye movement trace time series data, and the flight training video is shot. When the pilot's eyes look at the spatial orientation flight simulator screen through the eye tracker, the wearable eye tracker records his eye movement trajectory.

[0112] The basic working principle of the eye tracker: First, use infrared or other light sources to illuminate the eyes to produce two bright spots: corneal reflection and pupil reflection. Then use a camera or other image sensor integrated in the eye tracker to capture the image of the eye and extract the position information of the corneal reflection and pupil reflection. Then use advanced image processing algorithms and 3D eyeball models to calculate the direction of the eye and the coordinate position of the gaze point relative to the screen display. Finally, there is a calibration process to map the eye's gaze point coordinates to the visual gaze coordinates on the screen display, thereby obtaining the eye movement tracking time series data and flight training video of the test pilot.

[0113] Figure 5-Figure 6 The white circular outline area is the pilot's visual focus, and the red curve is the historical trajectory of the pilot's previous gaze movement. Figure 5 to Figure 6 It shows that the pilot's gaze focus shifts from the runway to the instrument panel. The characteristics of the acquired eye tracking time series data are shown in Table 1.

[0114] Table 1: Eye tracking time series data characteristics

[0115]

[0116]

[0117] The two-dimensional coordinates of visual gaze are split into two-dimensional features of X and Y coordinates, and the three-dimensional coordinates are split into three-dimensional features of X, Y and Z coordinates.

[0118] The eye movement coordinate position is an important feature that reflects the pilot's visual information. In addition, the pupil diameter feature also plays a pivotal role. The pilot's left and right pupil diameter feature is an indicator that reflects the pilot's physiological and psychological state. The pilot's pupil diameter will change with factors such as lighting, visual stimulation, cognitive load, emotions, and fatigue. Generally speaking, the larger the pupil diameter, the higher the pilot's cognitive load; conversely, the smaller the pupil diameter, the lower the pilot's cognitive load. In addition, the difference in the pilot's left and right pupil diameters can also reflect the pilot's spatial orientation ability. If the difference in the left and right pupil diameters is large, it means that the pilot's spatial orientation ability is poor and is prone to spatial orientation disorders.

[0119] (2) Obtaining AOI gaze sequence data

[0120] AOI is an important source of information for pilots to acquire and maintain situational awareness. Pilots form a comprehensive understanding and prediction of current and future situations by observing and analyzing information on aircraft instruments, radar, navigation, communication and other equipment, as well as external weather, terrain and other information. Therefore, pilots need to reasonably allocate and adjust their attention to AOI according to mission requirements and environmental changes. Studies have shown that the longer the time spent paying attention to the AOI subdivision area, the more attention they pay to SA information.

[0121] There are three main areas that the pilot observes during flight: in the figure, number 1 is the instrument panel, number 2 is the screen display, and number 3 is the exterior view. For example, in the present invention, the instrument panel, screen display, and exterior view are divided into 13 AOI subdivision areas, such as Figure 7 And as shown in Table 2.

[0122] The main basis for the division of AOI subdivision areas is the type of instrument. Pilots will use and pay attention to different instruments at different stages. The specific meanings of AOI subdivision areas are shown in Table 2.

[0123] Table 2: Meaning of AOI subdivision areas in the real scene inside the space directional flight simulator and the outdoor location

[0124]

[0125] Among them, the steps for determining the AOI subdivision area are as follows:

[0126] Step 1: Identify the types of instrument panels and screen displays in the flight simulator:

[0127] Step 2: Based on the instrument panel function module and the screen information display area, the instrument panel and the screen display are subdivided into a plurality of logical AOI subdivision areas; the exterior scene is one AOI subdivision area;

[0128] Step 3: Clarify the scope and meaning of each AOI segment to ensure that each AOI segment covers the information that pilots need to pay attention to in a specific scenario.

[0129] Step 4: number the AOI subdivision areas to obtain m AOI subdivision areas.

[0130] Exemplarily, based on the instrument categories of the space-oriented simulated aircraft in the present invention, 13 AOI subdivision areas are determined.

[0131] Among them, the exterior scene is counted as one AOI subdivision area.

[0132] Since the 6-DOF spatial orientation flight simulator has a non-fixed viewing angle, direct Figure 6 Transforming to region templates for matching does not scale well to all viewpoints.

[0133] If you directly Figure 7 The displayed flight simulator instrument panel is treated as a regional template and then matched with the pilot's eye movement data, resulting in the following problems:

[0134] When the pilot's view rotates or tilts, Figure 7 The regional template also needs to be rotated or tilted accordingly to align with the pilot's eye movement data, which increases the complexity and error of the calculation; when the pilot's perspective is translated or scaled, Figure 7 The area template also needs to be translated or scaled accordingly to align with the pilot's eye movement data, which will cause the position and shape of the AOI to change, thereby affecting the accuracy of AOI recognition and matching; when the pilot's perspective is blocked or distorted, Figure 7 The area template may not completely cover the pilot's eye movement data, or may be inconsistent with the pilot's eye movement data, which will result in loss or error of AOI.

[0135] In order to solve the above problems, the present invention uses a method of AOI recognition and matching based on deep learning to train the YOLOv8n model to recognize the AOI area, as follows:

[0136] The first step is to use the lightweight YOLOv8n model, input the image data of 13 AOI subdivision areas for training, and obtain a neural network model that can detect AOI. The YOLOv8n model is an improved target detection model based on the YOLO series. YOLOv8n replaces the C3 structure of YOLOv8 with a C2f structure with richer gradient flow, and adjusts the number of channels for models of different scales, thereby improving the performance and efficiency of the model; uses a new detection head to separate the classification and regression branches, and changes from Anchor-Based to Anchor-Free, thereby improving the accuracy and robustness of detection; uses a new loss function to balance the contributions of different targets, thereby improving the stability and generalization of detection;

[0137] The second step is to input the flight training video recorded by the eye tracker into the trained network for inference, which can detect the AOI subdivision area in the video, as well as the position and shape of the AOI subdivision area. No matter how the pilot's perspective changes, the pilot's visual gaze area can be identified in real time, thus adapting to all perspectives;

[0138] The third step is to use the focus coordinates in the eye tracking time series data to calculate whether it is located in a certain AOI subdivision area, and then get the AOI fixation sequence. The AOI fixation sequence refers to the number of AOIs that the pilot fixates on in chronological order during the flight, which can reflect the pilot's visual attention and cognitive process, as well as the pilot's situational awareness level.

[0139] Since the AOI area features are relatively simple, we use the lightweight YOLOv8n model to input 13 ( Figure 7 The AOI subdivision areas are trained, and then the flight training test video recorded by the eye tracker is input into the trained network for inference, so that the AOI subdivision areas can be detected. The AOI gaze sequence data can be obtained by calculating whether the focus coordinates in the eye movement tracking time series data are located in a certain AOI subdivision area.

[0140] Step S12: Generate category labels for the eye movement tracking time series data and the AOI gaze time series data.

[0141] In the real flight environment simulated by the spatial orientation simulator, pilot training is carried out based on the preset SA assessment items to obtain the corresponding eye tracking time series data and AOI gaze time series data;

[0142] The SA evaluation items include pitch illusion, Coriolis illusion, tilt illusion, false horizon illusion, runway width and depth illusion, and black hole approach illusion, as described in Table 3.

[0143] Table 3: SA assessment items

[0144]

[0145] If the correction result of the indicator corresponding to each SA assessment item is completely consistent with the expected result, the full score can be obtained. The specific SA assessment item and the corresponding correction success full score fullscore i Definition, based on the specific needs of each assessment.

[0146] Execute the following process to generate the corresponding category labels:

[0147] Based on the SA assessment items, measurement indicators, recovery results, expected results and the full score of the SA assessment items, the evaluation result Score of each pilot's SA assessment item is calculated, as shown in formula (1):

[0148]

[0149] Where n is the total number of indicators of SA evaluation items, D i M is the deviation between the pilot's recovery result and the expected result. i is the maximum deviation value in the indicator, fullscore i It is the full score obtained by the pilot when the recovery result of the indicator corresponding to the SA assessment item is completely consistent with the expected result;

[0150] The level corresponding to the evaluation result is used as the corresponding category label of the eye tracking time series data and AOI gaze time series data obtained during the flight simulation.

[0151] For each metric of the SA assessment item, illustratively, as shown in Table 4, for a pitch illusion dive situation, the pilot may recover successfully, or fail to recover, or be in the intermediate stage between successful recovery and failure to recover.

[0152] Table 4: Complexity of SA Assessment Items

[0153]

[0154] The eye tracking time series data and AOI gaze sequence data in the sample data are both 1 matrix, that is, the data itself is a three-dimensional tensor of multi-dimensional time series data, which is expanded and spliced ​​into two dimensions according to the samples. Each sample has only one category label.

[0155] During the 60-minute training course, the pilots were given the background knowledge of the SA problem and were instructed by the test personnel on how to use the space orientation flight simulator to complete the SA assessment project. Each pilot had a 30-minute space orientation flight simulator operation experience to ensure that the pilots could successfully operate the space orientation flight simulator to collect data.

[0156] During flight, pilots face various complex situations. Complex situation recovery for pilots refers to the process of the pilot restoring the aircraft from an abnormal flight state to a normal controllable state, which is an important part of pilot training.

[0157] The level corresponding to the scoring result is as shown in formula (2):

[0158]

[0159] The level corresponding to the scoring result is used as the corresponding category label for the eye tracking time series data and AOI gaze time series data obtained during the flight simulation. If the scoring result is less than 60 points, it is considered a simulated flight failure, and the sample is not counted. The spatial orientation flight simulation test is repeated.

[0160] If the microprocessor evaluation module determines that the predicted pilot's current SA ability value is less than 60 points, it is regarded as a failure of the SA evaluation item of the simulated flight, and a warning instruction is sent to the warning and voice broadcast module; the warning and voice broadcast module performs voice broadcast based on the evaluated SA item.

[0161] For example, the voice broadcast module states, “Pilot No. **’s score for ** indicator of ** project is ** points, which is lower than ** points, so the simulation training is terminated.”

[0162] Among them, the score thresholds of 100, 90, 75, and 60 are all set based on the editing interface of the visualization module; other SA thresholds can be preset according to specific needs.

[0163] Step S2 includes steps S21-S22.

[0164] Step S21, constructing a Ti-SA model for pilot situational awareness assessment;

[0165] like Figure 3 As shown in the overall structure diagram of the Ti-SA model, the Ti-SA model for pilot situational awareness assessment includes an eye tracking multi-dimensional attention enhancement module, an AOI multi-scale comprehensive feature module, a fusion module and a SA level output module;

[0166] The eye movement trace multi-dimensional attention enhancement module is used to receive the eye movement trace time series data, perform multi-dimensional attention feature extraction, and output the category probability value corresponding to the eye movement trace time series data;

[0167] The AOI multi-scale comprehensive feature module is used to receive the AOI gaze time series data, perform multi-scale feature fusion, and output the category probability value corresponding to the AOI fusion feature;

[0168] The fusion module fuses the eye tracking depth feature and the AOI fusion feature to obtain the pilot's current SA capability value;

[0169] The SA level output module is used to output the pilot's current SA capability value.

[0170] The present invention transforms the pilot SA evaluation problem into a multi-dimensional time series data classification problem. Since the amount of experimental data is huge and the sequence size is particularly long, it is particularly necessary to consider the model selection in this case. The overall structure of the proposed Ti-SA model is as follows: Figure 3 shown.

[0171] The eye tracking multi-dimensional attention enhancement module includes:

[0172] A first linear projection layer, used to map the eye tracking time series data features to a high dimension;

[0173] The first random deactivation layer is used to perform random deactivation processing on the mapped high-dimensional time series data to enhance the generalization of the eye movement trace multi-dimensional attention enhancement module and prevent overfitting;

[0174] A first, a second and a third multi-layer attention structure connected in sequence, for obtaining an encoded representation of the eye tracking temporal data;

[0175] Bi-LSTM, which uses the bidirectional capability of LSTM to enhance the understanding of the time series of the encoded representation output after three multi-layer attention structures;

[0176] Adaptive average pooling layer to reduce the dimensionality of Bi-LSTM output features while retaining important information;

[0177] A second linear projection layer is used to match the output dimension to obtain three category probability values ​​corresponding to the eye movement trace;

[0178] Wherein, the multi-layer attention structure includes:

[0179] The multi-head attention module in the first multi-layer attention structure is used to obtain the output features of the first random inactivation layer, and use multiple attention heads to obtain comprehensive features; the multi-head attention module in the second multi-layer attention structure obtains the output features of the first multi-layer attention structure; the multi-head attention module in the third multi-layer attention structure obtains the output features of the second multi-layer attention structure;

[0180] The first residual connection and normalization layer in the first multi-layer attention structure is used to connect the data residual of the first random inactivation layer to the comprehensive features output by the multi-head attention module and then normalize them to obtain a normalized result 1A;

[0181] The first residual connection and normalization layer in the second multi-layer attention structure are used to connect the output feature residuals of the second multi-layer attention structure and then normalize them to obtain a normalized result 1B;

[0182] The first residual connection and normalization layer in the third multi-layer attention structure are used to residually connect the output features of the third multi-layer attention structure and then normalize them to obtain a normalized result 1C;

[0183] The feedforward network in each multi-layer attention structure is used to learn the nonlinear representation of the normalized results 1A, 1B and 1C respectively;

[0184] The second residual connection and normalization layer in each multi-layer attention structure are respectively used to perform residual connection on the normalized results 1A, 1B and 1C and the output of the feedforward network and then normalize them to obtain normalized results 2A, 2B and 2C;

[0185] The second random deactivation layer in each multi-layer attention structure is used to perform random deactivation processing on the normalized results 2A, 2B and 2C respectively to enhance the generalization ability of the multi-layer attention structure.

[0186] Among them, the multi-head attention module in the three-layer attention structure divides the input features of the module into multiple attention heads. Each head processes the feature information independently and understands the features from different angles. Finally, the outputs of multiple attention heads are merged to obtain a comprehensive feature representation.

[0187] The core idea of ​​the attention mechanism is to use query vectors (Query), key vectors (Key) and value vectors (Value) to represent different parts of the input sequence, and obtain the part most relevant to the query vector by calculating the similarity between them. Then, by normalizing and weighted summing the similarity, an output vector is obtained, which is the output result of the multi-head attention module.

[0188] The attention mechanism imitates the principle of human attention, allowing the model to focus on key information in the input data. This mechanism can assign weights to the input data by calculating the importance of different parts, thereby more effectively capturing the key features of the data. Therefore, for eye tracking time series data, the attention mechanism is used to construct a network for training. Adding an LSTM module can use the implicit state in the LSTM to alleviate errors or noise in the past sequence and improve the robustness of the model.

[0189] The input of the eye movement track multi-dimensional attention enhancement module is the eye movement track time series data in the training data set, and the output is the category probability value corresponding to the eye movement track time series data.

[0190] The AOI multi-scale comprehensive feature module includes six sequentially connected first to sixth heuristic modules, a global average pooling layer, a third random inactivation layer and a third linear projection layer;

[0191] The first to sixth heuristic modules are used to perform multi-scale feature extraction on the AOI gaze time series data; wherein the AOI gaze time series data is connected to the output of the second heuristic module through a first residual; the output of the third heuristic module is connected to the output of the fifth heuristic module through a second residual;

[0192] The global average pooling layer is used to reduce the dimension of the output features of the first heuristic module and extract global features;

[0193] The third random deactivation layer is used to perform random deactivation processing on the extracted global features to enhance the generalization ability of the AOI multi-scale comprehensive feature module;

[0194] The third linear projection layer is used to reduce the dimension of the matching output dimension to obtain three category probability values ​​corresponding to the AOI fusion feature.

[0195] like Figure 8 As shown, the heuristic module is not a traditional convolutional layer.

[0196] The first to sixth heuristic modules each include a first component, a second component and an output layer, wherein:

[0197] The first component is the first bottleneck layer, which uses a filter with a length of 1 to reduce the dimension of the input multi-dimensional AOI gaze time series data;

[0198] The second component includes three parallel convolutions, a max pooling, and a second bottleneck layer;

[0199] The three parallel convolutions are used to perform convolution operations of three different lengths on the output of the first bottleneck layer respectively, so as to capture temporal features of different scales;

[0200] The maximum pooling is used to perform a maximum pooling operation on the input multi-dimensional AOI gaze time series data;

[0201] The second bottleneck layer is used to reduce the dimension of the maximum pooling output;

[0202] The output layer is used to concatenate the time series features of different scales captured by the three parallel convolutions and the output features of the maximum pooling to form a final multi-dimensional time series output.

[0203] Exemplarily, the three convolutions of different lengths are one-dimensional convolutions of different lengths, and the convolution kernels are 15, 25, and 50 respectively;

[0204] The first bottleneck layer performs dimensionality reduction, thereby reducing the complexity of the model and the risk of overfitting. The heuristic module is a structure that can simultaneously apply convolution kernels and pooling layers of different sizes, thereby improving the expressiveness and efficiency of the model.

[0205] The eye movement data category probability value output by the eye movement multi-dimensional attention enhancement module and the AOI gaze time series data category probability value output by the AOI multi-scale comprehensive feature module are fused through the fusion module to obtain the SA level state, which is output through the SA level output module.

[0206] The fusion module averages the probability values ​​of the three categories corresponding to the eye tracking time series data and the three category probability values ​​corresponding to the AOI fusion features to obtain an averaged probability value;

[0207] Comparing the averaged probability values ​​of the three categories to determine the category with the highest probability;

[0208] The SA value corresponding to the category with the largest average probability value is used as the pilot's current SA capability value.

[0209] Step S22: train the Ti-SA model using the training data set until a trained Ti-SA model is obtained when the loss function converges.

[0210] For the training feature data input into the Ti-SA model (including eye tracking time series data and AOI gaze time series data), Spearman correlation coefficient analysis was used, p < 0.05, which was significantly correlated, so the input feature data did not need to be screened again.

[0211] The loss function of the Ti-SA model training is the cross entropy function. Until the loss function converges, the trained Ti-SA model is obtained. The cross entropy function E is shown in formula (3):

[0212]

[0213] Among them, n is the number of category labels, y i and p i are the true label and predicted category probability value of the i-th category label respectively.

[0214] The pilot data of eye tracking time series data and AOI gaze time series data obtained in the present invention specially selected professional pilots as the subjects of this test group. Through simple random sampling, fighter pilots who met the standards were randomly selected from the professional pilot group participating in flight training by lottery. Inclusion criteria: The pilots were in good physical condition, the Minnesota Multiphasic Personality Inventory (MMPI) met the standards, all had the necessary flying skills and experience, and the pilots' physical and cognitive abilities were normal during the test. All professional pilots participating in the experiment understood the test process and precautions, and voluntarily signed a written informed consent. A total of 30 pilots were included in the simulated flight process. They were in good physical and mental condition during the experiment. They were all male, aged 23-38 years old, with an average of (25.15±4.52) years old, with a flight time of 300-2200 hours, and an average flight time of 350.00 (340.75,912.50) hours.

[0215] The performance comparison of the Ti-SA model is as follows:

[0216] For example, the Ti-SA model training is performed under the experimental conditions of Python 3.8 and the deep learning framework Pytorch 2.1.2. In order to evaluate the performance of the Ti-SA model, the accuracy, precision, recall, and F1-score are used for measurement, as shown in formulas (4)-(7):

[0217]

[0218]

[0219]

[0220]

[0221] Where Acc, Pre, Rec and F1 represent accuracy, precision, recall and F1 score respectively. TP represents the number of samples that are actually positive and correctly predicted as positive by the Ti-SA model. TN represents the number of samples that are actually negative and correctly predicted as negative by the Ti-SA model. FP represents the number of samples that are actually negative but incorrectly predicted as positive by the Ti-SA model. FN represents the number of samples that are actually positive but incorrectly predicted as negative by the Ti-SA model.

[0222] The performance comparison of the Ti-SA model of the present invention and other multi-dimensional time series classification domain models is shown in Table 5:

[0223] Table 5: Performance comparison between each model and the proposed model

[0224] Model Accuracy (%) Accuracy (%) Recall rate (%) F1-score (%) MLSTM-FCN 83.27 84.44 90.46 87.35 Res-CNN 83.36 85.24 89.90 87.51 MiniRocket 88.36 89.10 93.58 91.28 TST 89.09 90.43 93.41 91.89 Ti-SA 92.18 92.95 95.49 94.20

[0225] (1) MLSTM-FCN: Convert the long short-term memory fully convolutional network (LSTM-FCN) to a multivariate model and expand the fully convolutional block;

[0226] (2) Res-CNN: Integrates residual networks and convolutional neural networks for time series classification;

[0227] (3) MiniRocket: A fast implementation that uses random convolution kernels to transform input time series and uses the transformed features to train the model;

[0228] (4) TST: A multi-dimensional temporal learning framework based on Transformer encoder.

[0229] (5) The Ti-SA model proposed in this invention: a multi-dimensional SA data learning model that integrates multi-head attention mechanism and heuristic module.

[0230] The performance of different models for the training data set is shown in Table 4. The Ti-SA model achieved the best performance with an accuracy of 92.18%, which is better than other commonly used models in the field of multidimensional time series classification.

[0231] SA affects the pilot's decision-making process and information acquisition, and is related to flight safety, efficiency, and the smooth operation of the entire aviation system. Therefore, evaluating the pilot's SA level has always been a focus of concern in the aviation industry. The present invention constructs a training data set under spatial directional flight simulation conditions and evaluates the pilot's SA level through a deep learning method, obtaining a Ti-SA model with high evaluation accuracy.

[0232] During the flight simulation test, each pilot was randomly frozen in the simulation scene at least 5 times. The purpose of this is to simulate the emergencies and challenges in real flight. Freezing the simulation scene helps to assess their SA level. Such comprehensive training can enhance the pilot's physiological adaptability and coping ability in flight missions, thereby improving flight safety and operational effectiveness.

[0233] The Ti-SA model in the present invention integrates the multi-head attention mechanism and the heuristic module to extract and learn the features of the eye tracking time series data and the AOI gaze time series data. Compared with the model that only learns unilateral features, the Ti-SA model of the present invention can comprehensively consider the features of the eye tracking time series data and the AOI gaze sequence, and realize a more comprehensive pilot SA evaluation.

[0234] The Ti-SA model uses a multi-head attention mechanism to capture long-term dependencies in eye movement time series data and mitigates errors or noise in past sequences through a Bi-LSTM model.

[0235] At the same time, the local short-term dependency features in the AOI gaze sequence are captured through the heuristic module to adapt to the situation where the pilot focuses on a specific AOI segment within a specific time.

[0236] The results in Table 5 show that the proposed situational awareness assessment method that integrates eye tracking time series data and AOI gaze sequence is effective and can accurately judge the pilot's situational awareness level. The model in this study has a comprehensive feature learning mechanism, robustness, and a specific design for pilot SA tasks, which has obvious advantages in pilot situational awareness assessment.

[0237] In summary, the pilot SA ability assessment system based on eye tracking and spatial orientation flight simulation according to the embodiment of the present invention has the following beneficial effects:

[0238] 1. The present invention obtains eye tracking time series data by real-time recording of the eye movement trajectory of the pilot in the space orientation flight simulator during simulated flight. The eye tracking time series data is combined with the AOI gaze time series data obtained in a simulated real flight environment to provide an instant and accurate SA ability assessment, which significantly improves the timeliness and accuracy of the assessment.

[0239] 2. The system of the present invention has an early warning function. When the pilot's SA ability is lower than the safety threshold, the early warning and voice broadcast module can issue a real-time early warning, effectively preventing flight risks caused by insufficient situational awareness, thereby enhancing overall flight safety;

[0240] 3. The present invention utilizes eye tracking time series data and AOI gaze time series data, and realizes quantitative analysis of pilot situational awareness through the Ti-SA evaluation model built into the microprocessor, thereby reducing the interference of human subjective factors, and the data-driven objective quantification improves the objectivity of the evaluation;

[0241] 4. The present invention integrates a spatial orientation flight simulator, eye tracking and artificial intelligence deep learning. The present invention represents an innovative application of technology integration in the aviation field and provides a new scientific tool for pilot training and evaluation.

[0242] 5. The present invention uses a visualization module to personalize the AOI and preset SA thresholds, and conducts evaluation based on multi-dimensional features. The system can provide customized evaluation results for each pilot;

[0243] 6. The data storage module in the present invention can be local storage or cloud storage according to the specific needs of pilot training, providing data support for pilot flight training and decision-making.

[0244] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. A pilot situation awareness ability assessment system based on eye tracking and spatial directional flight simulation, characterized in that: include: Space directional flight simulator, used to simulate real flight environment; An eye tracker is used to record the pilot's eye movement trajectory in real time in the simulated real flight environment to obtain eye movement track time series data, and to shoot a flight training video; A microprocessor evaluation module includes a timing data processing unit and a Ti-SA model; the timing data processing unit is used to obtain corresponding AOI gaze timing data based on the eye movement tracking timing data, the flight training video and the preset AOI subdivision area; the Ti-SA model is used to predict the pilot's current SA ability value based on the eye movement tracking timing data and the AOI gaze timing data, and to determine whether the predicted pilot's current SA ability value is lower than a preset SA threshold value, and if so, send a warning instruction to the warning and voice broadcast module; An early warning and voice broadcast module, used for issuing an early warning when the SA capability value is lower than the preset SA threshold; The Ti-SA model includes an eye tracking multi-dimensional attention enhancement module, an AOI multi-scale comprehensive feature module, a fusion module and a SA level output module; The eye movement trace multi-dimensional attention enhancement module is used to receive the eye movement trace time series data, perform multi-dimensional attention feature extraction, and output the category probability value corresponding to the eye movement trace time series data; The AOI multi-scale comprehensive feature module is used to receive the AOI gaze time series data, perform multi-scale feature fusion, and output the category probability value corresponding to the AOI fusion feature; The fusion module averages the category probability values ​​corresponding to the eye tracking time series data and the AOI fusion features, and after comparison, selects the SA value corresponding to the category with the maximum probability as the pilot's current SA capability value; The SA level output module is used to output the pilot's current SA capability value; The AOI area is dynamically identified by using the YOLOv8n model to obtain the preset AOI subdivision area; The eye movement tracking time series data includes two-dimensional coordinates of gaze focus, three-dimensional coordinates of gaze focus, three-dimensional coordinates of left eye original position, three-dimensional coordinates of left eye moving direction, left eye pupil diameter, three-dimensional coordinates of right eye original position, three-dimensional coordinates of right eye moving direction and right eye pupil diameter.

2. The system according to claim 1, characterized in that The system further comprises: A data storage module, used to store the eye tracking time series data, flight training videos and AOI gaze time series data, corresponding SA ability levels, data collection logs and warning logs; The visualization module is used to display the eye tracking data and AOI gaze timing data in real time, as well as the corresponding SA capability value; and provide an editing interface for the preset AOI subdivision area and the preset SA threshold.

3. The system according to claim 1, characterized in that The timing data processing unit performs the following steps to obtain the corresponding AOI gaze timing data, as follows: Detect and obtain all AOI subdivision areas in the flight training video that are arranged in time sequence; Based on the two-dimensional and three-dimensional coordinates of the gaze focus in the eye tracking time series data, determine whether the coordinates are located in a certain AOI subdivision area, and record the AOI subdivision area number of the pilot's gaze in chronological order according to the matching result to obtain the AOI gaze time series data; The steps for presetting the AOI subdivision area are as follows: Identify the types of instrument panels and displays in a space-oriented flight simulator: Based on the instrument panel display area and the screen display area, the instrument panel and the screen display are subdivided into a plurality of logical AOI subdivision areas; the exterior scene is one AOI subdivision area; The AOI subdivision areas are numbered to obtain k AOI subdivision areas.

4. The system according to claim 1, characterized in that The eye tracking multi-dimensional attention enhancement module includes: A first linear projection layer, used to map the eye movement tracking time series data features to a high dimension; The first random dropout layer is used to perform random dropout processing on the mapped high-dimensional time series data; A first, a second and a third multi-layer attention structure connected in sequence, for obtaining an encoded representation of the eye tracking temporal data; Bi-LSTM, which uses the bidirectional capability of LSTM to enhance the understanding of the time series of the encoded representation output after three multi-layer attention structures; Adaptive average pooling layer to reduce the dimensionality of Bi-LSTM output features while retaining important information; The second linear projection layer is used to match the output dimension and obtain three category probability values ​​corresponding to the eye movement trace.

5. The system according to claim 4, characterized in that The AOI multi-scale comprehensive feature module includes six sequentially connected first to sixth heuristic modules, a global average pooling layer, a third random inactivation layer and a third linear projection layer; The first to sixth heuristic modules are used to perform multi-scale feature extraction on the AOI gaze time series data; wherein the AOI gaze time series data is connected to the output of the second heuristic module through a first residual; the output of the third heuristic module is connected to the output of the fifth heuristic module through a second residual; The global average pooling layer is used to reduce the dimension of the output features of the sixth heuristic module and extract global features; The third random deactivation layer is used to perform random deactivation processing on the extracted global features to enhance the generalization ability of the AOI multi-scale comprehensive feature module; The third linear projection layer is used to reduce the dimension of the matching output dimension to obtain three category probability values ​​corresponding to the AOI fusion feature.

6. The system according to claim 5, characterized in that The first to sixth heuristic modules each include a first component, a second component and an output layer, wherein: The first component is the first bottleneck layer, which uses a filter with a length of 1 to reduce the dimension of the input multi-dimensional AOI gaze time series data; The second component includes three parallel convolutions, a max pooling, and a second bottleneck layer; The three parallel convolutions are used to perform convolution operations of three different lengths on the output of the first bottleneck layer respectively, so as to capture temporal features of different scales; The maximum pooling is used to perform a maximum pooling operation on the input multi-dimensional AOI gaze time series data; The second bottleneck layer is used to reduce the dimension of the maximum pooling output; The output layer is used to concatenate the time series features of different scales captured by the three parallel convolutions and the output features of the maximum pooling to form a final multi-dimensional time series output.

7. The system according to claim 1, characterized in that The fusion module averages the probability values ​​of the three categories corresponding to the eye tracking time series data and the three category probability values ​​corresponding to the AOI fusion features to obtain an averaged probability value; Comparing the averaged probability values ​​of the three categories to determine the category with the highest probability; The SA value corresponding to the category with the largest average probability value is used as the pilot's current SA capability value.

8. The system according to any one of claims 1 to 7, characterized in that: The AOI gaze timing data includes a number sequence of AOI subdivision areas that the pilot gazes at in sequence based on time during flight; the AOI subdivision areas include A sub-areas divided based on instrument display areas and screen display areas, and exterior scenery areas.

9. The system according to any one of claims 1 to 7, characterized in that: In the real flight environment simulated by the spatial orientation simulator, pilot training is carried out based on the preset SA assessment items to obtain the corresponding eye tracking time series data and AOI gaze time series data; Among them, the SA assessment items include pitch illusion, Coriolis illusion, tilt illusion, false horizon illusion, runway width and depth illusion and black hole approach illusion.

Images