Data-driven expert pilot contextualized eye movement template construction method and system

Abstract

The present application relates to the technical field of aviation flight simulation training and human-computer interaction, and discloses a data-driven expert pilot situational eye movement template construction method and system. Eye movement data and synchronous flight parameters of expert pilots are collected; accurate denoising preprocessing is carried out through an algorithm based on pupil change speed; a double screening mechanism of flight control deviation and Euclidean distance typicality calculation is introduced to extract the optimal benchmark expert sample; event-driven non-equal section is carried out based on flight parameter trigger points; a two-dimensional cognitive model of static AOI fixation weight and dynamic Markov chain transition matrix is constructed; and finally a multi-modal enhanced eye movement teaching template containing visual enhancement clues, scanning path traces and synchronous auditory scripts is generated. The present application breaks through the limitation of shallow space mapping, innovates the multi-modal enhanced presentation form, eliminates the individual control rhythm difference, realizes the absolute alignment of cognitive stages, and guarantees the high quality and universality of the teaching template.

Description

Technical Field

[0001] This invention relates to the field of aviation flight simulation training and human-computer interaction technology, specifically a data-driven method and system for constructing contextualized eye-tracking templates for expert pilots. Background Technology

[0002] In the field of aviation safety, novice pilots often face bottlenecks such as imbalanced attention allocation, missing critical instrument check loops, and immature visual scanning strategies when dealing with special situations (such as wind shear go-around, engine failure, hydraulic system failure, etc.), which seriously affect flight safety.

[0003] Studies have shown that expert pilots, through long-term training, have developed efficient visual scanning patterns (such as "T-scan" and "center-radial scan"), enabling them to quickly acquire critical flight information and make correct decisions under high cognitive load conditions. Eye Movement Modeling Examples (EMME) is a teaching method that overlays expert eye movement data onto mission scenario videos in a visual format, and has proven effective in multiple fields. Flight simulators can output flight parameters to calibrate key state transition points in flight missions.

[0004] Existing EMME construction techniques typically include the following steps: collecting eye-tracking data from experts during task execution and extracting gaze coordinates using basic gaze point recognition algorithms; segmenting the video based on human experience to divide the task into several time segments; and visualizing the eye-tracking data and overlaying it onto the task video.

[0005] In terms of visualization, existing technologies typically employ two basic approaches: one is "dynamic dot trajectory," which involves the filtered gaze points moving around in real time as individual moving dots in the video; the other is "static heatmap," which involves spatially Gaussian smoothing of gaze points within a certain time window and using color depth (warm and cool tones) to display the area with the highest gaze density during that period.

[0006] Current eye-tracking playback teaching methods face numerous technical challenges: raw eye-tracking data contains a large number of artifacts caused by blinking, micro-saccades, and equipment noise; simple fixation filtering still results in severe image jitter in dynamic, high-frequency flight scenarios. Therefore, existing technologies have the following drawbacks:

[0007] First, traditional heatmaps or single-dot trajectories only offer a superficial description of "spatial location mapping," failing to extract the cognitive loops that have structured teaching significance. Specifically:

[0008] (1) Although static heatmaps can reflect the spatial proportion of attention distribution (i.e., "where to look the longest"), they completely lose the temporal information of the time dimension. In flight missions, the "cross-checking sequence" between instruments (such as switching from the attitude gauge to the airspeed indicator and then quickly returning to the attitude gauge) is the core of judging whether the strategy is correct. Heatmaps cannot show this dynamic transfer logic.

[0009] (2) Due to the high sampling rate of eye trackers and the high frequency of microscratching of the human eye, even the trajectory of a moving dot after basic filtering often appears as a chaotic network of jumps in complex flight scenarios. Learners passively follow a single fast-moving light dot, which easily exceeds their visual working memory capacity, making it difficult to connect discrete points into a line in the brain, and thus unable to grasp the underlying visual search strategy of experts.

[0010] Second, existing methods of manual video segmentation or fixed time window segmentation lack objective standards and cannot objectively align the cognitive processes of different pilots. During emergency handling, different pilots exhibit varying operational rhythms (such as the speed of retracting landing gear and pushing the throttle), and fixed time windows cannot achieve precise alignment of cognitive processes at the semantic level of operation.

[0011] Third, existing technologies lack a mechanism to automatically extract "common strategies" from data from multiple experts, making it easy to introduce personalized and extreme strategies that contaminate template quality. Playing back only the eye-tracking video of a single expert can easily introduce that expert's personalized and extreme saccadic habits, and such noisy data will seriously pollute the universality and teachability of the template.

[0012] Fourth, the purely visual single-channel presentation method piles instrument information, external scenes, and expert gaze points all into the visual processing channel. According to the multimedia learning cognitive load theory, this will bring learners an extremely high external cognitive load. Summary of the Invention

[0013] To address the problem of existing technologies' inefficient and high-quality construction of pilot eye-tracking instruction templates, this invention aims to provide a data-driven method and system for constructing contextualized eye-tracking templates for expert pilots. This method overcomes the limitations of shallow spatial mapping, innovates multimodal enhanced presentation, eliminates individual differences in operational rhythm, achieves absolute alignment of cognitive stages, and ensures the high quality and universality of the instruction templates. The technical solution is as follows:

[0014] A data-driven method for constructing contextualized eye-tracking templates for expert pilots includes the following steps:

[0015] Step 1: Determine the special flight scenario, collect eye-tracking data from expert pilots, simultaneously record the flight parameters output by the flight simulator, and conduct post-flight replay interviews;

[0016] Step 2: Preprocess the raw eye-tracking data, including blink detection and interpolation based on pupil change velocity, and I-VT velocity threshold noise filtering, and output the effective fixation point sequence;

[0017] Step 3: Sort the expert samples based on the flight control deviation index, and select the top preset proportion with the smallest deviation as candidate high-quality samples; construct the gaze distribution feature vector of each candidate high-quality sample, calculate the Euclidean distance between the gaze distribution feature vector and the average vector of the expert group, and select the sample with the smallest Euclidean distance as the benchmark expert sample.

[0018] Step 4: Extract flight parameters, use key state switching points as segment trigger points, divide the flight process into multiple cognitive stages, and obtain segmented eye-tracking data corresponding to each cognitive stage;

[0019] Step 5: For each cognitive stage, construct a static region of interest gaze weight model and a dynamic Markov chain transition matrix model to form a two-dimensional cognitive model;

[0020] Step 6: Obtain the first-person view cockpit video corresponding to the benchmark expert sample, generate a fixation point overlay, visual enhancement cues, scanning path trails and synchronous auditory scripts, perform multimodal integration, and output a multimodal enhanced eye-tracking teaching template.

[0021] A data-driven system for constructing contextualized eye-tracking templates for expert pilots, comprising:

[0022] The data acquisition module is used to determine special flight scenarios. It collects eye-tracking data from expert pilots through eye-tracking devices, synchronously records the flight parameters output by the flight simulator, and conducts post-event replay interviews after the scenario is completed.

[0023] The preprocessing module is used to sample the raw eye-tracking data, calculate the pupil diameter derivative, detect blinks, perform interpolation and fill in the gaps, and filter I-VT velocity threshold noise to output a valid gaze sequence.

[0024] The benchmark expert sample screening module is used to sort and filter candidate high-quality samples according to flight control deviation index, construct gaze distribution feature vector, calculate Euclidean distance to evaluate typicality, verify signal quality, and output benchmark expert samples.

[0025] The event-driven segmentation module is used to extract flight parameters, identify key state transition points as segmentation trigger points, and divide the flight process into multiple cognitive stages.

[0026] A two-dimensional modeling module is used to construct a static region of interest gaze weight model and a dynamic Markov chain transition matrix model for each cognitive stage.

[0027] The template generation module is used to acquire first-person perspective video, generate foveation overlay, visual enhancement cues, scan path trails and synchronized auditory scripts, and perform multimodal integration to output the final teaching template.

[0028] The beneficial effects of this invention are:

[0029] 1. This invention targets high-dynamic flight scenarios and uses a blink detection and interpolation algorithm based on pupil change speed and an I-VT velocity threshold filtering algorithm to perform engineered preprocessing on the raw eye movement data, effectively eliminating hardware noise and physiological artifacts in the chaotic trajectory, making the teaching template screen smooth and stable.

[0030] 2. This invention employs a dual screening mechanism combining flight control deviation filtering and Euclidean distance typicality calculation to automatically identify the most representative common scanning strategies from a massive dataset of expert flight operation samples. The screening process consists of two steps: first, selecting the top 20% of candidate high-quality samples based on control deviation ranking; second, calculating the baseline expert sample based on the sample whose average gaze distribution (smallest Euclidean distance) is closest to the expert group's average gaze distribution. This dual mechanism is objective and repeatable, avoiding subjective bias from manual selection and contamination from individual extreme strategies.

[0031] 3. This invention uses event-driven non-equal segmentation technology with actual flight parameter operation actions as segmentation trigger points, overcoming the problem that fixed time windows cannot adapt to individual operation rhythms and achieving precise alignment of the cognitive process.

[0032] 4. This invention proposes a two-dimensional cognitive modeling system consisting of a static attention allocation model and a dynamic scanning strategy model (i.e., a static interest region gaze weight model and a dynamic Markov chain transition matrix model). This system systematically describes the visual cognitive strategies of experts from two dimensions: "what to see" and "how to see." In particular, the dynamic dimension of the Markov chain transition matrix successfully extracts the pilot's crucial "cross-checking logic" into quantifiable and labelable high-frequency transition paths.

[0033] 5. This invention employs a multimodal enhanced presentation method to make implicit knowledge explicit through "scanning path trails." Learners no longer face chaotic, jumping individual light spots, but instead clearly see cognitive logical trajectories shaped like "T" or "triangles." By integrating multi-channel information from the visual and auditory channels, and based on the cognitive load theory of multimedia learning, it effectively reduces the cognitive load of learners from a single visual channel, significantly improving the teaching efficiency and safety of special situation flight training.

[0034] 6. The entire process of this invention is automated based on data-driven approaches, which reduces subjective bias caused by human intervention and improves the efficiency and standardization of template construction. Attached Figure Description

[0035] Figure 1 This is a flowchart of a data-driven method for constructing contextualized eye-tracking templates for expert pilots, according to an embodiment of the present invention.

[0036] Figure 2 The above is a flowchart of the eye-tracking data preprocessing process according to an embodiment of the present invention.

[0037] Figure 3 This is a flowchart of the benchmark expert sample screening process according to an embodiment of the present invention.

[0038] Figure 4 This is a schematic diagram of event-driven non-equal segmentation according to an embodiment of the present invention.

[0039] Figure 5 This is a schematic diagram illustrating the construction of a two-dimensional cognitive model according to an embodiment of the present invention.

[0040] Figure 6 This is a system architecture diagram according to an embodiment of the present invention.

[0041] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0042] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0043] One embodiment of the present invention: a data-driven method for constructing contextualized eye-tracking templates for expert pilots, such as... Figure 1 As shown in the flowchart, the complete processing flow from data acquisition to template generation is illustrated below:

[0044] Step S1: Special Situation Flight Scenario Simulation and Eye Movement Data Acquisition: Determine the special situation flight scenario, collect eye movement data from expert pilots, synchronously record the flight parameters output by the flight simulator, and conduct post-event playback interviews.

[0045] S11: Determine and encode the special flight scenarios that need to be simulated. The special flight scenarios include, but are not limited to, wind shear go-around, engine failure, hydraulic system failure, etc.

[0046] S12: Set the parameters for the special flight scenario in the flight simulator, and collect eye movement data from several expert pilots under each special flight scenario. The eye movement data includes gaze position, gaze duration, and pupil diameter. The eye movement data is collected synchronously using an eye-tracking device (sampling rate greater than or equal to 60Hz) worn on the expert pilots' heads.

[0047] S13: Synchronously records flight parameters output by the flight simulator, including flight attitude parameters (pitch angle, roll angle), altitude parameters, speed parameters, configuration change parameters (such as flap position), and control input parameters. All data is accessed through a unified interface and then undergoes anomaly removal and consistency checks to provide reliable input for subsequent processing.

[0048] S14: After each special scenario is completed, conduct a post-event replay interview with each expert pilot to record the expert's thought process and decision-making basis, which will be used for the generation of subsequent auditory scripts.

[0049] Step S2: Eye movement data preprocessing: Preprocess the raw eye movement data, including blink detection and interpolation based on pupil change speed, and I-VT (Velocity-Threshold Identification) velocity threshold noise filtering, and output the effective gaze point sequence.

[0050] like Figure 2 The eye-tracking data preprocessing flowchart shown below illustrates the complete preprocessing process from raw data to a valid fixation sequence, as detailed below:

[0051] S21: Eye-tracking data for a special flight scenario is sampled at a preset frequency to obtain several sampling points.

[0052] S22: Obtain the pupil diameter at each sampling point, and differentiate the pupil diameter at all sampling points to obtain the pupil change rate curve.

[0053] S23: Obtain the minimum and maximum values ​​in the pupil change rate curve, take the last sampling point close to 0 before the minimum value as the blink start time, and take the first sampling point close to 0 after the maximum value as the blink end time.

[0054] S24: The time interval between the start and end of a blink is defined as the blink interval. Linear or polynomial interpolation is used to fill in the pupil diameter and fixation position within the blink interval. The calculation expression is as follows:

[0055] (1);

[0056] In the formula, and , where are the gaze positions (or pupil diameters) at the start and end of the blink, respectively, and t is the time to be interpolated within the blink interval.

[0057] S25: The pupil diameter time series is composed of the interpolated pupil diameters from all sampling points.

[0058] S26: Obtain the gaze position of each sampling point, and construct a gaze position time series from the gaze positions of all sampling points.

[0059] S27: Apply the I-VT (Velocity-Threshold Identification) algorithm to calculate the gaze angular velocity between adjacent sampling points. Set a velocity threshold (e.g., greater than 30 degrees / second) to identify saccades. Use saccade suppression technology to hide high-speed visual jerks, retaining only valid gaze points whose duration exceeds a preset duration threshold (e.g., 100ms), thus forming the final gaze position time series.

[0060] Step S3: Benchmark expert sample screening: Based on the flight control deviation index, the expert samples are sorted, and the top preset proportion with the smallest deviation is selected as the candidate high-quality samples; the gaze distribution feature vector of each candidate high-quality sample is constructed, the Euclidean distance between it and the average vector of the expert group is calculated, and the sample with the smallest Euclidean distance is selected as the benchmark expert sample.

[0061] This approach automatically selects the most representative expert eye-tracking data from a database using a data-driven method, rather than relying on manual specification. The flowchart for the benchmark expert sample selection is as follows: Figure 3 As shown, the specific process is as follows:

[0062] S31: Obtain the flight performance indicators of all expert pilots under the same special flight scenario. The ranking of the flight performance indicators is based on the flight control deviation values ​​sorted from smallest to largest, including but not limited to: root mean square of pitch angle deviation (RMS), heading maintenance deviation, altitude deviation, landing point deviation, and other comprehensive scores.

[0063] S32: The flight operation samples with the smallest pre-set proportion (such as the top 20%) are identified as the samples with the highest flight control quality and are used as candidate high-quality samples.

[0064] S33: Construct a gaze distribution feature vector v_trial for each candidate high-quality sample. The gaze distribution feature vector includes the gaze time percentage of each region of interest (AOI). Regions of interest include: Primary Flight Display (PFD), Airspeed Indicator (ASI), Altimeter (ALT), Vertical Speed ​​Indicator (VSI), Attitude Director Indicator (ADI), Navigation Display (ND), Engine Parameter Display Area, Out-The-Window (OTW), etc.

[0065] S34: Calculate the average of the fixation distribution feature vectors of all candidate high-quality samples to obtain the expert group average vector. .

[0066] S35: Calculate the Euclidean distance d between the gaze distribution feature vector of each candidate high-quality sample and the average vector of the expert group.

[0067] (2);

[0068] In the formula, The proportion of gaze time in the i-th region of interest within the gaze distribution feature vector of candidate high-quality samples. represents the percentage of fixation time for the i-th region of interest in the average vector of the expert group, where n is the total number of regions of interest.

[0069] S36: Select the candidate high-quality sample with the smallest Euclidean distance d value as the final benchmark expert sample (i.e., the optimal representation base). Selecting the sample with the smallest distance aims to exclude situations where some experts, although manipulating high performance, have adopted extremely personalized or non-standard visual scanning strategies, thereby ensuring that the selected benchmark expert sample best represents the common scanning strategies of the expert group.

[0070] S37: Confirm the signal quality of the benchmark expert sample and verify that there is no serious tracking loss or blink artifact exceeding the preset threshold (e.g., data loss rate not exceeding 5%) in the record; if the signal quality does not meet the requirements, select the candidate high-quality sample with the second smallest Euclidean distance for verification until a sample that meets the conditions is found.

[0071] Step S4: Event-driven non-equal segmentation: Extract flight parameters, use key state switching points as segmentation trigger points, divide the flight process into multiple cognitive stages, and obtain segmented eye-tracking data corresponding to each cognitive stage.

[0072] This step does not use a fixed time window for data segmentation. Instead, it uses the actual operational actions of flight parameters as trigger points for segmentation to eliminate the impact of differences in the control rates of different pilots on the alignment of the cognitive process.

[0073] This invention presents a schematic diagram of event-driven non-equal segmentation (taking a special situation go-around scenario as an example), as follows. Figure 4 As shown, this demonstrates a method for dividing the flight process into multiple cognitive stages based on flight parameter trigger points.

[0074] S41: Extract flight parameters output by the flight simulator in real time. The flight parameters include flight attitude parameters (pitch angle, roll angle), altitude parameters, speed parameters, and configuration change parameters.

[0075] S42: The key state transition points of the flight mission are used as segmented trigger points. These key state transition points are determined by identifying actual operational actions rather than based on a fixed time period. Taking a go-around scenario as an example, the segmented trigger points include: Trigger point 1 is the moment the emergency occurs (from the start to the establishment of the go-around attitude is phase A, with the focus on the Attitude Controller Indicator (ADI) and Airspeed Indicator (ASI); Trigger point 2 is the moment when the flap retraction action is actually performed (the start of phase B, with the focus on the Altimeter (ALT) and Velocity Indicator (VSI)); Trigger point 3 is the moment when the leveling altitude is reached (the start of phase C, with the focus on the Navigation Display (ND) and External View of the World (OTW)).

[0076] S43: Take the time period between two adjacent segment trigger points as a cognitive stage, and obtain segmented eye movement data corresponding to multiple cognitive stages.

[0077] Step S5: Construction of a two-dimensional cognitive model: For each cognitive stage, construct a static interest region gaze weight model and a dynamic Markov chain transition matrix model to form a two-dimensional cognitive model.

[0078] A schematic diagram of the construction of the two-dimensional cognitive model according to an embodiment of the present invention is shown below. Figure 5 As shown, Figure 5 It demonstrates the parallel modeling process of the static dimension (AOI gaze weight model) and the dynamic dimension (Markov chain transition matrix model).

[0079] Two-dimensional modeling was performed on segmented eye-tracking data for each cognitive stage:

[0080] 1. Static dimensional modeling (solving the question of what to look at):

[0081] S51: Calculate the fixation contribution rate of each region of interest within each cognitive stage. The fixation contribution rate is the proportion of the cumulative fixation time of that region of interest to the total fixation time of that cognitive stage. The calculation expression is as follows:

[0082] (3);

[0083] In the formula, The gaze contribution rate for the i-th region of interest. The cumulative gaze duration for this region of interest. is the cumulative gaze duration for the j-th region of interest; n is the total number of regions of interest.

[0084] S52: Mark critical regions of interest where the contribution rate of expert gaze is significantly higher than that of novice gaze. Taking the emergency go-around scenario as an example, experimental data shows that after the emergency is triggered, the average gaze duration of the expert group on the airspeed indicator (ASI) increased by 40%, while that of the novice group only increased by 10%. Therefore, the airspeed indicator was marked as a critical region of interest.

[0085] 2. Dynamic Dimensional Modeling (solving the question of how to view):

[0086] S53: Extracting the gaze transition probability matrix between regions of interest based on Markov chains:

[0087] (4);

[0088] In the formula, Let i be the transition probability from region of interest i to region of interest j. Let n be the number of transitions from region of interest i to region of interest j, and n be the total number of regions of interest.

[0089] S54: For transition probabilities exceeding a preset probability threshold (e.g.) The high-frequency transfer paths are marked. Taking the special situation go-around phase B as an example, the transfer matrix shows that there is a very strong bidirectional loop between the Attitude Controller Indicator (ADI) and the Vertical Speed ​​Indicator (VSI), and its transfer probability exceeds the threshold. This path is marked as a high-frequency transfer path, that is, the expert structured cross-check loop.

[0090] Step S6: Generation of multimodal enhanced eye-tracking teaching template: Obtain the first-person view cockpit video corresponding to the benchmark expert sample, generate a fixation point overlay, visual enhancement cues, scan path trails and synchronous auditory scripts, perform multimodal integration, and output a multimodal enhanced eye-tracking teaching template.

[0091] This step uses a multimodal enhanced dynamic gaze overlay to generate the final teaching template.

[0092] S61: Obtain first-person view cockpit video corresponding to the benchmark expert sample. The field of view must cover the main flight displays (PFD or six instruments), engine parameter display area, and the external view of the windshield.

[0093] S62: Map the preprocessed gaze coordinates onto the first-person cockpit video to generate a gaze point overlay. The gaze point is presented as a semi-transparent ring with a diameter corresponding to a 2-degree field of view (the high-resolution visual range of the human fovea), and the color is high-contrast magenta or cyan to avoid confusion with the red, yellow, and green warning lights on the instrument panel.

[0094] S63: Generate visual enhancement cues for the marked key regions of interest in the static attention allocation model, including: generating a shrinking aperture forward guidance cue 150 to 200 milliseconds before the expert gaze point reaches the key region of interest (a shrinking circle or slight highlighting appears on the target instrument area to guide learner attention to pre-allocation); and applying a spotlight effect when the expert gaze point enters the key region of interest (the surrounding area darkens by 20%, highlighting only the instrument, lasting for about 300 milliseconds).

[0095] S64: For the high-frequency transition paths marked in the dynamic scanning strategy model, generate a scan path trail with a fade-out duration of 500 to 600 milliseconds. This trail appears as a semi-transparent magenta trail that gradually fades out behind the fixation point, demonstrating the movement path and sequence of the gaze and helping learners identify the cross-checking loop structure of experts.

[0096] S65: Based on post-experimental playback interviews with experts, synchronized auditory scripts are extracted. These auditory scripts are short, imperative statements reconstructed from experimental data and are strictly synchronized with eye-tracking events. For example, the voiceover at the moment of an emergency is "Wind shear warning. First check airspeed trend, maintain attitude, maximum thrust go-around"; the voiceover at the moment of flap retraction is "Altitude safe, speed sufficient, retract flaps".

[0097] S66: Overlay the current task stage label text in the corner of the video to help learners build a mental model. Taking the emergency go-around scenario as an example, the stage labels include: "Stage A: Establish go-around attitude", "Stage B: Establish climb and recovery mode", and "Stage C: En route entry".

[0098] S67: Multimodal integration of first-person cockpit video, fixation overlay, visual enhancement cues, scan path trails, stage labels, and auditory scripts to generate the final multimodal enhanced eye-tracking teaching template.

[0099] This application provides a data-driven system for constructing contextualized eye-tracking templates for expert pilots. Figure 6 This is a system architecture diagram according to an embodiment of the present invention, including six main functional modules: a data acquisition module, a preprocessing module, a benchmark expert sample screening module, an event-driven segmentation module, a two-dimensional modeling module, and a template generation module. Details are as follows:

[0100] (1) Data acquisition module;

[0101] The data acquisition module is used to determine special flight scenarios. It collects eye-tracking data from expert pilots through eye-tracking devices, records flight parameters output by the flight simulator, and conducts post-scenario replay interviews after the scenario is completed.

[0102] This module includes a scenario determination unit, an eye-tracking acquisition unit, a flight parameter recording unit, and an interview recording unit. The scenario determination unit identifies the specific flight scenario to be collected based on training objectives and scenario types. The eye-tracking acquisition unit uses eye-tracking equipment to collect eye-tracking data from expert pilots during the execution of the special situation flight mission, including gaze position, gaze duration, saccade trajectory, and pupil diameter. The flight parameter recording unit synchronously records flight parameters output by the flight simulator, such as altitude, speed, heading, attitude, control inputs, and system status. The interview recording unit conducts post-flight replay interviews after the flight scenario is completed, recording expert pilots' explanations of key observation points, attention allocation strategies, operational intentions, and judgment criteria.

[0103] (2) Preprocessing module;

[0104] The preprocessing module is used to sample the raw eye-tracking data, differentiate the pupil diameter, detect blinks, interpolate and fill in the gaps, and filter I-VT velocity threshold noise, outputting a valid gaze point sequence.

[0105] This module includes a sampling unit, a pupil differentiation unit, a blink detection unit, an interpolation unit, and a noise filtering unit. The sampling unit samples or resamples the raw eye-tracking data at a preset frequency to form a continuous sequence of sampling points. The pupil differentiation unit obtains the pupil diameter at each sampling point and calculates the pupil change velocity curve. The blink detection unit identifies the start and end times of blinks based on feature points in the pupil change velocity curve. The interpolation unit fills in missing or abnormal fixation positions and pupil diameters within the blink interval using interpolation. The noise filtering unit removes abnormal jump points and invalid eye-tracking data using the I-VT velocity thresholding method, outputting a valid fixation point sequence.

[0106] (3) Benchmark expert sample screening module;

[0107] The benchmark expert sample screening module is used to sort and filter candidate high-quality samples according to flight control deviation index, construct gaze distribution feature vector, calculate Euclidean distance to evaluate typicality, verify signal quality, and output benchmark expert samples.

[0108] This module includes a performance filtering unit, a feature vector construction unit, an Euclidean distance calculation unit, and a signal quality verification unit. The performance filtering unit sorts and filters expert samples based on flight control deviation indicators to obtain candidate high-quality samples. The feature vector construction unit constructs gaze distribution feature vectors for candidate high-quality samples based on eye-tracking characteristics such as the gaze duration ratio of each region of interest. The Euclidean distance calculation unit calculates the Euclidean distance between the gaze distribution feature vectors of candidate high-quality samples and the average vector of the expert group to assess the typicality of the sample's gaze pattern. The signal quality verification unit verifies the completeness, effective sampling rate, missing data ratio, and synchronization quality of the candidate samples' eye-tracking data, outputting benchmark expert samples that meet the requirements.

[0109] (4) Event-driven segmentation module;

[0110] The event-driven segmentation module is used to extract flight parameters, identify key state transition points as segmentation trigger points, and divide the flight process into multiple cognitive stages.

[0111] This module includes a flight parameter extraction unit, a trigger point identification unit, and a segmented execution unit. The flight parameter extraction unit is used to extract key flight parameters and operational status parameters related to the emergency response process from the data recorded by the flight simulator; the trigger point identification unit is used to determine segmented trigger points based on key state switching points such as the time of emergency occurrence, the time of actual operation, and the time of reaching a specific flight state; the segmented execution unit is used to divide the complete flight process into multiple cognitive stages based on the segmented trigger points and obtain segmented eye-tracking data corresponding to each cognitive stage.

[0112] (5) Two-dimensional modeling module;

[0113] The two-dimensional modeling module is used to construct a static AOI gaze weight model and a dynamic Markov chain transition matrix model for each cognitive stage.

[0114] This module includes a static AOI weight modeling unit and a dynamic Markov transfer modeling unit. The static AOI weight modeling unit is used to calculate the cumulative fixation duration and fixation contribution rate of different regions of interest within each cognitive stage, and construct a static fixation weight model for regions of interest. The dynamic Markov transfer modeling unit is used to calculate the number of fixation transfers and the transfer probability between different regions of interest by expert pilots, and construct a dynamic Markov chain transfer matrix model, thereby characterizing the dynamic visual scanning patterns of expert pilots during emergency response.

[0115] (6) Template generation module;

[0116] The template generation module is used to acquire first-person perspective videos, generate foveation overlays, visual enhancement cues, scan path trails, and synchronized auditory scripts, and perform multimodal integration to output the final teaching template.

[0117] This module includes a video acquisition unit, a gaze overlay unit, a visual enhancement unit, an auditory scripting unit, and a multimodal integration unit. The video acquisition unit acquires first-person perspective cockpit videos corresponding to benchmark expert samples; the gaze overlay unit overlays expert pilots' gaze points, gaze trajectories, or gaze hotspots onto the first-person perspective videos; the visual enhancement unit generates visual enhancement cues based on key regions of interest and expert gaze paths to highlight instruments, areas, or risk cues that learners need to focus on; the auditory scripting unit generates auditory cue scripts synchronized with the video based on post-interview playback and eye-tracking events; and the multimodal integration unit integrates the first-person perspective video, gaze overlay layer, visual enhancement cues, scan path trails, and synchronized auditory scripts to generate a multimodal enhanced eye-tracking teaching template.

[0118] Figure 7 The schematic diagram of the electronic device 50 provided in the embodiments of this application includes: a processor 501, a memory 502, and a network interface 503.

[0119] Specifically, the program may include program code, which includes computer operation instructions. Memory 502 may include high-speed RAM, and may also include non-volatile memory, such as at least one disk storage device.

[0120] The processor 501 is used to execute computer execution instructions stored in the memory 502 to implement the technical solution of a data-driven expert pilot contextualized eye-tracking template construction method described in the foregoing method embodiments.

[0121] The processor 501 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.

[0122] Optionally, the electronic device 50 may also include a communication interface 503, which allows communication and interaction with external devices, such as management terminals (e.g., laptops, desktops) or user terminals (e.g., mobile phones, tablets). In specific implementations, if the network interface 503, memory 502, and processor 501 are implemented independently, they can be interconnected via a bus to complete communication with each other.

[0123] A bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc., but this does not mean there is only one bus or one type of bus.

[0124] Optionally, in a specific implementation, if the network interface 503, memory 502, and processor 501 are integrated on a single chip, then the network interface 503, memory 502, and processor 501 can communicate through an internal interface.

[0125] This application also provides a computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it implements the technical solution of the above-described data-driven expert pilot contextualized eye-tracking template construction method embodiment. Its implementation principle and technical effects are similar and will not be repeated here.

[0126] In one possible implementation, the computer-readable medium may include random access memory (RAM), read-only memory (ROM), compact discread-only memory (CD ROM) or other optical disc storage, disk storage or other magnetic storage device, or any other medium on which the desired program code is intended to be carried or stored in the form of instructions or data structures, and may be accessible by a computer.

[0127] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the technical solution of the above-described data-driven expert pilot contextualized eye-tracking template construction method embodiment. Its implementation principle and technical effects are similar and will not be repeated here.

[0128] In the specific implementation of the aforementioned terminal device or server, it should be understood that the processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the examples of this application can be directly manifested as being executed by a hardware processor, or being executed by a combination of hardware and software modules within the processor.

[0129] Those skilled in the art will understand that all or part of the steps in any of the above method embodiments can be implemented by hardware associated with program instructions. The aforementioned program can be stored in a computer-readable storage medium, and when the program is executed, all or part of the steps in the above method embodiments are performed.

[0130] If the technical solution of this application is implemented in software form and sold or used as a product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the technical solution of this application can be embodied in the form of a software product, which is stored in a storage medium and includes a computer program or several instructions. This computer software product causes a computer device (which may be a personal computer, server, network device, or similar electronic device) to execute all or part of the steps of the method described in the embodiments of this application.

[0131] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0132] It should be further noted that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0133] It should be understood that the above-described device embodiments are merely illustrative, and the device of this application can be implemented in other ways. For example, the division of units / modules in the above embodiments is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units, modules, or components can be combined, integrated into another system, or some features can be ignored or not executed. In addition, unless otherwise specified, the functional units / modules in the various embodiments of this application can be integrated into one unit / module, or each unit / module can exist physically separately, or two or more units / modules can be integrated together. The integrated units / modules described above can be implemented in hardware or in the form of software program modules.

[0134] When integrated units / modules are implemented in hardware, the hardware can be digital circuits, analog circuits, etc. The physical implementation of the hardware structure includes, but is not limited to, transistors, memristors, etc. Unless otherwise specified, the processor can be any suitable hardware processor, such as a CPU, GPU, FPGA, DSP, and ASIC, etc. Unless otherwise specified, the storage unit can be any suitable magnetic or magneto-optical storage medium, such as Resistive Random Access Memory (RRAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), Enhanced Dynamic Random Access Memory (EDRAM), High Bandwidth Memory (HBM), Hybrid Memory Cube (HMC), etc.

[0135] If the integrated unit / module is implemented as a software program module and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0136] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0137] In summary, the key technical points of this invention are summarized as follows:

[0138] 1. Blink detection and interpolation preprocessing method based on extreme values ​​of pupil change rate: The change rate curve is obtained by differentiating the pupil diameter. The extreme value method is used to accurately locate the blink interval and perform interpolation filling. Combined with I-VT velocity threshold filtering to eliminate hardware noise and physiological artifacts, the image jitter in high-frequency dynamic flight environment is effectively eliminated.

[0139] 2. A dual screening mechanism combining flight control deviation filtering with Euclidean distance typicality calculation: Based on the flight control deviation values ​​sorted from smallest to largest, the top 20% of the smallest deviations are selected as candidate high-quality samples. Then, by calculating the Euclidean distance between the gaze distribution feature vector and the average vector of the expert group, the most representative benchmark expert sample is automatically identified from the candidate high-quality samples, eliminating the interference of extremely personalized strategies.

[0140] 3. Event-driven non-equal segmentation technology: Segmentation is performed using actual operational actions of flight parameters (rather than fixed time windows) as trigger points to eliminate the impact of differences in the control rates of different pilots on the alignment of the cognitive process, and to achieve absolute alignment between operational semantics and cognitive stages.

[0141] 4. Two-dimensional cognitive modeling of static AOI gaze weight and dynamic Markov chain transition matrix: Breaking through the limitations of shallow spatial mapping of traditional heatmaps or single-point trajectories, it systematically solves the two core teaching problems of "what to look at" and "how to look at", and accurately strips away the structured "cross-checking loops" of expert pilots.

[0142] 5. Generation method of multimodal enhanced eye-tracking teaching template: The implicit cross-checking loop is made explicit by using the scanning path trail with fade-out duration. It integrates first-person perspective video, fixation point superposition (magenta ring with 2-degree field of view), forward guidance cues and spotlight effects, synchronous auditory script and stage labels in a multi-channel presentation, which effectively reduces the learner's external cognitive load.

[0143] 6. The corresponding implementation forms of devices, electronic equipment and storage media.

Claims

1. A data-driven method for constructing contextualized eye-tracking templates for expert pilots, characterized in that, Includes the following steps: Step 1: Determine the special flight scenario, collect eye-tracking data from expert pilots, simultaneously record the flight parameters output by the flight simulator, and conduct post-flight replay interviews; Step 2: Preprocess the raw eye-tracking data, including blink detection and interpolation based on pupil change velocity, and I-VT velocity threshold noise filtering, and output the effective fixation point sequence; Step 3: Sort the expert samples based on the flight control deviation index, and select the top preset proportion with the smallest deviation as candidate high-quality samples; construct the gaze distribution feature vector of each candidate high-quality sample, calculate the Euclidean distance between the gaze distribution feature vector and the average vector of the expert group, and select the sample with the smallest Euclidean distance as the benchmark expert sample. Step 4: Extract flight parameters, use key state switching points as segment trigger points, divide the flight process into multiple cognitive stages, and obtain segmented eye-tracking data corresponding to each cognitive stage; Step 5: Perform two-dimensional modeling on the segmented eye-tracking data of each cognitive stage, construct a static interest region fixation weight model and a dynamic Markov chain transition matrix model to form a two-dimensional cognitive model; Step 6: Obtain the first-person cockpit video corresponding to the benchmark expert sample, and based on the dual-dimensional cognitive model, generate a gaze point overlay, visual enhancement cues, scanning path trails and synchronous auditory scripts, perform multimodal integration, and output a multimodal enhanced eye-tracking teaching template. In step 5, the construction of the static interest region gaze weight model includes: calculating the gaze contribution rate of each interest region within each cognitive stage. The gaze contribution rate is the proportion of the cumulative gaze duration of the interest region to the total gaze duration of the cognitive stage, and its calculation expression is: （3）； In the formula, The gaze contribution rate for the i-th region of interest. The cumulative gaze duration for this region of interest. Let n be the cumulative gaze duration of the j-th region of interest; n is the total number of regions of interest. In step 6, the visual enhancement cues include: generating a shrinking aperture forward guidance cue 150 to 200 milliseconds before the expert gaze point reaches the region of critical interest; applying a spotlight effect when the expert gaze point enters the region of critical interest; and the synchronized auditory script is based on post-replay interviews and is strictly time-synchronized with the eye-tracking events.

2. The data-driven method for constructing contextualized eye-tracking templates for expert pilots according to claim 1, characterized in that, Step 2, specifically, includes blink detection and interpolation based on pupil change rate: Step 21: Sample eye-tracking data for a special flight scenario at a preset frequency to obtain several sampling points; Step 22: Obtain the pupil diameter at each sampling point, and take the derivative of the pupil diameter at all sampling points to obtain the pupil change rate curve; Step 23: Obtain the minimum and maximum values ​​in the pupil change rate curve. Take the last sampling point close to 0 before the minimum value as the start time of blinking, and take the first sampling point close to 0 after the maximum value as the end time of blinking. Step 24: Set the moment the blink begins and the moment the blink ends The time interval between blinks is defined as the blink interval. Linear interpolation or polynomial interpolation is used to fill in the pupil diameter and fixation position within the blink interval. The calculation expression is as follows: （1）； in, and , respectively, are the gaze position or pupil diameter at the start and end of the blink, and t is the time to be interpolated within the blink interval; This represents the compensation data value calculated by interpolation at time t within the blink interval; when x represents the pupil diameter, This represents the interpolated pupil diameter at that moment; when x represents the fixation position, The interpolated gaze position corresponds to this moment; the pupil diameter time series is composed of the interpolated pupil diameters from all sampling points, and the gaze position time series is composed of the interpolated gaze positions from all sampling points; compensated data values. Used to replace missing or abnormal data caused by blinking, so that the fixation position time series and pupil diameter time series within the blink interval remain continuous; Step 25: Apply the I-VT algorithm to calculate the gaze angular velocity between adjacent sampling points, set a velocity threshold to identify saccades, suppress and hide high-speed visual jitters through saccades, and retain only valid gaze points whose duration exceeds the preset duration threshold to form the final gaze position time series.

3. The data-driven method for constructing contextualized eye-tracking templates for expert pilots according to claim 1, characterized in that, In step 3, the flight control deviation index includes one or more of the following: root mean square pitch angle deviation, heading maintenance deviation, altitude deviation, and landing point deviation; the pre-preset ratio is the first 20%; the gaze distribution feature vector includes the gaze time percentage of each region of interest. Euclidean distance The calculation expression is: （2）； In the formula, The proportion of gaze time in the i-th region of interest within the gaze distribution feature vector of candidate high-quality samples. represents the percentage of fixation time for the i-th region of interest in the average vector of the expert group, where n is the total number of regions of interest.

4. The data-driven method for constructing contextualized eye-tracking templates for expert pilots according to claim 1, characterized in that, In step 4, the key state switching points are determined by identifying actual operational actions, including the moment when an emergency occurs, the moment when the flap retraction action is actually performed, and the moment when the leveling altitude is reached.

5. The data-driven method for constructing contextualized eye-tracking templates for expert pilots according to claim 1, characterized in that, In step 5, the construction of the dynamic Markov chain transition matrix model includes: calculating the gaze transition probability matrix between each region of interest, where the expression for calculating the transition probability is: （4）； In the formula, Let i be the transition probability from region of interest i to region of interest j. Let n be the number of transitions from region of interest i to region of interest j, and n be the total number of regions of interest.

6. The data-driven method for constructing contextualized eye-tracking templates for expert pilots according to claim 1, characterized in that, In step 6, the gaze point overlay is presented as a semi-transparent ring with a diameter corresponding to a 2-degree field of view, and the color is magenta or cyan; the fade-out duration of the scanning path trail is 500 to 600 milliseconds.

7. A template construction system for the data-driven, contextualized eye-tracking template construction method for expert pilots as described in any one of claims 1-6, characterized in that, include: The data acquisition module is used to determine special flight scenarios. It collects eye-tracking data from expert pilots through eye-tracking devices, synchronously records the flight parameters output by the flight simulator, and conducts post-event replay interviews after the scenario is completed. The preprocessing module is used to sample the raw eye-tracking data, calculate the pupil diameter derivative, detect blinks, perform interpolation and fill in the gaps, and filter I-VT velocity threshold noise to output a valid gaze sequence. The benchmark expert sample screening module is used to sort and filter candidate high-quality samples according to flight control deviation index, construct gaze distribution feature vector, calculate Euclidean distance to evaluate typicality, verify signal quality, and output benchmark expert samples. The event-driven segmentation module is used to extract flight parameters, identify key state transition points as segmentation trigger points, and divide the flight process into multiple cognitive stages. A two-dimensional modeling module is used to construct a static interest region gaze weight model and a dynamic Markov chain transition matrix model for each cognitive stage. The template generation module is used to acquire first-person perspective video, generate foveation overlay, visual enhancement cues, scan path trails and synchronized auditory scripts, and perform multimodal integration to output the final teaching template.

8. The template construction system according to claim 7, characterized in that, The specific details of each module are as follows: 1) The data acquisition module includes a scene determination unit, an eye-tracking acquisition unit, a flight parameter recording unit, and an interview recording unit; The scenario determination unit is used to determine the flight scenario to be collected based on the training objective and the type of special situation; The eye-tracking acquisition unit is used to collect eye-tracking data of expert pilots during special flight missions, including gaze position, gaze duration, saccade trajectory, and pupil diameter, through an eye-tracking device. The flight parameter recording unit is used to synchronously record the flight altitude, speed, heading, attitude, control input, and system status output by the flight simulator; The interview recording unit is used to conduct post-flight replay interviews after the flight scenario is completed, recording expert pilots' explanations of key observation points, attention allocation strategies, operational intentions, and judgment criteria. 2) The preprocessing module includes a sampling unit, a pupil differentiation unit, a blink detection unit, an interpolation unit, and a noise filtering unit; The sampling unit is used to sample or resample the original eye-tracking data at a preset frequency to form a continuous sampling point sequence. The pupil differentiation unit is used to obtain the pupil diameter at each sampling point and calculate the pupil change rate curve; The blink detection unit is used to identify the start and end times of blinking based on feature points in the pupil change rate curve; The interpolation unit is used to interpolate and fill in missing or abnormal gaze positions and pupil diameters within the blink interval; The noise filtering unit is used to remove abnormal jump points and invalid eye movement data using the I-VT velocity thresholding method, and outputs a valid fixation point sequence. 3) The benchmark expert sample screening module includes a performance filtering unit, a feature vector construction unit, an Euclidean distance calculation unit, and a signal quality verification unit; The performance filtering unit is used to sort and filter expert samples according to flight control deviation indicators to obtain candidate high-quality samples; The feature vector construction unit is used to construct a gaze distribution feature vector of candidate high-quality samples based on eye movement features including the gaze time ratio of each region of interest. The Euclidean distance calculation unit is used to calculate the Euclidean distance between the gaze distribution feature vector of the candidate high-quality sample and the average vector of the expert group, so as to evaluate the typicality of the sample gaze pattern. The signal quality verification unit is used to verify the integrity, effective sampling rate, missing proportion and synchronization quality of eye-tracking data of candidate samples, and output benchmark expert samples that meet the requirements. 4) The event-driven segmentation module includes a flight parameter extraction unit, a trigger point identification unit, and a segmented execution unit; The flight parameter extraction unit is used to extract key flight parameters and operational status parameters related to the emergency response process from the data recorded by the flight simulator. The trigger point identification unit is used to determine segmented trigger points based on key state switching points, including the time of special event occurrence, the time of actual operation, and the time of reaching the predetermined flight state. The segmented execution unit is used to divide the complete flight process into multiple cognitive stages based on the segmented trigger points, and obtain segmented eye-tracking data corresponding to each cognitive stage. 5) The dual-dimensional modeling module includes a static interest region weight modeling unit and a dynamic Markov transition modeling unit; The static interest region weight modeling unit is used to calculate the cumulative fixation duration and fixation contribution rate of different interest regions in each cognitive stage, and to construct a static interest region fixation weight model. The dynamic Markov transition modeling unit is used to count the number of gaze transitions and transition probabilities between different regions of interest for expert pilots, and to construct a dynamic Markov chain transition matrix model, thereby characterizing the dynamic visual scanning patterns of expert pilots during emergency handling. 6) The template generation module includes a video acquisition unit, a gaze overlay unit, a visual enhancement unit, an auditory script unit, and a multimodal integration unit; The video acquisition unit is used to acquire first-view cockpit videos corresponding to benchmark expert samples. The gaze overlay unit is used to overlay the expert pilot's gaze point, gaze trajectory, or gaze hotspot onto the first-person perspective video; The visual enhancement unit is used to generate visual enhancement cues based on key regions of interest and expert gaze paths to highlight instruments, areas, or risk cues that learners need to focus on. The auditory scripting unit is used to generate an auditory cueing script synchronized with the video based on the replay of the interview content and eye movement events. The multimodal integration unit is used to integrate first-view video, fixation overlay, visual enhancement cues, scan path trails, and synchronized auditory scripts to generate a multimodal enhanced eye-tracking teaching template.

Images