A method and system for identifying pilot EBT training target competency driven by eye-tracking data and flight data fusion.

Abstract

This invention discloses a method and system for identifying pilot EBT training target competencies driven by the fusion of eye-tracking data and flight data. The method includes: S0: acquiring eye-tracking data from the pilot's EBT training process, constructing a GDETC network to achieve spatiotemporal trajectory clustering of the eye-tracking data, and extracting the pilot's gaze spatiotemporal trajectory; S1: extracting competency assessment scores and error spatiotemporal point data from the EBT competency assessment worksheet and flight data; S2: constructing a VAE-STRF network, fusing the data extracted in S0 and S1, extracting the pilot's key competency assessment score matrix, establishing a pilot key competency assessment model, and outputting the target competencies for EBT training. This invention identifies the target competencies that pilots need to further improve through eye-tracking data during EBT training, replacing existing manual evaluation systems and providing effective decision support for improving the targeting of pilot training.

Description

Technical Field

[0001] This invention belongs to the field of aircraft pilot training and physiological data analysis, specifically relating to a method and system for identifying pilot EBT training target competency driven by the fusion of eye-tracking data and flight data. Background Technology

[0002] Aviation safety is the core cornerstone of the sustainable development of the civil aviation industry, and the comprehensive competence of pilots is the last line of defense for ensuring safe flight operations. Currently, the primary method for evaluating pilots' comprehensive competence is through manual assessment of nine key competencies. Competency assessment can effectively predict and measure pilots' work performance, thus becoming an important basis for pilot training and evaluation. In the currently implemented Evidence-Based Training (EBT), instructors need to assess trainees' competence by observing their behavior in performing activities or tasks using relevant technical and non-technical knowledge, skills, and attitudes under specific conditions. This assessment determines whether trainees can effectively perform their work and demonstrate the required proficiency, and the results form part of the EBT data chain. Currently, the determination of target competencies still relies on the instructor's subjective evaluation. Before each training session, the simulator instructor determines the "key competencies" based on previous training results as the "target competencies" for that training session, and selects applicable scenarios as needed. After the refresher training, which is conducted in three sessions, the simulator instructor will determine the "key competencies" that need to be improved based on the trainees' performance, which will serve as the "target competencies" for subsequent courses.

[0003] While subjective evaluation methods can meet basic training needs, their heavy reliance on instructors' subjective judgment makes it difficult to fully ensure the scientific rigor and consistency of target competency determination. Data fusion-driven approaches are a crucial direction for pilot competency identification. Data-driven approaches are the most important principle in EBT implementation; the training frequency and framework of training topics in EBT must be determined based on data and analysis reports. Outputting high-quality training data is also a key objective of EBT. The data output from each training session becomes one of the data sources for the next EBT cycle, is transformed into training requirements, and ultimately forms a closed loop. To achieve data-driven approaches through reliable training and evaluation data, the data transmission chain needs to be fully established, allowing data flow to be guided by objective data links. Therefore, developing an evaluation system for determining target competency based on objective data has become an urgent need for civil aviation industry management. Summary of the Invention

[0004] To streamline the data flow of EBT training and enable data-driven EBT training, addressing the issues of relying primarily on instructors' subjective evaluations to determine target competencies and the lack of objective data assessment support for pilot EBT training, this invention provides a method and system for identifying pilot EBT training target competencies driven by the fusion of eye-tracking and flight data. This system can supplement and improve the existing manual evaluation system using multi-source data fusion evaluation, providing effective decision support for enhancing the targeting of pilot training.

[0005] To achieve the above objectives, the present invention provides the following solution:

[0006] A method for identifying pilot EBT training target competency driven by eye-tracking data and flight data fusion, the method comprising:

[0007] S0: Acquire eye-tracking data of pilots during EBT training, construct GDETC network to achieve spatiotemporal trajectory clustering of eye-tracking data, and extract the spatiotemporal trajectory of pilots' gaze;

[0008] S1: Obtain the trained EBT competency assessment worksheet, flight data, standard baseline flight data and parameter thresholds, construct a multi-source data fusion hierarchical extraction network, and extract the flight data stream of associated error spatiotemporal points, the associated score and the OBj data stream of competency;

[0009] S2: Construct a VAE-STRF network, fuse the data extracted from S0 and S1, extract the pilot key competency assessment score matrix, establish a pilot key competency assessment model, and output the target competency for EBT training.

[0010] Preferably, in step S0, acquiring eye-tracking data during the pilot's EBT training process, constructing a GDETC network to achieve spatiotemporal trajectory clustering of the eye-tracking data, and extracting the pilot's gaze spatiotemporal trajectory includes:

[0011] S0.1: Collect eye movement data of pilots during EBT training, segment the cockpit visual space, correlate the in-cockpit visual space information with eye movement data, and extract a set of spatiotemporally related eye movement features.

[0012] S0.2: The eye-tracking feature set is sliced ​​using a sliding dynamic time window technique. Dynamic time warping is used to align non-linearly changing eye-tracking data across different time segments. Point-by-point matching is then used to calculate similarity, ensuring data accuracy and capturing subtle differences between trajectories, thus forming an eye-tracking spatiotemporal trajectory similarity matrix D. DTW_0 Set a similarity threshold θ DTW Filter out invalid features to form the trajectory similarity matrix D for the next input step. DTW_new ;

[0013] S0.3: The eye movement features G after slicing k The similarity matrix D of the gaze trajectory DTW_new The constructed GDETC network is input, and the density of the trajectories is calculated. All time-segment trajectories are clustered into multiple clusters, each cluster representing a gaze behavior pattern. Low-density trajectory clusters represent atypical behaviors, while high-density trajectory clusters represent gaze behavior patterns. The spatiotemporal gaze trajectory feature F is constructed from the feature data of the centers of the high-density trajectory clusters. gaze ;

[0014] Among them, the spatiotemporally correlated eye-tracking feature set is: G EBT =D EBT ∪A cockpit In the formula, A cockpit For the eye-tracking visual spatial characteristics of the cockpit, D EBT For eye-tracking data;

[0015] Trajectory Similarity Matrix D DTW_new for: In the formula, the retained element D DTW_new (m,n) represents pairs of eye-tracking spatiotemporal segments with high similarity, which retain more gaze trajectory patterns;

[0016] Gaze at the spatiotemporal trajectory feature F gaze for: In the formula, k represents the trajectory segment number, i represents the trajectory point number, and j represents the region number. This represents the coordinates of the i-th gaze point of the k-th trajectory segment. This represents the eye movement feature vector at the i-th fixation point of the k-th trajectory segment. This represents the time point of the i-th gaze point in the k-th trajectory segment. This represents the characteristic vector of the i-th gaze point in the k-th trajectory segment. This represents the vector of the i-th gaze point in the k-th trajectory segment. This represents the j-th visual region associated with the i-th gaze point of the k-th trajectory segment. This indicates that the i-th gaze point of the k-th trajectory is in the region. Total length of stay This indicates that the i-th gaze point of the k-th trajectory is in the region. Number of fixations This indicates that the i-th gaze point of the k-th trajectory is in the region. Average gaze duration This represents the region switching frequency of the i-th gaze point in the k-th trajectory segment.

[0017] Preferably, in step S1, acquiring the trained EBT competency assessment worksheet, flight data, standard baseline flight data, and parameter thresholds, constructing a multi-source data fusion hierarchical extraction network, and extracting the flight data stream of associated error spatiotemporal points, the associated score, and the OBj data stream of competency include:

[0018] S1.1: The deviation threshold of flight parameters is determined by reference standards and based on expert assessment. max_deviation and standard baseline flight data Q base Acquire simulator flight data Q fobs With post-training EBT competency assessment worksheet D FSTD ;

[0019] S1.2: Construct the bias mapping architecture I of the SFHE network and extract the flight data stream Q of the structured data association error spatiotemporal points. error ;

[0020] S1.3: Construct the SFHE network structure data extraction architecture II, inputting FSTD data stream D. FSTD Extract the associated scores and competency data stream C from the EBT assessment worksheet. score ;

[0021] Among them, the post-training EBT competency assessment worksheet D FSTD For: D FSTD ={D1,D2,...,D n}, where n is the total number of FSTD work orders included, and D n For a single FSTD text data;

[0022] Flight data stream Q error for: In the formula, t is time, a is a parameter, j is position, and R j For the cockpit space mapping function, T error For the set of operational error times, P error P is the set of operational error parameters. error (t) is the set of error locations, t n Represents flight data stream Q error The time point associated with the nth error point in the data, a dn Represents flight data stream Q error The parameter a associated with the nth error point in the data is the parameter a that indicates the occurrence of the error. d R jn Represents flight data stream Q error The cockpit space location R associated with the nth error point in the error. j ;

[0023] OBj data stream C scorefor: In the formula C i OB represents the i-th type of competence. i,j S represents the j-th sub-item of the i-th competency category. i,j S represents the score corresponding to the j-th sub-item under the i-th competency category. i,j ∈[1,5], where i is the competency category index with a total of 9 categories, and j is the sub-item index under the corresponding category.

[0024] Preferably, in step S2, a VAE-STRF network is constructed to fuse the data extracted from S0 and S1, extract the pilot key competency assessment score matrix, establish a pilot key competency assessment model, and output the target competencies for EBT training, including:

[0025] S2.1: Based on the variational autoencoder (VAE), the gaze spatiotemporal trajectory features F gaze OBj data flow Q related to scores and competence error Flight data stream C of associated error spatiotemporal points score Feature fusion is performed to form an eye-tracking feature label dataset F containing competency information. G ;

[0026] S2.2: Establish the score matrix E describing the competency assessment comp A scoring network with a spatiotemporal random forest network (STRF) as its kernel is constructed, and the input is eye-tracking features F containing competency information labels. G Output the predicted competency assessment score matrix E comp ;

[0027] S2.3: Competency assessment score matrix E comp To establish a core pilot key competency assessment model, a scoring threshold was set to identify missing competency items (OBs). i，j And generate the competency gap matrix D OBj This enables the identification of competency deficiencies and outputs the target competencies for EBT training.

[0028] Among them, the eye-tracking feature label dataset F G for:

[0029]

[0030] In the formula, f Gn Let x represent a feature vector in the dataset. n ,y n ,t n f represents the spatiotemporal location of eye movement features. n ,p n ,n n f represents the latent variables fused by the encoder.n It is an eye-tracking feature vector, p n It is the characteristic vector of the gaze point, n n gaze vector, R j C represents the area label related to competence. i ,OB i,j ,S i,j Indicates competency rating labels;

[0031] Competency assessment score matrix E comp for:

[0032]

[0033] In the formula, matrix E comp The element represents the score of the j-th sub-item of the i-th competency category, where i is the competency category index (9 categories in total), and j is the sub-item index under the corresponding category (j∈[j1,j9]). comp (i,j)∈[1,5];

[0034] Output competency gap matrix D OBj for:

[0035]

[0036] In the formula, element OB i,j This refers to the competencies that pilots lack during EBT training, and the missing competencies are the target competencies for the next EBT training.

[0037] The present invention also provides a method and system for identifying pilot EBT training target competency driven by the fusion of eye-tracking data and flight data. The system is used to implement the method described in any one of the inventions. The system includes: a first extraction module, a second extraction module, and an evaluation module.

[0038] The first extraction module is used to acquire eye-tracking data of the pilot during EBT training, construct a GDETC network to achieve spatiotemporal trajectory clustering of eye-tracking data, and extract the pilot's gaze spatiotemporal trajectory;

[0039] The second extraction module is used to acquire the trained EBT competency assessment worksheet, flight data, standard baseline flight data and parameter thresholds, construct a multi-source data fusion hierarchical extraction network, and extract the flight data stream of associated error spatiotemporal points, the associated score and the OBj data stream of competency;

[0040] The evaluation module is used to construct a VAE-STRF network, fuse the data extracted from the first extraction module and the second extraction module, extract the pilot key competency evaluation score matrix, establish a pilot key competency evaluation model, and output the target competency for EBT training.

[0041] Preferably, the first extraction module includes: an eye-tracking feature set extraction unit, a trajectory similarity matrix generation unit, and a gaze spatiotemporal trajectory feature extraction unit;

[0042] The eye movement feature set extraction unit is used to collect eye movement data of pilots during EBT training, segment the cockpit visual space, associate the in-cabin visual space information with eye movement data, and extract the spatiotemporally correlated eye movement feature set.

[0043] The trajectory similarity matrix generation unit is used to slice the eye-tracking feature set using a sliding dynamic time window technique, align non-linearly changing eye-tracking data in different time segments using dynamic time warping, and perform similarity calculations through point-by-point matching to ensure data accuracy and capture minute differences between trajectories, thus forming an eye-tracking spatiotemporal trajectory similarity matrix D. DTW_0 Set a similarity threshold θ DTW Filter out invalid features to form the trajectory similarity matrix D for the next input step. DTW_new ;

[0044] The gaze spatiotemporal trajectory feature extraction unit is used to extract the sliced ​​eye movement features G k The similarity matrix D of the gaze trajectory DTW_new The constructed GDETC network is input, and the density of the trajectories is calculated. All time-segment trajectories are clustered into multiple clusters, each cluster representing a gaze behavior pattern. Low-density trajectory clusters represent atypical behaviors, while high-density trajectory clusters represent gaze behavior patterns. The spatiotemporal gaze trajectory feature F is constructed from the feature data of the centers of the high-density trajectory clusters. gaze ;

[0045] Among them, the spatiotemporally correlated eye-tracking feature set is: G EBT =D EBT ∪A cockpit In the formula, A cockpit For the eye-tracking visual spatial characteristics of the cockpit, D EBT For eye-tracking data;

[0046] Trajectory Similarity Matrix D DTW_new for: In the formula, the retained element D DTW_new (m,n) represents pairs of eye-tracking spatiotemporal segments with high similarity, which retain more gaze trajectory patterns;

[0047] Gaze at the spatiotemporal trajectory feature F gaze for: In the formula, k represents the trajectory segment number, i represents the trajectory point number, and j represents the region number. This represents the coordinates of the i-th gaze point of the k-th trajectory segment. This represents the eye movement feature vector at the i-th fixation point of the k-th trajectory segment. This represents the time point of the i-th gaze point in the k-th trajectory segment. This represents the characteristic vector of the i-th gaze point in the k-th trajectory segment. This represents the vector of the i-th gaze point in the k-th trajectory segment. This represents the j-th visual region associated with the i-th gaze point of the k-th trajectory segment. This indicates that the i-th gaze point of the k-th trajectory is in the region. Total length of stay This indicates that the i-th gaze point of the k-th trajectory is in the region. Number of fixations This indicates that the i-th gaze point of the k-th trajectory is in the region. Average gaze duration This represents the region switching frequency of the i-th gaze point in the k-th trajectory segment.

[0048] Preferably, the second extraction module includes: a flight data and evaluation work order acquisition unit, a flight data stream extraction unit, and an OBj data stream extraction unit;

[0049] The flight data and evaluation worksheet acquisition unit is used to determine the deviation threshold of flight parameters based on reference standards and expert assessment. max_deviation and standard baseline flight data Q base Acquire simulator flight data Q fobs With post-training EBT competency assessment worksheet D FSTD ;

[0050] The flight data stream extraction unit is used to construct the deviation mapping architecture I of the SFHE network and extract the flight data stream Q of the structured data-associated error spatiotemporal points. error ;

[0051] The OBj data stream extraction unit is used to construct the SFHE network's structured data extraction architecture II, and inputs the FSTD data stream D. FSTD Extract the associated scores and competency data stream C from the EBT assessment worksheet. score ;

[0052] Among them, the post-training EBT competency assessment worksheet D FSTD For: D FSTD ={D1,D2,...,D n}, where n is the total number of FSTD work orders included, and D n For a single FSTD text data;

[0053] Flight data stream Q error for: In the formula, t is time, a is a parameter, j is position, and R j For the cockpit space mapping function, T error For the set of operational error times, P error P is the set of operational error parameters. error (t) is the set of error locations, t n Represents flight data stream Q error The time point associated with the nth error point in the data, a dn Represents flight data stream Q error The parameter a associated with the nth error point in the data is the parameter a that indicates the occurrence of the error. d R jn Represents flight data stream Q error The cockpit space location R associated with the nth error point in the error. j ;

[0054] OBj data stream C score for: In the formula C i OB represents the i-th type of competence. i,j S represents the j-th sub-item of the i-th competency category. i,j S represents the score corresponding to the j-th sub-item under the i-th competency category. i,j ∈[1,5], where i is the competency category index with a total of 9 categories, and j is the sub-item index under the corresponding category.

[0055] Preferably, the evaluation module includes: an eye-tracking feature label dataset generation unit, a competency evaluation score matrix generation unit, and a competency evaluation score matrix generation unit;

[0056] The eye-tracking feature labeling dataset generation unit is used to generate gaze spatiotemporal trajectory features F based on a variational autoencoder (VAE). gaze OBj data flow Q related to scores and competence error Flight data stream C of associated error spatiotemporal points score Feature fusion is performed to form an eye-tracking feature label dataset F containing competency information. G ;

[0057] The competency assessment score matrix generation unit is used to establish a score matrix E describing the competency assessment. comp A scoring network with a spatiotemporal random forest network (STRF) as its kernel is constructed, and the input is eye-tracking features F containing competency information labels. G Output the predicted competency assessment score matrix E comp ;

[0058] The competency assessment score matrix generation unit is used to generate the competency assessment score matrix E. compTo establish a core pilot key competency assessment model, a scoring threshold was set to identify missing competency items (OBs). i,j And generate the competency gap matrix D OBj This enables the identification of competency deficiencies and outputs the target competencies for EBT training.

[0059] Among them, the eye-tracking feature label dataset F G for:

[0060]

[0061] In the formula, f Gn Let x represent a feature vector in the dataset. n ,y n ,t n f represents the spatiotemporal location of eye movement features. n ,p n ,n n f represents the latent variables fused by the encoder. n It is an eye-tracking feature vector, p n It is the characteristic vector of the gaze point, n n gaze vector, R j C represents the area label related to competence. i ,OB i,j ,S i,j Indicates competency rating labels;

[0062] Competency assessment score matrix E comp for:

[0063]

[0064] In the formula, matrix E comp The element represents the score of the j-th sub-item of the i-th competency category, where i is the competency category index (9 categories in total), and j is the sub-item index under the corresponding category (j∈[j1,j9]). comp (i,j)∈[1,5];

[0065] Output competency gap matrix D OBj for:

[0066]

[0067] In the formula, element OB i,j This refers to the competencies that pilots lack during EBT training, and the missing competencies are the target competencies for the next EBT training.

[0068] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0069] This invention discloses a method and system for identifying pilot EBT training target competency driven by the fusion of eye-tracking data and flight data. Specifically, it includes the following steps: S0: Acquire eye-tracking data during pilot training, extract spatiotemporally correlated eye-tracking features, slice the data using a dynamic time window, calculate the gaze trajectory similarity matrix based on dynamic time warping, construct a GDETC network to achieve spatiotemporal trajectory clustering of eye-tracking features, and extract the pilot's gaze spatiotemporal trajectory; S1: Acquire the trained EBT competency assessment worksheet, flight data, standard baseline flight data, and parameter thresholds, construct a multi-source data fusion hierarchical extraction network, and extract the flight data stream associated with spatiotemporal errors, the associated scores, and the OBj data stream of competency; S2: Construct a VAE-STRF network, fuse the data extracted in S0 and S1, extract the pilot's key competency assessment score matrix, establish a pilot's key competency assessment model, and output the target competency of EBT training. This invention aims to identify the target competencies that pilots need to continue to improve by using eye-tracking data during pilot EBT training. It can replace the existing manual evaluation system and provide effective decision support for improving the targeting of pilot training. Attached Figure Description

[0070] To more clearly illustrate the technical solution of the present invention, the drawings used in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0071] Figure 1 This is a schematic diagram of the main steps S0-S2 described in the embodiments of the present invention;

[0072] Figure 2 This is a schematic flowchart of a method for identifying pilot EBT training target competency driven by the fusion of eye-tracking data and flight data, according to an embodiment of the present invention. Detailed Implementation

[0073] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0074] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0075] Example 1

[0076] like Figure 1 , Figure 2 As shown, this invention provides a method for identifying pilot EBT training target competency driven by the fusion of eye-tracking data and flight data, comprising the following steps:

[0077] S0: Acquire eye movement data during pilot training, extract spatiotemporally correlated eye movement features, slice the data using a dynamic time window, calculate the eye movement trajectory similarity matrix based on DTW, construct a GDETC network to achieve spatiotemporal trajectory clustering of eye movement features, and extract the gaze spatiotemporal trajectory.

[0078] The specific steps are as follows:

[0079] S0.1 Collect eye-tracking data of the pilot during EBT training. EBT The cockpit visual space is segmented into regions, and the visual space information A inside the cockpit is divided into regions. cockpit By correlating with eye-tracking data, a spatiotemporally correlated eye-tracking feature set G is extracted. EBT .

[0080] Eye-tracking data D input by the system EBT =D blink_t ∪D blink_i The continuous dynamic time series D associated with time t blink_t ={D blink_t ={(x t ,f t The static feature set D of discrete gaze points of the associated sequence i, where t ∈ [t0, T] and t ∈ [t0, T] are given. blink_i ={(n i ,p i The union of x and i is represented by the set x ∈ [1, N]. In a dynamic time series, x t The coordinate x representing the eye movement trajectory t =(x t ,y t ), where x t ,y t f represents the coordinates of the gaze point at time t; t The time-dependent eye-tracking feature vector f t =(v t ,a t ,φ t ,p t ), where v t Indicates the speed of saccades, a t φ represents saccadic acceleration. t Indicates the direction of scanning, p t represents the pupil diameter; t represents the time point of data acquisition. There are N fixation points in the static feature set, n... i p represents the i-th fixation point;i =(t i ,l i ,r i ,g i ,f i ) represents the characteristic of the i-th gaze point itself, where t i l represents the fixation duration at the i-th fixation point. i r represents the saccade path length between two consecutive fixations. i gaze convergence, g i The area where the point of fixation is located.

[0081] The pilot's gaze area in the cockpit is divided into M different areas R according to instruments, gaze angle, and whether it is inside or outside the cockpit. cockpit_j ={R j |j∈[1,M]}, cockpit eye-tracking visual spatial features A including regional characteristics cockpit ={(R j ,τ j ,c j ,μ j ,s t ) |j∈[1,M],t∈[t0,T]}, where R j τ represents the j-th visual region within the cockpit. j This indicates that the line of sight is in region R. j Total stay time, c j This indicates that the line of sight is in region R. j Number of fixations, μ j This indicates that the line of sight is in region R. j Average gaze duration, s t Indicates the area switching frequency.

[0082] Cockpit Visual Spatial Information A cockpit With eye-tracking data D EBT By associating them through union, eye-tracking data is made to include information about the cabin environment, forming a spatiotemporally correlated set of eye-tracking features G. EBT =D EBT ∪A cockpit .

[0083] S0.2 uses the sliding dynamic time window technique to analyze the spatiotemporally correlated eye-tracking feature set G. EBT The data segments were sliced, and the similarity of eye-tracking spatiotemporal trajectories was calculated using DTW on the sliced ​​data segments.

[0084] Based on a fixed time t eyetrack_0 Percentage of scans within P glance_all As an indicator of gaze state, the sliding window length ΔT is dynamically set. blink and sliding step size δT blinkThe set of spatiotemporally correlated eye-tracking features G collected over a long period of time. EBT Slicing is performed on the spatiotemporally correlated eye-tracking feature dataset G. EBT The time segment is divided into K time segments G by a sliding dynamic time window. EBT_k ={G k [t k :t k +ΔT blink ] |k∈[0,K]};When the gaze state index P glance_all A value ≤0.5 indicates that the pilot's gaze is changing rapidly, and a smaller sliding window length ΔT should be set. blink and sliding step size δT blink To capture behavioral characteristics during rapid switching states; when P glance_all When the value is greater than 0.5, set a larger sliding window length ΔT. blink and sliding step size δT blink The continuous slice features are preserved. The eye-tracking data segment after dynamic slicing is as follows:

[0085] G k ={(x t ,f t ,t)∪(n i ,p i )∪(R j ,τ j ,c j ,μ j ,s t )∣t∈[t k ,t k +ΔT blink The dataset G of eye-tracking segments with associated spatiotemporal features is obtained by considering the following relationships: i∈[1,N],j∈[1,M],k∈[0,K]. EBT_k .

[0086] DTW (Time-Digital Wavelength Wrapping) is used to align nonlinear eye-tracking data from different time segments, and similarity calculations are performed through point-by-point matching to ensure data accuracy and capture minute differences between trajectories, forming an eye-tracking spatiotemporal trajectory similarity matrix D. DTW_0 Set a similarity threshold θ DTW Filter out invalid features to form the trajectory similarity matrix D for the next input step. DTW_new .

[0087] For two different eye-tracking spatiotemporal trajectory segments G m and G n Define its eye-tracking trajectory comprehensive deviation d m :

[0088]

[0089] Where i is the time segment G m The index of the i-th trajectory point indicates that the current calculation involves the i-th trajectory point, i∈[1,N]. m ], N m It is time segment G m The number of trajectory points in G; j is the time segment G n The index of the j-th trajectory point indicates that the current calculation involves the j-th trajectory point, j∈[1,N]. n ], N n It is time segment G n The number of trajectory points in the [data]. Where [the data is]... It is a dynamic time series feature distance, used to represent the spatial difference between two segment trajectory points, including the spatial distance between the two segment trajectory points. and dynamic feature vector v t ,a t ,φ t ,p t Euclidean distance between It is the static fixation feature distance, used to represent the fixation feature vector p. m and p n The Euclidean distance between them It is the characteristic distance of the cockpit area, which is represented by the normalized weighted Euclidean distance.

[0090] Three weight parameters, w1, w2, and w3, are defined in the range [0,1], such that the total weight normalization is w1 + w2 + w3 = 1. Initial values ​​are set for w1 = 0.5, w2 = 0.3, and w3 = 0.2. A grid search is performed, with a search step size δ. w =0.05 Optimize the weights on the training set.

[0091] Construct the cumulative cost matrix D STEM_ac Matrix D STEM_ac The elements are calculated progressively from the top left corner to the bottom right corner using an accumulative method. The matrix is ​​initialized to a size of N. m ×N n N m N n It is time segment G m and G n Number of trajectory points in:

[0092] The recursive formula for the cumulative cost matrix and its boundary conditions:

[0093]

[0094] Matrix D STEM_ac elements The local matching cost represents the local similarity between two segments, and the overall matching cost is D. STEM_ac (N m N n DTW(G) represents the overall similarity between two time segments, specifically the similarity between the trajectories m and n of the two time segments. m G n ) = D STEM_ac (N m N n ).

[0095] Calculate the similarity between all eye-tracking time segment pairs to generate an initial gaze trajectory similarity matrix D. DTW_0 Matrix D DTW_0 Element D m,n =DTW(G m G n The matrix () represents the similarity between time segments (m,n), and its size is K×K.

[0096]

[0097] Set the gaze trajectory similarity threshold θ DTW All trajectories with a similarity greater than θ DTW The time segment (m, n) is excluded, and the discrimination condition for each element is:

[0098]

[0099] Initial gaze trajectory similarity matrix D DTW_0 After filtering out invalid feature pairs with low similarity, the gaze trajectory similarity matrix D is obtained. DTW_new :

[0100]

[0101] The element D that is retained DTW_new (m,n) represents pairs of eye-tracking spatiotemporal segments with high similarity, which retain more gaze trajectory patterns.

[0102] S0.3 Construct the GDETC network to extract gaze spatiotemporal trajectory features.

[0103] The sliced ​​eye movement features G k ={(x t ,f t ,t)∪(n i ,p i )∪(R j ,τ j ,c j ,μ j ,s tThe similarity matrix D between the gaze trajectory and the gaze trajectory DTW_new The constructed GDETC network is input, and the density of the trajectories is calculated. All time-segment trajectories are clustered into multiple clusters, each cluster representing a gaze behavior pattern. Low-density trajectory clusters represent atypical behaviors such as saccades, anomalies, and noise, while high-density trajectory clusters represent gaze behavior patterns. The spatiotemporal gaze trajectory feature F is constructed by using the feature data of the centers of the high-density trajectory clusters. gaze .

[0104] Through the gaze trajectory similarity matrix D DTW_new Calculate the density ρ of trajectory points k Calculate the density distance δ of the trajectory points k The system outputs the initial trajectory and its center point. Further, it optimizes the center of the initial trajectory cluster to generate more representative gaze behavior trajectory features, which are then fed into the gaze trajectory generator G. GDETC_track Input random noise vector z and initial trajectory cluster centers, generate trajectory cluster center feature vector V centre The center of the real trajectory cluster is compared with the generated trajectory input to the staring trajectory discriminator D. GDETC_track Determine if the input comes from a real trajectory cluster, iterate and train d times for the gaze trajectory generator G. GDETC_track Optimize and output the final trajectory cluster center features V centre_true The training is performed using a generative adversarial loss function, which is as follows:

[0105] L GDETC =E x～truth [logD GDETC_track (x)]+E z～noise [log(1-D GDETC_track (G GDETC_trackk (z)))](7)

[0106] Based on the generated trajectory cluster centers, trajectory clusters are reassigned, the similarity from trajectory points to cluster centers is calculated, and the trajectory cluster assignment results are updated until clustering converges, generating high-density trajectory clusters and outputting gaze behavior patterns. For each high-density trajectory cluster, the gaze behavior pattern is represented, and dynamic features are extracted from it. Static features Regional characteristics These gaze characteristics form a regular spatiotemporal trajectory feature of gaze. Where k represents the trajectory segment number, i represents the trajectory point number, and j represents the region number. This represents the coordinates of the i-th gaze point of the k-th trajectory segment. This represents the eye movement feature vector at the i-th fixation point of the k-th trajectory segment. This represents the time point of the i-th gaze point in the k-th trajectory segment. This represents the characteristic vector of the i-th gaze point in the k-th trajectory segment. This represents the vector of the i-th gaze point in the k-th trajectory segment. This represents the j-th visual region associated with the i-th gaze point of the k-th trajectory segment. This indicates that the i-th gaze point of the k-th trajectory is in the region. Total length of stay This indicates that the i-th gaze point of the k-th trajectory is in the region. Number of fixations This indicates that the i-th gaze point of the k-th trajectory is in the region. Average gaze duration This represents the region switching frequency of the i-th gaze point in the k-th trajectory segment.

[0107] S1: Obtain the trained EBT competency assessment worksheet, simulator flight data, standard baseline flight data, and parameter thresholds; construct the SFHE network; and extract the flight data stream associated with error spatiotemporal points, the associated score, and the competency OBj data stream C. score .

[0108] The specific steps are as follows:

[0109] S1.1 The deviation threshold of flight parameters is determined by reference standards and based on expert assessment. max_deviation and standard flight baseline data Q base Acquire simulator flight data Q fobs With post-training EBT competency assessment worksheet D FSTD .

[0110] Referencing aircraft flight quality monitoring standards and relying on flight safety experts to determine permissible deviation thresholds for flight parameters ∈ max_deviation ={∈1,∈2,…,∈ d} and standard flight baseline data It contains d parameters. This serves as the baseline for flight parameters. Flight data Q from the training phase is also acquired. fobs (t)={(q1(t), q2(t),...,q d (t))|t∈[0,T]}, which contains d parameters, q d (t) represents the parameters of the flight data during the training phase.

[0111] Acquire flight data during the training phase Q fobs (t) and the corresponding EBT competency assessment worksheet (FSTD) for training, with multiple EBT competency assessment worksheets constituting the FSTD data stream D. FSTD ={D1,D2,...,D n}, where n is the total number of FSTD work orders included, and D n This is a single FSTD text data set. The FSTD is completed by the course simulator instructor after assessing the trainee's overall competence performance according to the standards stipulated by the Civil Aviation Administration of China, recording the pilot's overall competence performance during EBT training. The competence score data in the FSTD provides competence labeling data for the eye-tracking data in this method.

[0112] S1.2 Constructs the bias mapping architecture I of the SFHE network, and extracts the structured data flight data stream Q of the associated error spatiotemporal points by calculating the weighted bias. error .

[0113] Input flight data Q fobs (t), Standard baseline flight data Q base (t) and cockpit area division data R cockpit_j By using the high-dimensional spatiotemporal parameter joint bias mapping of SFHE, the flight data stream Q of the associated error spatiotemporal points is output. error (t,j). Q error (t,a,R j It includes data on the location and timing of operational errors during simulator training, provides time labels for errors that lead to a lack of competence, and spatial labels for cockpit areas associated with the error parameters.

[0114] The weighted deviation ΔQ between flight operation data and standard baseline is calculated based on dynamic system high-dimensional mapping and the introduction of a weight function w. w_EBT (t):

[0115] ΔQ w_EBT (t)=w·(Q fobs (t)-Q base (t))={w1·Δa1(t),w2·Δa2(t),...,w d ·Δa d (t)}(8)

[0116] When |Δa d (t)|>∈ i When assuming parameter a d An error occurs at time t, forming a set of operational error times. and the set of operational error parameters P error (t)={a d :|Δa d (t)|>∈ i ,t∈T error Define the cockpit space mapping function f(a) d ) = R j , through f(a iThis establishes a correlation between operational error parameters and the position of the cockpit space, resulting in an error location set R. error (t)={f(a i ):a i ∈P error (t)}, the time t, parameter a, and position j of the integrated error data constitute the flight data stream Q of the associated error spatiotemporal point. error It contains n error points, where t n Represents flight data stream Q error The time point associated with the nth error point in the data, a dn Represents flight data stream Q error The parameter a associated with the nth error point in the data is the parameter a that indicates the occurrence of the error. d R jn Represents flight data stream Q error The cockpit space location R associated with the nth error point in the error. j ;

[0117]

[0118] S1.3 Construct the SFHE network structure data extraction architecture II, inputting FSTD data stream D. FSTD ={D1,D2,…,D n}, via SFHE v5 The module determines the table area and uses the table structure parsing module SFHE. FSR Extract cell data using SFHE embedding The module embeds the text content of cells into a multi-layered semantic space, using SFHE. classify The module classifies and groups competencies, behavioral indicators, and scores, and extracts the OBj data stream C of associated scores and competencies from the EBT assessment worksheet. score .

[0119] Define set C Competency ={C i,j |i∈[1,9],j∈[1,11]} describes the Obj sub-item corresponding to each competency in the framework of nine competencies and behavioral indicators, C i,j This represents the j-th sub-item of the i-th competency category.

[0120] Input FSTD data stream D FSTD ={D1,D2,...,D n}, via the SFHE network v5 Module to D FSTD PDF format D in the data stream n To locate the area of ​​the competency rating table and output the rectangular boundary R of the table. table_area =SFHEv5 (D i = {(x1,y1,x2,y2)}. This is achieved through the table structure parsing module SFHE. FSR Perform row and column segmentation on the image within the table area, and extract the position and content of each cell. Where R i,j T represents the coordinates of the cell in the i-th row and j-th column. i,j This represents the text content of the cell in the i-th row and j-th column. This represents the image of the cropped table area.

[0121] Through the semantic hierarchical embedding module SFHE embedding The cell content T i,j Mapped to a multi-layered semantic space Φ(T) of competency, behavioral indicators, and scores ij ):

[0122] Φ(T ij )=(Φ comp (T ij ),Φ obj (T ij ),Φ score (T ij (10)

[0123] Where Φ comp (T ij ) represents the competency embedding vector, Φ obs (T ij ) represents the embedding vector of the behavioral indicator, Φ score (T ij ) represents the embedding vector of the rating.

[0124] Furthermore, through SFHE classify The module will embed the vector Φ(T) ij The fully connected layer and Softmax function calculate the conditional probability distribution of the input text T, and the argmax operator selects the category or score with the highest conditional probability. ij Mapping to a multi-task semantic space to achieve semantic classification:

[0125]

[0126] in Embedding vector Φ representing competence comp (T ij The corresponding competency category, OB i,j The embedding vector Φ represents the behavioral indicator. obs (T ij The corresponding behavioral indicator categories, The embedding vector Φ represents the rating. score(T ij The corresponding score.

[0127] Obtain the OBj data stream C containing the associated scores and competencies. score :

[0128]

[0129] Where C i OB represents the i-th type of competence. i,j S represents the j-th sub-item of the i-th competency category. i,j S represents the score corresponding to the j-th sub-item under the i-th competency category. i,j ∈[1,5], where i is the competency category index with a total of 9 categories, and j is the sub-item index under the corresponding category.

[0130] S2: Construct a VAE-STRF network to fuse the data extracted from S0 and S1, and extract the pilot key competency assessment score matrix E. comp Establish a pilot key competency assessment model to identify competency deficiencies (D) OBj It can identify competency deficiencies and output them as target competencies for EBT training.

[0131] The specific steps are as follows:

[0132] S2.1 uses a variational autoencoder to extract gaze-spatiotemporal trajectory features F gaze OBj data stream Q error Flight data stream C of associated error spatiotemporal points score Feature fusion is performed to form an eye-tracking feature label dataset F containing competency information. G .

[0133] The three classes of data input from S0 and S1 are aligned to ensure feature fusion within the same time frame. Normalization is then used to adjust the F... gaze Standardized to Φ gaze Q error Standardized to Φ QAR C score Standardized to Φ OBJ Constructing a variational autoencoder with a multi-source feature module (VAE(Φ)). gaze ,Φ QAR ,Φ OBJ The features are fused to extract the potential low-dimensional representation of the high-dimensional features, while simultaneously reducing noise and removing redundant information.

[0134] The loss function of a variational autoencoder is defined as:

[0135]

[0136] Where DKL It is the KL divergence, used to regularize the latent space, q φ (z|x) is the output distribution of the encoder, p θ (x|z) represents the reconstruction probability of the decoder.

[0137] The fused eye-tracking feature label dataset F, which includes competency information G :

[0138]

[0139] Where f Gn Let x represent a feature vector in the dataset. n ,y n ,t n f represents the spatiotemporal location of eye movement features. n ,p n ,n n f represents the latent variables fused by the encoder. n It is an eye-tracking feature vector, p n It is the characteristic vector of the gaze point, n n gaze vector, R j C represents the area label related to competence. i ,OB i,j ,S i,j This indicates a competency rating label.

[0140] This yields the fused eye-tracking feature label dataset F, which contains competency information. G The dataset is divided into a training set and a validation set in an 8:2 ratio.

[0141] S2.2 Establish the score matrix E describing the competency assessment comp A scoring network with a Spatiotemporal Random Forest (STRF) network as its kernel is constructed, and the input is eye-tracking features F containing competency information labels. G Output the predicted competency assessment score matrix E comp .

[0142] Define the competency assessment score matrix Describe the evaluation results of the competency assessment method based on the core competency behavioral indicator framework:

[0143]

[0144] Wherein, matrix E comp The element represents the score of the j-th sub-item of the i-th competency category, where i is the competency category index (9 categories in total), and j is the sub-item index under the corresponding category (j∈[j1,j9]). comp(i,j)∈[1,5].

[0145] Constructed STRF network f STRF (·) is composed of multiple random trees {T1,T2,...,T T Composed of} each tree T t Independent training, from features f G A dimension k is randomly selected as a candidate splitting dimension, and information gain is used as the splitting criterion for recursive splitting. During splitting, if a feature has high correlation in consecutive time points or spatial neighborhoods, f is calculated using weighted averages. G The importance index W of comprehensive spatiotemporal characteristics FI To adjust feature selection:

[0146]

[0147] Where w temporal w represents the weight of time-related features. spatial Let I(·) represent the weights of spatially relevant features, and let I(·) represent the importance evaluation function. Representing feature f G The time-related part, Representing feature f G The spatial related parts.

[0148] The leaf nodes of each tree ultimately output a local prediction. The target score for samples within the leaf nodes, i.e., the competency sub-item OB. i,j A score is obtained, and the ensemble prediction of multiple trees is calculated by weighted averaging of all trees, resulting in E. comp A sub-item Where T is the number of random trees. Represents the score E of the t-th tree. ij Some E predictions ij The competency assessment score matrix E constitutes the prediction. comp .

[0149] S2.3 Competency Assessment Score Matrix E comp To establish a core pilot key competency assessment model, a competency assessment score matrix E was developed. comp Each sub-item has a scoring threshold. Sub-items with low competency scores are identified and marked as competency deficiencies (OBs) within the core competency behavior indicator framework. i,j Generate competency gap matrix D OBj This enables the identification of competency deficiencies. Based on the competency deficiency matrix, the key competency deficiencies corresponding to low scores are output as target sub-items for pilot EBT training.

[0150] Output competency gap matrix D OBj :

[0151]

[0152] Among them, element OB i,j This refers to the competencies that pilots lack during EBT training, and the missing competencies are the target competencies for the next EBT training.

[0153] Example 2

[0154] The present invention also provides a method and system for identifying pilot EBT training target competency driven by the fusion of eye-tracking data and flight data. The system is used to implement the method described in any one of the inventions. The system includes: a first extraction module, a second extraction module, and an evaluation module.

[0155] The first extraction module is used to acquire eye-tracking data of pilots during EBT training, construct a GDETC network to achieve spatiotemporal trajectory clustering of eye-tracking data, and extract the spatiotemporal trajectory of the pilot's gaze.

[0156] The second extraction module is used to acquire the trained EBT competency assessment worksheet, flight data, standard baseline flight data and parameter thresholds, and to construct a multi-source data fusion hierarchical extraction network to extract the flight data stream of associated error spatiotemporal points, the associated score and the OBj data stream of competency.

[0157] The evaluation module is used to construct a VAE-STRF network, fuse the data extracted from the first extraction module and the second extraction module, extract the pilot key competency evaluation score matrix, establish a pilot key competency evaluation model, and output the target competency for EBT training.

[0158] In this embodiment, the first extraction module includes: an eye-tracking feature set extraction unit, a trajectory similarity matrix generation unit, and a gaze spatiotemporal trajectory feature extraction unit;

[0159] The eye movement feature set extraction unit is used to collect pilots' eye movement data during EBT training, segment the cockpit visual space, associate the in-cabin visual space information with the eye movement data, and extract the spatiotemporally correlated eye movement feature set.

[0160] The trajectory similarity matrix generation unit is used to slice the eye-tracking feature set using a sliding dynamic time window technique, align non-linearly changing eye-tracking data in different time segments using dynamic time warping, and perform similarity calculations through point-by-point matching to ensure data accuracy and capture subtle differences between trajectories, thus forming the eye-tracking spatiotemporal trajectory similarity matrix D. DTW_0 Set a similarity threshold θ DTWFilter out invalid features to form the trajectory similarity matrix D for the next input step. DTW_new ;

[0161] The gaze spatiotemporal trajectory feature extraction unit is used to extract the sliced ​​eye movement features G k The similarity matrix D of the gaze trajectory DTW_new The constructed GDETC network is input, and the density of the trajectories is calculated. All time-segment trajectories are clustered into multiple clusters, each cluster representing a gaze behavior pattern. Low-density trajectory clusters represent atypical behaviors, while high-density trajectory clusters represent gaze behavior patterns. The spatiotemporal gaze trajectory feature F is constructed from the feature data of the centers of the high-density trajectory clusters. gaze ;

[0162] Among them, the spatiotemporally correlated eye-tracking feature set is: G EBT =D EBT ∪A cockpit In the formula, A cockpit For the eye-tracking visual spatial characteristics of the cockpit, D EBT For eye-tracking data;

[0163] Trajectory Similarity Matrix D DTW_new for: In the formula, the retained element D DTW_new (m,n) represents pairs of eye-tracking spatiotemporal segments with high similarity, which retain more gaze trajectory patterns;

[0164] Gaze at the spatiotemporal trajectory feature F gaze for: In the formula, k represents the trajectory segment number, i represents the trajectory point number, and j represents the region number. This represents the coordinates of the i-th gaze point of the k-th trajectory segment. This represents the eye movement feature vector at the i-th fixation point of the k-th trajectory segment. This represents the time point of the i-th gaze point in the k-th trajectory segment. This represents the characteristic vector of the i-th gaze point in the k-th trajectory segment. This represents the vector of the i-th gaze point in the k-th trajectory segment. This represents the j-th visual region associated with the i-th gaze point of the k-th trajectory segment. This indicates that the i-th gaze point of the k-th trajectory is in the region. Total length of stay This indicates that the i-th gaze point of the k-th trajectory is in the region. Number of fixations This indicates that the i-th gaze point of the k-th trajectory is in the region. Average gaze duration This represents the region switching frequency of the i-th gaze point in the k-th trajectory segment.

[0165] In this embodiment, the second extraction module includes: a flight data and evaluation work order acquisition unit, a flight data stream extraction unit, and an OBj data stream extraction unit;

[0166] The flight data and evaluation worksheet acquisition unit is used to determine the deviation thresholds of flight parameters based on reference standards and expert assessment. max_deviation and standard baseline flight data Q base Acquire simulator flight data Q fobs With post-training EBT competency assessment worksheet D FSTD ;

[0167] The flight data stream extraction unit is used to construct the deviation mapping architecture I of the SFHE network and extract the flight data stream Q of the structured data-associated error spatiotemporal points. error ;

[0168] The OBj data stream extraction unit is used to construct the structured data extraction architecture II of the SFHE network, and inputs the FSTD data stream D. FSTD Extract the associated scores and competency data stream C from the EBT assessment worksheet. score ;

[0169] Among them, the post-training EBT competency assessment worksheet D FSTD For: D FSTD ={D1,D2,…,D n}, where n is the total number of FSTD work orders included, and D n For a single FSTD text data;

[0170] Flight data stream Q error for: In the formula, t is time, a is a parameter, j is position, and R j For the cockpit space mapping function, T error For the set of operational error times, P error P is the set of operational error parameters. error (t) is the set of error locations, t n Represents flight data stream Q error The time point associated with the nth error point in the data, a dn Represents flight data stream Q error The parameter a associated with the nth error point in the data is the parameter a that indicates the occurrence of the error. d R jn Represents flight data stream Q error The cockpit space location R associated with the nth error point in the error. j ;

[0171] OBj data stream C score for: In the formula C i OB represents the i-th type of competence. i,j S represents the j-th sub-item of the i-th competency category. i,j S represents the score corresponding to the j-th sub-item under the i-th competency category. i,j ∈[1,5], where i is the competency category index with a total of 9 categories, and j is the sub-item index under the corresponding category.

[0172] In this embodiment, the evaluation module includes: an eye-tracking feature label dataset generation unit, a competency evaluation score matrix generation unit, and a competency evaluation score matrix generation unit;

[0173] The eye-tracking feature label dataset generation unit is used to generate gaze spatiotemporal trajectory features F based on a variational autoencoder (VAE). gaze OBj data flow Q related to scores and competence error Flight data stream C of associated error spatiotemporal points score Feature fusion is performed to form an eye-tracking feature label dataset F containing competency information. G ;

[0174] The competency assessment score matrix generation unit is used to build a score matrix E describing the competency assessment. comp A scoring network with a spatiotemporal random forest network (STRF) as its kernel is constructed, and the input is eye-tracking features F containing competency information labels. G Output the predicted competency assessment score matrix E comp ;

[0175] The competency assessment score matrix generation unit is used to generate the competency assessment score matrix E comp To establish a core pilot key competency assessment model, a scoring threshold was set to identify missing competency items (OBs). i，j And generate the competency gap matrix D OBj This enables the identification of competency deficiencies and outputs the target competencies for EBT training.

[0176] Among them, the eye-tracking feature label dataset F G for:

[0177]

[0178] In the formula, f Gn Let x represent a feature vector in the dataset. n ,y n ,t n f represents the spatiotemporal location of eye movement features. n ,p n ,n n f represents the latent variables fused by the encoder. nIt is an eye-tracking feature vector, p n It is the characteristic vector of the gaze point, n n gaze vector, R j C represents the area label related to competence. i ,OB i,j ,S i,j Indicates competency rating labels;

[0179] Competency assessment score matrix E comp for:

[0180]

[0181] In the formula, matrix E comp The element represents the score of the j-th sub-item of the i-th competency category, where i is the competency category index (9 categories in total), and j is the sub-item index under the corresponding category (j∈[j1,j9]). comp (i,j)∈[1,5];

[0182] Output competency gap matrix D OBj for:

[0183]

[0184] In the formula, element OB i,j This refers to the competencies that pilots lack during EBT training, and the missing competencies are the target competencies for the next EBT training.

[0185] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made to the technical solutions of the present invention by those skilled in the art without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A method for identifying pilot EBT training target competency driven by the fusion of eye-tracking data and flight data, characterized in that, The method includes: S0: Acquire eye-tracking data of pilots during EBT training, construct a trajectory clustering network to achieve spatiotemporal trajectory clustering of eye-tracking data, and extract the pilot's gaze spatiotemporal trajectory, including: S0.1: Collect eye movement data of pilots during EBT training, segment the cockpit visual space, correlate the in-cockpit visual space information with eye movement data, and extract a set of spatiotemporally related eye movement features. S0.2: The eye-tracking feature set is sliced ​​using a sliding dynamic time window technique. Dynamic time warping is used to align non-linearly changing eye-tracking data across different time segments. Point-by-point matching is then used to calculate similarity, forming the eye-tracking spatiotemporal trajectory similarity matrix D. DTW_0 Set a similarity threshold θ DTW Filter out invalid features to form the trajectory similarity matrix D for the next input step. DTW_new ; S0.3: The eye movement features G after slicing k And trajectory similarity matrix D DTW_new The constructed trajectory clustering network is input, and the density of the trajectories is calculated. All time-segment trajectories are clustered into multiple clusters, each cluster representing a gaze behavior pattern. Low-density trajectory clusters represent atypical behaviors, while high-density trajectory clusters represent gaze behavior patterns. The spatiotemporal trajectory features F of gaze are constructed using the feature data of the centers of high-density trajectory clusters. gaze ; S1: Obtain the trained EBT competency assessment worksheet, flight data, standard baseline flight data and parameter thresholds, construct a multi-source data fusion hierarchical extraction network, and extract the flight data stream of associated error spatiotemporal points, the associated score and the OBj data stream of competency; S2: Construct a VAE-STRF network, fuse the data extracted from S0 and S1, extract the pilot key competency assessment score matrix, establish a pilot key competency assessment model, and output the target competency for EBT training.

2. The method according to claim 1, characterized in that, In S0, the spatiotemporally correlated eye-tracking feature set is: G EBT =D EBT ∪A cockpit In the formula, A cockpit For the eye-tracking visual spatial characteristics of the cockpit, D EBT For eye-tracking data; Trajectory Similarity Matrix D DTW_new for: Gaze at the spatiotemporal trajectory feature F gaze for: In the formula, k represents the trajectory segment number, i represents the trajectory point number, and j represents the region number. This represents the coordinates of the i-th gaze point of the k-th trajectory segment. This represents the eye movement feature vector at the i-th fixation point of the k-th trajectory segment. This represents the time point of the i-th gaze point in the k-th trajectory segment. This represents the characteristic vector of the i-th gaze point in the k-th trajectory segment. This represents the vector of the i-th gaze point in the k-th trajectory segment. This represents the j-th visual region associated with the i-th gaze point of the k-th trajectory segment. This indicates that the i-th gaze point of the k-th trajectory is in the region. Total stay time This indicates that the i-th gaze point of the k-th trajectory is in the region. Number of fixations This indicates that the i-th gaze point of the k-th trajectory is in the region. Average gaze duration This represents the region switching frequency of the i-th gaze point in the k-th trajectory segment.

3. The method according to claim 2, characterized in that, In step S1, the trained EBT competency assessment worksheet, flight data, standard baseline flight data, and parameter thresholds are acquired. A multi-source data fusion hierarchical extraction network is constructed to extract flight data streams associated with spatiotemporal points of errors, associated scores, and competency OBj data streams, including: S1.1: The deviation threshold of flight parameters is determined by reference standards and based on expert assessment. max_deviation and standard baseline flight data Q base Acquire simulator flight data Q fobs With post-training EBT competency assessment worksheet D FSTD S1.2: Construct the deviation mapping architecture I of the multi-source data fusion hierarchical extraction network to extract the flight data stream Q of the structured data association error spatiotemporal points. error ; S1.3: Construct a structured data extraction architecture II for a multi-source data fusion hierarchical extraction network, input D FSTD Extract the associated scores and competency data stream C from the EBT assessment worksheet. score ; Among them, D FSTD ={D1,D2,...,D n }, where n is the total number of work orders included; Flight data stream Q error for: In the formula, t is time, a is a parameter, and R j For the cockpit space mapping function, T error For the set of operational error times, P error For the set of operational error parameters, t n Represents flight data stream Q error The time point associated with the nth error point in the data, a dn Represents flight data stream Q error The parameter a associated with the nth error point in the data is the parameter a that indicates the occurrence of the error. d R jn Represents flight data stream Q error The cockpit space location associated with the error that occurs at the nth error point in the data; OBj data stream C score for: In the formula C p OB represents the p-th type of competence. p,q S represents the q-th sub-item of the p-th competency. p,q S represents the score corresponding to the q-th sub-item under the p-th competency category. p,q ∈[1,5], where p is the competency category index with a total of 9 categories, and q is the sub-item index under the corresponding category.

4. The method according to claim 3, characterized in that, In S2, a VAE-STRF network is constructed to fuse the data extracted from S0 and S1, extract the pilot key competency assessment score matrix, establish a pilot key competency assessment model, and output the target competencies for EBT training, including: S2.1: Based on the variational autoencoder (VAE), the gaze spatiotemporal trajectory features F gaze The eye-tracking feature label dataset F, which includes competency information, is formed by fusing features from the OBj data stream containing associated scores and competency data, and the flight data stream containing associated error spatiotemporal points. G ; S2.2: Establish a score matrix describing competency evaluation, construct a scoring network with a spatiotemporal random forest network (STRF) as its kernel, and input F. G Output the predicted competency assessment score matrix E comp ; S2.3: Competency assessment score matrix E comp A pilot key competency assessment model was established, with scoring thresholds set to identify competency deficiencies, and a competency deficiency matrix D was generated. OBj This enables the identification of competency deficiencies and outputs the target competencies for EBT training. Competency assessment score matrix E comp for: In the formula, matrix E comp The elements represent the scores for each sub-item within each competency category; Output competency gap matrix D OBj The elements in the output represent the competencies that pilots lack during EBT training, and the output of the lacking competencies is the target competency for the next EBT training.

5. A system for recognizing pilot EBT training target competency driven by the fusion of eye-tracking data and flight data, the system being used to implement the method according to any one of claims 1-4, characterized in that, The system includes: a first extraction module, a second extraction module, and an evaluation module; The first extraction module is used to acquire eye-tracking data of pilots during EBT training, construct a trajectory clustering network to achieve spatiotemporal trajectory clustering of eye-tracking data, and extract the pilot's gaze spatiotemporal trajectory. The second extraction module is used to acquire the trained EBT competency assessment worksheet, flight data, standard baseline flight data and parameter thresholds, construct a multi-source data fusion hierarchical extraction network, and extract the flight data stream of associated error spatiotemporal points, the associated score and the OBj data stream of competency; The evaluation module is used to construct a VAE-STRF network, fuse the data extracted from the first extraction module and the second extraction module, extract the pilot key competency evaluation score matrix, establish a pilot key competency evaluation model, and output the target competency for EBT training.

Images