Eye movement / flight data fusion driven pilot EBT training target competency identification method and system

Abstract

The invention discloses a pilot EBT training target competency identification method and system based on eye movement / flight data fusion driving, and the method comprises the steps: S0, obtaining eye movement data of a pilot in an EBT training process, constructing a GDETC network to achieve the spatial-temporal trajectory clustering of the eye movement data, and extracting the staring spatial-temporal trajectory of the pilot; s1, extracting competency assessment scores and error time-space point data from EBT competency assessment worksheets and flight data; and S2, constructing a VAE-STRF network, fusing the data extracted in the step S0 and the step S1, extracting a pilot key competency evaluation score matrix, establishing a pilot key competency evaluation model, and outputting the target competency of EBT training. According to the method, the target competency of the pilot needing to continue to be improved subsequently is recognized through the eye movement data in the EBT training process of the pilot, an existing manual evaluation system can be replaced, and effective decision support is provided for improving the training pertinence of the pilot.

Description

Technical Field

[0001] The present invention belongs to the field of aircraft pilot training and physiological data analysis, and in particular relates to a method and system for pilot EBT training target competency identification driven by eye movement / flight data fusion. Background Art

[0002] Aviation safety is the core cornerstone of the sustainable development of the civil aviation industry, and the comprehensive ability level of pilots is the last line of defense to ensure the safe operation of flights. At present, the method of manually evaluating the nine competencies of pilots is mainly used to evaluate the comprehensive ability of pilots. Competency evaluation can effectively predict and measure the work performance level of pilots, and therefore becomes an important basis for pilot training and evaluation. In the evidence-based training (EBT) currently implemented, instructors need to evaluate the trainees' competencies by observing the trainees' use of relevant technical and non-technical knowledge, skills, and attitudes to perform activities or tasks under specific conditions, so as to determine whether they can effectively carry out their work and demonstrate the skilled skills required by the job, and form the results into a link of the EBT data chain. At present, the determination of target competencies still relies on the subjective evaluation of instructors. Before the start of each training, the "key competencies" determined by the simulator instructor based on the previous training results are used as the "target competencies" of this training course, and the applicable scenarios are selected as needed. After the refresher training is completed, which is implemented according to three courses, the simulator instructor will determine the "key competencies" that need to be improved based on the trainees' performance, which will serve as the "target competencies" for subsequent courses.

[0003] Although the subjective evaluation method can meet basic training needs, it is difficult to fully ensure the scientificity and consistency of the target competency determination due to its high reliance on the subjective judgment of instructors. Data fusion drive is an important development direction for pilot competency identification. Data drive is the most important principle for the implementation of EBT. The training frequency and training framework of the training topics in EBT need to be determined based on data and analysis reports. Outputting high-quality training data is also an important goal of EBT. The data output from each training will become one of the data sources for the next cycle of EBT training, which will be converted into training needs and eventually form a closed loop. In order to form data drive through reliable training and evaluation data, it is necessary to completely open up the data transmission chain so that the data flow can flow based on the objective data chain. Therefore, developing an evaluation system based on objective data to determine the target competency has become an urgent need for the current civil aviation industry management. Summary of the invention

[0004] In order to open up the transmission chain of EBT training data flow, realize data-driven EBT training, and solve the problem that target competency is mainly determined by instructors' subjective evaluation and pilot EBT training lacks objective data evaluation support, the present invention provides a method and system for pilot EBT training target competency identification driven by eye movement / flight data fusion, which can use multi-source data fusion evaluation to supplement and improve the existing manual evaluation system, and provide effective decision support for improving the targeted training of pilots.

[0005] To achieve the above object, the present invention provides the following solutions:

[0006] A method for identifying pilot EBT training target competency driven by eye movement / flight data fusion, the method comprising:

[0007] S0: Obtain the eye movement data of the pilot during the EBT training process, build a GDETC network to cluster the spatiotemporal trajectory of the eye movement data, and extract the spatiotemporal trajectory of the pilot's gaze;

[0008] S1: Obtain the trained EBT competency assessment worksheet, flight data, standard baseline flight data and parameter thresholds, build a multi-source data fusion hierarchical extraction network, extract the flight data stream of the associated error spatiotemporal points, and the OBj data stream of the associated scores and competency;

[0009] S2: Construct a VAE-STRF network, fuse the data extracted in S0 and S1, extract the pilot key competency assessment score matrix, establish a pilot key competency assessment model, and output the target competency of EBT training.

[0010] Preferably, in S0, obtaining the eye movement data of the pilot during the EBT training process, constructing a GDETC network to cluster the spatiotemporal trajectory of the eye movement data, and extracting the spatiotemporal trajectory of the pilot's gaze includes:

[0011] S0.1: Collect the pilot's eye movement data during EBT training, segment the cockpit visual space, associate the cabin visual space information with the eye movement data, and extract a set of eye movement features associated with time and space;

[0012] S0.2: Slice the eye movement feature set through the sliding dynamic time window technology, align the nonlinear eye movement data in different time segments through the dynamic time warping method, and perform similarity calculation through point-by-point matching to ensure data accuracy and capture the slight differences between trajectories, forming the eye movement spatiotemporal trajectory similarity matrix D DTW_0 , set the similarity threshold θ DTW Filter invalid features to form the trajectory similarity matrix D for the next step DTW_new ;

[0013] S0.3: Slice the eye movement feature G k and the gaze trajectory similarity matrix D DTW_new into the constructed GDETC network, calculate the density of the trajectory, and cluster all time segment trajectories into multiple clusters. Each cluster represents a gaze behavior pattern, where the low-density trajectory cluster represents atypical behavior, and the high-density trajectory cluster represents the gaze behavior pattern. The feature data passing through the center of the high-density trajectory cluster constitutes the gaze spatio-temporal trajectory feature F gaze ;

[0014] Among them, the spatio-temporal correlated eye movement feature set is: G EBT = D EBT ∪A cockpit , where A cockpit is the cockpit eye movement visual space feature, and D EBT is the eye movement data;

[0015] The trajectory similarity matrix D DTW_new is: In the formula, the retained element D DTW_new (m,n) represents pairs of spatio-temporal segments of eye movement with high similarity, and more gaze trajectory rules are retained between these segments;

[0016] The gaze spatio-temporal trajectory feature F gaze is: In the formula, k represents the trajectory segment number, i represents the trajectory point number, j represents the region number, represents the coordinates of the i-th fixation point of the k-th trajectory, represents the eye movement feature vector of the i-th fixation point of the k-th trajectory, represents the time point of the i-th fixation point of the k-th trajectory, represents the characteristic vector of the i-th fixation point of the k-th trajectory itself, represents the vector of the i-th fixation point of the k-th trajectory, represents the j-th visual region associated with the i-th fixation point of the k-th trajectory, represents the i-th fixation point of the k-th trajectory in the region total residence time, represents the i-th fixation point of the k-th trajectory in the region number of fixations, represents the i-th fixation point of the k-th trajectory in the region average fixation duration, represents the region switching frequency of the i-th fixation point of the k-th trajectory.

[0017] Preferably, in S1, after obtaining the trained EBT competency assessment worksheet, flight data, standard baseline flight data, and parameter thresholds, a multi-source data fusion hierarchical extraction network is constructed to extract the flight data stream related to the error time and space points, the OBj data stream related to the associated score and competency, including:

[0018] S1.1: By referring to the standard and according to the deviation threshold ∈ of the flight parameters determined by experts max_deviation and the standard baseline flight data Q base , the simulator flight data Q fobs and the trained EBT competency assessment worksheet D FSTD are obtained;

[0019] S1.2: Construct the deviation mapping architecture I of the SFHE network to extract the flight data stream Q related to the structured data associated with the error time and space points error ;

[0020] S1.3: Construct the structured data extraction architecture II of the SFHE network, input the FSTD data stream D FSTD , and extract the OBj data stream C related to the associated score and competency in the EBT assessment worksheet score ;

[0021] Among them, the trained EBT competency assessment worksheet D FSTD is: D FSTD ={D1, D2, …, D n}, where n is the total number of FSTD worksheets included, and D n is the FSTD text data of a single copy;

[0022] The flight data stream Q error is: In the formula, t is time, a is a parameter, j is a position, R j is the cockpit space mapping function, T error is the set of operation error times, P error is the set of operation error parameters, P error (t) is the set of error positions, and t n represents the time point associated with the nth error point in the flight data stream Q error , a dn represents the parameter a where the error occurs associated with the nth error point in the flight data stream Q error , and R d represents the cockpit space position R associated with the error that occurs at the nth error point in the flight data stream Q jn ; error ; j ;

[0023] The OBj data stream C score is: In the formula, C i represents the i-th type of competency, OB i,j represents the j-th sub-item of the i-th type of competency, S i,j represents the score corresponding to the j-th sub-item under the i-th type of competency, S i,j ∈[1, 5], where i is the competency category index with a total of 9 categories, and j is the sub-item index under the corresponding category.

[0024] Preferably, in S2, a VAE-STRF network is constructed to fuse the data extracted from S0 and S1, extract the key competency evaluation score matrix of the pilot, establish a key competency evaluation model for the pilot, and the target competencies output for EBT training include:

[0025] S2.1: Based on the variational autoencoder VAE, the gaze spatio-temporal trajectory feature F gaze , the OBj data stream Q of the associated score and competency error , and the flight data stream C of the associated error spatio-temporal points score are feature-fused to form an eye movement feature label data set F G containing competency information;

[0026] S2.2: Establish a score matrix E comp describing the competency evaluation, construct a scoring network with the spatio-temporal random forest network STRF as the kernel, input the eye movement feature F G containing the competency information label, and output the predicted competency evaluation score matrix E comp ;

[0027] S2.3: Establish a key competency evaluation model for the pilot with the competency evaluation score matrix E comp as the core, set a scoring threshold to judge the competency missing item OB i,j , and generate a competency missing matrix D OBj , realize the identification of competency missing, and output the target competencies for EBT training;

[0028] Among them, the eye movement feature label data set F G is:

[0029]

[0030] In the formula, f Gn represents a feature vector in the data set, x n , y n , t n represent the spatio-temporal positions of the eye movement features, f n , p n , n n represent the latent variables fused by the encoder, fn is the eye movement feature vector, p n is the characteristic vector of the fixation point, n n fixation point vector, R j represents the area label related to competency, C i , OB i,j , S i,j represents the competency score label;

[0031] Competency evaluation score matrix E comp is:

[0032]

[0033] In the formula, matrix E comp 's element represents the score of the j-th sub-item of the i-th type of competency, where i is the competency category index with a total of 9 categories, and j is the sub-item index under the corresponding category, j ∈ [j1, j9], E comp (i, j) ∈ [1, 5];

[0034] The output competency missing matrix D OBj is:

[0035]

[0036] In the formula, the element OB i,j is the competency missing of the pilot during the EBT training process, and the output missing competency is also the target competency of the next EBT training.

[0037] The present invention also provides a method and system for identifying the target competency of pilot EBT training driven by eye movement / flight data fusion. The system is used to implement any one of the methods, and the system includes: a first extraction module, a second extraction module, and an evaluation module;

[0038] The first extraction module is used to obtain the eye movement data of the pilot during the EBT training process, construct a GDETC network to realize the spatio-temporal trajectory clustering of the eye movement data, and extract the pilot's gaze spatio-temporal trajectory;

[0039] The second extraction module is used to obtain the EBT competency evaluation work sheet, flight data, standard baseline flight data, and parameter thresholds after training, construct a multi-source data fusion hierarchical extraction network, and extract the flight data stream associated with the error spatio-temporal point, the OBj data stream associated with the score and competency;

[0040] The evaluation module is used to construct a VAE-STRF network, fuse the data extracted by the first extraction module and the second extraction module, extract the pilot's key competency evaluation score matrix, establish a pilot's key competency evaluation model, and output the target competency of the EBT training.

[0041] Preferably, the first extraction module includes: an eye movement feature set extraction unit, a trajectory similarity matrix generation unit, and a gaze spatio-temporal trajectory feature extraction unit;

[0042] The eye movement feature set extraction unit is used to collect the eye movement data of the pilot during the EBT training, segment the visual space of the cockpit, associate the in-cockpit visual space information with the eye movement data, and extract the spatio-temporal associated eye movement feature set;

[0043] The trajectory similarity matrix generation unit is used to slice the eye movement feature set through the sliding dynamic time window technology, align the non-linearly changing eye movement data in different time segments by the dynamic time warping method, and perform similarity calculation through point-by-point matching to ensure data accuracy and capture the small differences between trajectories, forming the eye movement spatio-temporal trajectory similarity matrix D DTW_0 , set the similarity threshold θ DTW Filter out invalid features to form the trajectory similarity matrix D for the next step DTW_new ;

[0044] The gaze spatio-temporal trajectory feature extraction unit is used to input the sliced eye movement feature G k and the gaze trajectory similarity matrix D DTW_new into the constructed GDETC network, calculate the density of the trajectory, cluster all the time segment trajectories into multiple clusters, each cluster represents a gaze behavior pattern, where the low-density trajectory cluster represents atypical behavior, and the high-density trajectory cluster represents the gaze behavior pattern. The feature data through the center of the high-density trajectory cluster constitutes the gaze spatio-temporal trajectory feature F gaze ;

[0045] Among them, the spatio-temporal associated eye movement feature set is: G EBT = D EBT ∪A cockpit , where A cockpit is the cockpit eye movement visual space feature, and D EBT is the eye movement data;

[0046] The trajectory similarity matrix D DTW_new is: Where the retained element D DTW_new (m,n) represents the pair of eye movement spatio-temporal segments with higher similarity, and more gaze trajectory rules are retained between these segments;

[0047] The gaze spatio-temporal trajectory feature F gaze is: Where k represents the trajectory segment number, i represents the trajectory point number, j represents the region number, represents the coordinate of the i-th fixation point of the k-th trajectory, The eye movement feature vector representing the i-th fixation point of the k-th trajectory The time point representing the i-th fixation point of the k-th trajectory The characteristic vector representing the i-th fixation point of the k-th trajectory itself The vector representing the i-th fixation point of the k-th trajectory The j-th visual area associated with the i-th fixation point of the k-th trajectory The i-th fixation point of the k-th trajectory represents the area Total residence time The i-th fixation point of the k-th trajectory represents the area Number of fixations The i-th fixation point of the k-th trajectory represents the area Average fixation duration The area switching frequency representing the i-th fixation point of the k-th trajectory

[0048] Preferably, the second extraction module includes: a flight data and evaluation work sheet acquisition unit, a flight data stream extraction unit, and an OBj data stream extraction unit;

[0049] The flight data and evaluation work sheet acquisition unit is used to obtain the deviation threshold ∈ of the flight parameters determined by experts by referring to the reference standard max_deviation And the standard baseline flight data Q base , to obtain the simulator flight data Q fobs And the trained EBT competency evaluation work sheet D FSTD ;

[0050] The flight data stream extraction unit is used to construct the deviation mapping architecture I of the SFHE network and extract the flight data stream Q of the structured data associated error space-time points error ;

[0051] The OBj data stream extraction unit is used to construct the structure data extraction architecture II of the SFHE network, input the FSTD data stream D FSTD , and extract the OBj data stream C of the associated score and competency in the EBT evaluation work sheet score ;

[0052] Among them, the trained EBT competency evaluation work sheet D FSTD Is: D FSTD ={D1,D2,…,D n},n is the total number of FSTD work sheets included, D n Is a single FSTD text data;

[0053] Flight data stream Q error Is: Wherein, t is time, a is a parameter, j is a position, and R j is the cockpit space mapping function, T error is the set of operation error times, P error is the set of operation error parameters, P error (t) is the set of error positions, t n represents the nth error point associated time point in the flight data stream Q error ; a dn represents the parameter a of the error occurrence associated with the nth error point in the flight data stream Q error ; R d represents the cockpit space position R associated with the error occurrence of the nth error point in the flight data stream Q jn ; error ; j ;

[0054] The OBj data stream C score is: Wherein C i represents the i-th type of competency, OB i,j represents the j-th sub-item of the i-th type of competency, S i,j represents the score corresponding to the j-th sub-item under the i-th type of competency, S i,j ∈[1, 5], where i is the competency category index with a total of 9 categories, and j is the sub-item index under the corresponding category.

[0055] Preferably, the evaluation module includes: an eye movement feature label data set generation unit, a competency evaluation score matrix generation unit, and a competency evaluation score matrix generation unit;

[0056] The eye movement feature label data set generation unit is used to perform feature fusion on the gaze spatio-temporal trajectory feature F gaze , the OBj data stream Q associated with the correlation score and competency error , and the flight data stream C associated with the correlation error spatio-temporal point score to form an eye movement feature label data set F containing competency information G ;

[0057] The competency evaluation score matrix generation unit is used to establish a score matrix E describing the competency evaluation comp , construct a scoring network with a spatio-temporal random forest network STRF as the kernel, input the eye movement feature F containing the competency information label G , and output the predicted competency evaluation score matrix E comp ;

[0058] The competency evaluation score matrix generation unit is used to use the competency evaluation score matrix E compBuild a pilot key competency assessment model with the core, set a scoring threshold to identify the missing competency item OB i,j , and generate a missing competency matrix D OBj , realize the identification of missing competencies, and output the target competencies for EBT training;

[0059] Among them, the eye movement feature label dataset F G is:

[0060]

[0061] In the formula, f Gn represents a feature vector in the dataset, x n , y n , t n represent the spatio-temporal positions of eye movement features, f n , p n , n n represent the latent variables fused by the encoder, f n is the eye movement feature vector, p n is the characteristic vector of the fixation point, n n fixation point vector, R j represents the region label related to the competency, C i , OB i,j , S i,j represent the competency scoring label;

[0062] The competency assessment score matrix E comp is:

[0063]

[0064] In the formula, the elements of the matrix E comp represent the scores of the j-th sub-item of the i-th type of competency, where i is the competency category index with a total of 9 categories, and j is the sub-item index under the corresponding category, j ∈ [j1, j9], E comp (i, j) ∈ [1, 5];

[0065] The output missing competency matrix D OBj is:

[0066]

[0067] In the formula, the element OB i,j is the missing competency of the pilot during the EBT training process, and the output missing competency is also the target competency for the next EBT training.

[0068] Compared with the prior art, the beneficial effects of the present invention are:

[0069] The present invention discloses a method and system for identifying the target competence of pilot EBT training driven by the fusion of eye movement / flying data, which specifically includes the following steps: S0: Obtain the eye movement data during the pilot training process, extract the spatio-temporal related eye movement features, slice them through a dynamic time window, calculate the gaze trajectory similarity matrix based on the dynamic time warping method, construct a GDETC network to achieve spatio-temporal trajectory clustering of eye movement features, and extract the pilot's gaze spatio-temporal trajectory; S1: Obtain the EBT competence evaluation worksheet, flying data, standard baseline flying data, and parameter thresholds after training, construct a multi-source data fusion hierarchical extraction network, and extract the flying data stream related to the associated error spatio-temporal points and the OBj data stream related to the associated score and competence; S2: Construct a VAE-STRF network, fuse the data extracted in S0 and S1, extract the pilot's key competence evaluation score matrix, establish a pilot's key competence evaluation model, and output the target competence of EBT training. The purpose of the present invention is to identify the target competence that the pilot needs to continue to improve in the future through the eye movement data during the pilot EBT training process, which can replace the existing manual evaluation system and provide effective decision-making support for improving the pertinence of pilot training. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] In order to more clearly illustrate the technical solutions of the present invention, the accompanying drawings required for use in the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention, and those of ordinary skill in the art can also obtain other drawings based on these drawings without creative efforts.

[0071] Figure 1 It is a schematic diagram of the main steps S0 - S2 described in the embodiments of the present invention;

[0072] Figure 2 It is a schematic flow chart of a method for identifying the target competence of pilot EBT training driven by the fusion of eye movement / flying data according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0073] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0074] In order to make the above objects, features, and advantages of the present invention more obvious and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0075] Embodiment 1

[0076] As Figure 1 , Figure 2 shown, the present invention provides a method for identifying the pilot's EBT training target competency driven by eye movement / flying data fusion, including the following steps:

[0077] S0: Obtain the eye movement data during the pilot's training process, extract the spatio-temporal correlated eye movement features, slice through a dynamic time window, calculate the eye movement trajectory similarity matrix based on DTW, construct a GDETC network to achieve spatio-temporal trajectory clustering of the eye movement features, and extract the gaze spatio-temporal trajectory.

[0078] The specific steps are as follows:

[0079] S0.1 Collect the eye movement data D of the pilot during the EBT training process EBT , perform regional segmentation on the cockpit visual space, associate the in-cabin visual space information A cockpit with the eye movement data, and extract the spatio-temporal correlated eye movement feature set G EBT .

[0080] The input eye movement data D of the system EBT = D blink_t ∪ D blink_i is represented by the union of the continuous dynamic time series D blink_t = {D blink_t = {(x t , f t , t) | t ∈ [t0, T]} associated with the t moment and the static feature set D blink_i = {(n i , p i ) | i ∈ [1, N]} of the discrete fixation points of the associated sequence i. In the dynamic time series, x t represents the coordinate x of the eye movement trajectory t = (x t , y t ), where x t , y t are the coordinates of the fixation point at the t moment; f t represents the time-dependent eye movement feature vector f t = (v t , a t , φ t , p t ), where v t represents the saccade speed, a t represents the saccade acceleration, φ t represents the saccade direction, p t represents the pupil diameter; t represents the data acquisition time point. There are N fixation points in the static feature set, and n i represents the i-th fixation point; pi =(t i , l i , r i , g i , f i ) represents the characteristics of the i-th fixation point itself, where t i represents the fixation duration of the i-th fixation point, l i represents the saccade path length between two consecutive fixation points, r i fixation point aggregation degree, g i the area where the fixation point is located.

[0081] The fixation area of the pilot in the cockpit is divided into M different regions R cockpit_j ={R j | j ∈ [1, M]}, and the cockpit eye movement visual space feature A including regional characteristics cockpit ={(R j , τ j , c j , μ j , s t ) | j ∈ [1, M], t ∈ [t0, T]}, where R j represents the j-th visual region in the cockpit, τ j represents the total residence time of the line of sight in region R j , c j represents the number of fixations of the line of sight in region R j , μ j represents the average fixation duration of the line of sight in region R j , s t represents the region switching frequency.

[0082] The visual space information A in the cockpit cockpit is associated with the eye movement data D EBT through the union, so that the eye movement data includes the in-cockpit environment information, forming a spatio-temporal associated eye movement feature set G EBT = D EBT ∪ A cockpit .

[0083] S0.2 slices the spatio-temporal associated eye movement feature set G EBT using the sliding dynamic time window technology, and calculates the similarity of the eye movement spatio-temporal trajectory for the sliced data segments using DTW.

[0084] According to the saccade ratio P eyetrack_0 within the fixed time t glance_all as the fixation state index, dynamically set the sliding window length ΔT blink and the sliding step size δT blink, the set G of eye movement features with spatio-temporal correlation in the collected long time series EBT is sliced, and the spatio-temporal correlation eye movement feature data set G EBT is divided into K time segment sets G EBT_k ={G k [t k :t k +ΔT blink ∣k∈[0,K]}; when the fixation state index P glance_all ≤0.5, it indicates that the pilot's fixation is changing rapidly, and a smaller sliding window length ΔT blink and sliding step δT blink are set to capture the behavioral characteristics in the rapidly switching state; when P glance_all >0.5, a larger sliding window length ΔT blink and sliding step δT blink are set to retain the continuous slice features. The eye movement data segments after dynamic slicing are:

[0085] G k ={(x t ,f t ,t)∪(n i ,p i )∪(R j ,τ j ,c j ,μ j ,s t )∣t∈[t k ,t k +ΔT blink ,i∈[1,N],j∈[1,M]},k∈[0,K], and the eye movement segment data set G EBT_k with associated spatio-temporal features is obtained.

[0086] The non-linearly changing eye movement data in different time segments is aligned through DTW, and similarity calculation is performed through point-by-point matching to ensure data accuracy and capture the small differences between trajectories, forming the eye movement spatio-temporal trajectory similarity matrix D DTW_0 , and a similarity threshold θ DTW is set to filter out invalid features, forming the trajectory similarity matrix D DTW_new for the next step.

[0087] For two different eye movement spatio-temporal trajectory segments G m and G n , the comprehensive deviation d m of their eye movement trajectories is defined as:

[0088]

[0089] where i is the serial number of the i-th trajectory point in time segment G, indicating that the current calculation involves the i-th trajectory point, and i ∈ [1, N m , N m is the number of trajectory points in time segment G m ; j is the serial number of the j-th trajectory point in time segment G m , indicating that the current calculation involves the j-th trajectory point, and j ∈ [1, N n , N n is the number of trajectory points in time segment G n . Among them, n is the dynamic time series feature distance, which is used to represent the spatial difference between the trajectory points of two segments, including the spatial distance between the trajectory points of the two segments and the dynamic feature vector v , a t , φ t , p t The Euclidean distance between t is the static fixation point feature distance, which is used to represent the Euclidean distance between the fixation point characteristic vectors p and p m , n is the cockpit area characteristic distance, which is represented by the normalized weighted Euclidean distance .

[0090] Set three weight parameters w1, w2, w3 ∈ [0, 1], satisfying the total weight normalization w1 + w2 + w3 = 1, set the initial values w1 = 0.5, w2 = 0.3, w3 = 0.2, and through grid search, set the search step δ w = 0.05 to optimize the weights on the training set

[0091] Construct the cumulative cost matrix D STEM_ac . The elements of matrix D STEM_ac are calculated step by step from the upper left corner to the lower right corner in a cumulative manner. The initial size of the matrix is N m ×N n , N m , N n is the number of trajectory points in time segments G m and G n :

[0092]

[0093] The recursive formula and boundary conditions of the cumulative cost matrix

[0094]

[0095] The elements of matrix D STEM_ac ​The local matching cost represents the local similarity between two segments, and the overall matching cost is D STEM_ac (N m ,N n ) represents the overall similarity between two time segments, that is, the similarity DTW(G m ,G n ) = D STEM_ac (N m ,N n ).

[0096] Calculate the similarity between all pairs of time segments of eye movement data to generate an initial gaze trajectory similarity matrix D DTW_0 , the element D DTW_0 of the matrix D m,n = DTW(G m ,G n ) represents the similarity between time segments (m, n), and the size of the matrix is K×K:

[0097]

[0098] Set the gaze trajectory similarity threshold θ DTW , and exclude all time segment pairs (m, n) with a trajectory similarity greater than θ DTW . For each element, the discriminant condition is:

[0099]

[0100] In the initial gaze trajectory similarity matrix D DTW_0 , after filtering out invalid feature pairs with low similarity, the gaze trajectory similarity matrix D DTW_new is obtained:

[0101]

[0102] Among them, the remaining element D DTW_new (m, n) represents pairs of spatio-temporal eye movement segments with higher similarity, and more gaze trajectory rules are retained between these segments.

[0103] Build a GDETC network and extract gaze spatio-temporal trajectory features.

[0104] The sliced eye movement features G k = {(x t ,f t ,t) ∪ (n i ,p i ) ∪ (R j ,τ j ,c j ,μ j ,s t)} and the gaze trajectory similarity matrix D DTW_new Input the constructed GDETC network, calculate the density of the trajectory, cluster all the time-segment trajectories into multiple clusters, each cluster representing a gaze behavior pattern, where the low-density trajectory clusters represent atypical behaviors such as saccades, anomalies, noises, etc., and the high-density trajectory clusters represent the gaze behavior pattern. The characteristic data passing through the center of the high-density trajectory clusters constitutes the gaze spatio-temporal trajectory feature F gaze .

[0105] Through the gaze trajectory similarity matrix D DTW_new Calculate the trajectory point density ρ k , calculate the density distance δ of the trajectory points k , output the initial trajectory and its center point. Further, optimize the initial trajectory cluster center to generate a more representative gaze behavior trajectory feature, and input a random noise vector GDETC_track and the initial trajectory cluster center into the gaze trajectory generator G z to generate the trajectory cluster center feature vector V centre , input the real trajectory cluster center and the generated trajectory into the gaze trajectory discriminator D GDETC_track to judge whether the input comes from the real trajectory cluster, iterate and train d times, optimize the gaze trajectory generator G GDETC_track and output the final trajectory cluster center feature V centre_true , and train it in the way of the generative adversarial loss function. The loss function is as follows:

[0106] L GDETC =E x～truth [logD GDETC_track (x)] + E z～noise [log(1 - D GDETC_track (G GDETC_trackk (z)))] (7)

[0107] Reallocate the trajectory clusters based on the generated trajectory cluster center, calculate the similarity between the trajectory points and the cluster center, update the allocation result of the trajectory clusters until the clustering converges, generate high-density trajectory clusters, and output the gaze behavior pattern. For each high-density trajectory cluster, which represents the gaze behavior pattern rule, extract its dynamic features static features region features and other gaze features to form the gaze spatio-temporal trajectory feature containing rules where k represents the trajectory segment number, i represents the trajectory point number, j represents the region number, represents the coordinate of the i-th fixation point of the k-th trajectory, represents the eye movement feature vector of the i-th fixation point of the k-th trajectory, represents the time point of the i-th fixation point of the k-th trajectory, The eigenvector of the i-th fixation point of the k-th trajectory itself, represents the i-th fixation point vector of the k-th trajectory, represents the j-th visual area associated with the i-th fixation point of the k-th trajectory, represents the i-th fixation point of the k-th trajectory in the area total residence time, represents the i-th fixation point of the k-th trajectory in the area number of fixations, represents the i-th fixation point of the k-th trajectory in the area average fixation duration, represents the area switching frequency of the i-th fixation point of the k-th trajectory.

[0108] S1: Obtain the trained EBT competency assessment worksheet, simulator flight data, standard baseline flight data, and parameter thresholds, construct the SFHE network, and extract the flight data stream associated with the spatio-temporal points of associated errors, the OBj data stream C of associated scores and competencies score .

[0109] The specific steps are as follows:

[0110] S1.1 Determine the deviation threshold ∈ of flight parameters by referring to the standard and relying on experts max_deviation and the standard flight baseline data Q base , and obtain the simulator flight data Q fobs and the trained EBT competency assessment worksheet D FSTD .

[0111] Refer to the aircraft flight quality monitoring standard and rely on flight safety experts to determine the allowable deviation threshold ∈ of flight parameters max_deviation ={∈1, ∈2,…, ∈ d} and the standard flight baseline data which contains d parameters, is the flight parameter baseline. And obtain the flight data Q fobs (t)={(q1(t), q2(t),..., q d (t))|t ∈ [0, T]}, which contains d parameters, q d (t) is the parameter of the flight data in the training stage.

[0112] Obtain the flight data Q fobs (t) and the corresponding EBT competency assessment worksheet (FSTD) in the training. Multiple EBT competency assessment worksheet data constitute the FSTD data stream D FSTD ={D1, D2,..., D n}, where n is the total number of FSTD work sheets included, D n is the FSTD text data for a single copy. The FSTD is filled in by the course simulator instructor after evaluating the overall competency performance of the trainee according to the standards stipulated by the Civil Aviation Administration of China, recording the overall competency performance of the pilot during the EBT training process. The competency scoring data in the FSTD provides competency label data for the eye movement data in this method.

[0113] S1.2 Construct the deviation mapping architecture Ⅰ of the SFHE network, and extract the structured data flight data stream Q of the associated error spatio-temporal points by calculating the weighted deviation error .

[0114] Input the flight data Q fobs (t), the standard baseline flight data Q base (t) and the cockpit area division data R cockpit_j , and through the high-dimensional spatio-temporal parameter joint deviation mapping of the SFHE, output the flight data stream Q error (t,j). Q error (t,a,R j ) contains the operation error location and time point data during the simulator training process, provides the time label corresponding to the error that causes the lack of competency to appear, and the spatial label of the cockpit area associated with the error parameter.

[0115] Based on the high-dimensional mapping of the dynamic system and introducing the weight function w, calculate the weighted deviation ΔQ w_EBT (t) between the flight operation data and the standard baseline:

[0116] ΔQ w_EBT (t) = w · (Q fobs (t) - Q base (t)) = {w1 · Δa1(t), w2 · Δa2(t),..., w d · Δa d (t)} (8)

[0117] When |Δa d (t)| > ∈ i , it is considered that the parameter a d has an error at time t, forming the operation error time set and the operation error parameter set P error (t) = {a d : |Δa d (t)| > ∈ i , t ∈ T error}, define the cockpit space mapping function f(a d ) = R j , through f(a i)Establish the positional association between operation error parameters and the cockpit space, and the result forms the error position set R error (t) = {f(a i ): a i ∈P error (t)}, comprehensively combine the time t, parameter a, and position j of the error data to form the flight data stream Q of the associated error space-time points error , which contains n error points, where t n represents the time point associated with the nth error point in the flight data stream Q error , a dn represents the parameter a where an error occurs associated with the nth error point in the flight data stream Q error , R d , R jn represents the cockpit space position R associated with the error that occurs at the nth error point in the flight data stream Q error ; j ;

[0118]

[0119] S1.3 Construct the structure data extraction architecture Ⅱ of the SFHE network, and input the FSTD data stream D FSTD = {D1, D2,..., D n}, determine the table area through the SFHE v5 module, use the table structure parsing module SFHE FSR to extract cell data, embed the text content of the cells into the multi-layer semantic space through the SFHE embedding module, classify and group competencies, behavior indicators, and scores through the SFHE classify module, and extract the OBj data stream C of the associated scores and competencies in the EBT evaluation work sheet score .

[0120] Define the set C Competency = {C i,j ∣i ∈ [1, 9], j ∈ [1, 11]} to describe each competency-corresponding Obj sub-item in the nine major competency and behavior indicator frameworks, and C i,j represents the jth sub-item of the ith type of competency

[0121] Input the FSTD data stream D FSTD = {D1, D2,..., D n}, through the SFHE v5 module of the SFHE network, for each PDF-format D FSTD in the D n data stream, locate the competency scoring table area and output the rectangular boundary R of the table table_area=SFHE v5 (D i ) = {(x1, y1, x2, y2)}. Through the table structure parsing module SFHE FSR Perform row and column splitting on the image within the table area, and extract the position and content of each cell where R i,j represents the coordinates of the cell in the i-th row and j-th column, and T i,j represents the text content of the cell in the i-th row and j-th column, represents the image of the cropped table area.

[0122] Through the semantic hierarchical embedding module SFHE embedding Map the cell content T i,j to the multi-layer semantic space of competency, behavior indicators, and scores Φ(T ij ):

[0123] Φ(T ij ) = (Φ comp (T ij ), Φ obj (T ij ), Φ score (T ij )) (10)

[0124] where Φ comp (T ij ) represents the embedding vector of competency, Φ obs (T ij ) represents the embedding vector of behavior indicators, Φ score (T ij ) represents the embedding vector of scores.

[0125] Furthermore, through the SFHE classify module, classify the embedding vector Φ(T ij ) input to the fully connected layer and calculate the conditional probability distribution through the Softmax function, and select the category or score with the maximum conditional probability through the argmax operator, thereby mapping the input text T ij to the multi-task semantic space to achieve semantic classification:

[0126]

[0127] where represents the competency category corresponding to the embedding vector Φ comp (T ij ), OB i,j represents the behavior indicator category corresponding to the embedding vector Φ obs (T ij ), represents the embedding vector Φscore (T ij ) corresponding scoring value.

[0128] Obtain the OBj data stream C of the associated score and competency score :

[0129]

[0130] Among them, C i represents the i-th type of competency, OB i,j represents the j-th sub-item of the i-th type of competency, S i,j represents the score corresponding to the j-th sub-item under the i-th type of competency, S i,j ∈[1, 5], where i is the competency category index with a total of 9 categories, and j is the sub-item index under the corresponding category.

[0131] S2: Construct a VAE-STRF network, fuse the data extracted from S0 and S1, and extract the pilot's key competency evaluation score matrix E comp , establish a pilot's key competency evaluation model to identify the missing competency items D OBj , realize the identification of missing competency items, and output the target competency for EBT training.

[0132] The specific steps are as follows:

[0133] S2.1 Based on the variational autoencoder, fuse the gaze spatio-temporal trajectory feature F gaze , the OBj data stream Q error , and the flight data stream C of the associated error spatio-temporal points score to form an eye movement feature label dataset F G .

[0134] Align the three types of data input from S0 and S1 to ensure feature fusion within the same time range, and normalize F gaze to Φ gaze , Q error to Φ QAR , C score to Φ OBJ , and construct the multi-source feature module VAE(Φ gaze , Φ QAR , Φ OBJ ) of the variational autoencoder to fuse the features, extract the potential low-dimensional representation of the high-dimensional features, and at the same time reduce noise and remove redundant information.

[0135] The loss function of the variational autoencoder is defined as:

[0136]

[0137] Among them, D KL is the KL divergence, which is used to regularize the latent space, and q φ (z∣x) is the output distribution of the encoder, and p θ (x∣z) represents the reconstruction probability of the decoder.

[0138] The eye movement feature label dataset F containing competency information after fusion G :

[0139]

[0140] Among them, f Gn represents a feature vector in the dataset, and x n , y n , t n represent the spatio-temporal positions of eye movement features, and f n , p n , n n represent the latent variables fused by the encoder, and f n is the eye movement feature vector, and p n is the characteristic vector of the fixation point, and n n Fixation point vector, R j represents the region label related to competency, and C i , OB i,j , S i,j represent the competency score label.

[0141] Thus, the eye movement feature label dataset F containing competency information after fusion is obtained G , and the dataset is divided into a training set and a validation set according to the ratio of 8:2.

[0142] S2.2 Establish the score matrix E for describing competency evaluation comp , construct a scoring network with the Spatiotemporal Random Forest (STRF) as the kernel, and input the eye movement features F containing competency information labels G , and output the predicted competency evaluation score matrix E comp .

[0143] Define the competency evaluation score matrix E comp , describe the evaluation results of the competency evaluation method based on the core competency behavior index framework:

[0144]

[0145] Among them, the matrix E compThe element represents the score of the j-th sub-item of the i-th type of competency, where i is the competency category index with a total of 9 categories, and j is the sub-item index under the corresponding category, j ∈ [j1, j9], E comp (i, j) ∈ [1, 5].

[0146] The constructed STRF network f STRF (·) consists of multiple random trees {T1, T2,..., T T} Each tree T t is independently trained. A candidate split dimension of dimension k is randomly selected from the features f G , and the information gain is used as the split criterion for recursive splitting. During splitting, when a certain feature has a high correlation at consecutive time points or in a spatial neighborhood, the importance index W G that synthesizes the spatio-temporal characteristics is calculated by weighting to adjust feature selection: FI where w

[0147]

[0148] represents the weight of time-related features, w temporal represents the weight of space-related features, I(·) represents the evaluation function of importance, spatial represents the time-related part of the feature f G G , represents the space-related part of the feature f G G

[0149] The leaf node of each tree finally outputs a local prediction as the target score of the samples within the leaf node, that is, the sub-item of the competency OB i,j gets a score. The integrated prediction of multiple trees is calculated by the weighted average of all trees to obtain a sub-item of E comp where T is the number of random trees, represents the prediction of the t-th tree for the score E ij Some E ij constitute the evaluation matrix of the prediction, the competency evaluation score matrix E comp .

[0150] S2.3 Establish a pilot key competency evaluation model with the competency evaluation score matrix E comp as the core. By setting a score threshold for each sub-item of the competency evaluation score matrix E comp , judge the sub-items with lower competency scores, and record the sub-items of the core competency behavior index framework with scores lower than the threshold as competency missing items OB i,j , and generate the competency missing matrix D OBj, achieve the identification of competency deficiency. According to the competency deficiency matrix, output the target sub-items of the pilot's EBT training for the key competency deficiency items corresponding to the low scores.

[0151] The output competency deficiency matrix D OBj :

[0152]

[0153] Where the element OB i,j is the competency deficiency of the pilot during the EBT training process, and the output deficient competency is also the target competency for the next EBT training.

[0154] Embodiment 2

[0155] The present invention also provides a method and system for identifying the target competency of a pilot's EBT training driven by the fusion of eye movement / flight data. The system is used to implement any one of the methods described above. The system includes: a first extraction module, a second extraction module, and an evaluation module;

[0156] The first extraction module is used to obtain the eye movement data of the pilot during the EBT training process, construct a GDETC network to realize the spatio-temporal trajectory clustering of the eye movement data, and extract the pilot's gaze spatio-temporal trajectory;

[0157] The second extraction module is used to obtain the EBT competency evaluation worksheet, flight data, standard baseline flight data, and parameter thresholds after training, construct a multi-source data fusion hierarchical extraction network, and extract the flight data stream associated with the error spatio-temporal point, the OBj data stream associated with the score and competency;

[0158] The evaluation module is used to construct a VAE-STRF network, fuse the data extracted by the first extraction module and the second extraction module, extract the pilot's key competency evaluation score matrix, establish a pilot's key competency evaluation model, and output the target competency of the EBT training.

[0159] In this embodiment, the first extraction module includes: an eye movement feature set extraction unit, a trajectory similarity matrix generation unit, and a gaze spatio-temporal trajectory feature extraction unit;

[0160] The eye movement feature set extraction unit is used to collect the eye movement data of the pilot during the EBT training process, perform regional segmentation on the cockpit visual space, associate the in-cockpit visual space information with the eye movement data, and extract the spatio-temporally associated eye movement feature set;

[0161] The trajectory similarity matrix generation unit is used to slice the eye movement feature set through the sliding dynamic time window technology, align the non-linearly changing eye movement data in different time segments by the dynamic time warping method, and calculate the similarity through point-by-point matching to ensure data accuracy and capture the subtle differences between trajectories, forming the eye movement spatio-temporal trajectory similarity matrix D DTW_0 , set the similarity threshold θ DTW Filter out invalid features to form the trajectory similarity matrix D for the next step DTW_new ;

[0162] The gaze spatio-temporal trajectory feature extraction unit is used to input the sliced eye movement feature G k and the gaze trajectory similarity matrix D DTW_new into the constructed GDETC network, calculate the density of the trajectory, cluster all time segment trajectories into multiple clusters, and each cluster represents a gaze behavior pattern. Among them, the low-density trajectory cluster represents atypical behavior, and the high-density trajectory cluster represents the gaze behavior pattern. The feature data through the center of the high-density trajectory cluster constitutes the gaze spatio-temporal trajectory feature F gaze ;

[0163] Among them, the spatio-temporal correlated eye movement feature set is: G EBT = D EBT ∪A cockpit , where A cockpit is the cockpit eye movement visual space feature, and D EBT is the eye movement data;

[0164] The trajectory similarity matrix D DTW_new is: In the formula, the retained element D DTW_new (m,n) represents the pair of spatio-temporal eye movement segments with higher similarity, and more gaze trajectory rules are retained between these segments;

[0165] The gaze spatio-temporal trajectory feature F gaze is: In the formula, k represents the trajectory segment number, i represents the trajectory point number, j represents the region number, represents the coordinate of the i-th fixation point of the k-th trajectory, represents the eye movement feature vector of the i-th fixation point of the k-th trajectory, represents the time point of the i-th fixation point of the k-th trajectory, represents the characteristic vector of the i-th fixation point of the k-th trajectory itself, represents the i-th fixation point vector of the k-th trajectory, represents the j-th visual region associated with the i-th fixation point of the k-th trajectory, represents the i-th fixation point of the k-th trajectory in the region total residence time, indicates the number of fixations of the i-th fixation point of the k-th trajectory in the region , indicates the average fixation duration of the i-th fixation point of the k-th trajectory in the region ; indicates the region switching frequency of the i-th fixation point of the k-th trajectory.

[0166] In this embodiment, the second extraction module includes: a flight data and evaluation work order acquisition unit, a flight data stream extraction unit, and an OBj data stream extraction unit;

[0167] The flight data and evaluation work order acquisition unit is used to obtain the simulated flight data Q max_deviation and the standard baseline flight data Q base by referring to the standard and based on the deviation threshold ∈ fobs determined by experts, FSTD and the trained EBT competency evaluation work order D

[0168] The flight data stream extraction unit is used to construct the deviation mapping architecture I of the SFHE network and extract the flight data stream Q of the structured data associated with the error spatio-temporal points error ;

[0169] The OBj data stream extraction unit is used to construct the structure data extraction architecture II of the SFHE network, input the FSTD data stream D FSTD , and extract the OBj data stream C of the associated scores and competencies in the EBT evaluation work order score ;

[0170] Among them, the trained EBT competency evaluation work order D FSTD is: D FSTD ={D1, D2,..., D n}, where n is the total number of FSTD work orders included, and D n is the FSTD text data of a single copy;

[0171] The flight data stream Q error is: In the formula, t is time, a is a parameter, j is a position, R j is the cockpit space mapping function, T error is the set of operation error times, P error is the set of operation error parameters, P error (t) is the set of error positions, and t n represents the time point associated with the n-th error point in the flight data stream Q error , and a dn represents the flight data stream Q errorThe parameter a with an error associated with the nth error point in d , R jn represents the flight data stream Q error The cockpit spatial position R associated with the error that appears at the nth error point in j ;

[0172] OBj data stream C score is: In the formula, C i represents the i-th type of competency, OB i,j represents the j-th sub-item of the i-th type of competency, S i,j represents the score corresponding to the j-th sub-item under the i-th type of competency, S i,j ∈[1, 5], where i is the competency category index with a total of 9 categories, and j is the sub-item index under the corresponding category.

[0173] In this embodiment, the evaluation module includes: an eye movement feature label dataset generation unit, a competency evaluation score matrix generation unit, and a competency evaluation score matrix generation unit;

[0174] The eye movement feature label dataset generation unit is used to perform feature fusion on the gaze spatio-temporal trajectory feature F gaze , the associated score and the OBj data stream Q of the competency error , the flight data stream C associated with the associated error spatio-temporal point score to form an eye movement feature label dataset F containing competency information G ;

[0175] The competency evaluation score matrix generation unit is used to establish a score matrix E describing the competency evaluation comp , construct a scoring network with a spatio-temporal random forest network STRF as the kernel, input the eye movement feature F containing the competency information label G , and output the predicted competency evaluation score matrix E comp ;

[0176] The competency evaluation score matrix generation unit is used to establish a pilot key competency evaluation model with the competency evaluation score matrix E comp as the core, set a scoring threshold to judge the missing item OB of the competency i,j , and generate a competency missing matrix D OBj , realize the identification of competency missing, and output the target competency for EBT training;

[0177] Among them, the eye movement feature label dataset F G is:

[0178]

[0179] Wherein, f Gn represents a feature vector in the dataset, x n , y n , t n represent the spatio-temporal positions of eye movement features, f n , p n , n n represent the latent variables fused by the encoder, f n is the eye movement feature vector, p n is the characteristic vector of the fixation point, n n is the fixation point vector, R j represents the area label related to the competency, C i , OB i,j , S i,j represent the competency scoring labels;

[0180] The competency evaluation score matrix E comp is:

[0181]

[0182] Wherein, the elements of the matrix E comp represent the scores of the j-th sub-item of the i-th type of competency, where i is the competency category index with a total of 9 categories, and j is the sub-item index under the corresponding category, j ∈ [j1, j9], E comp (i, j) ∈ [1, 5];

[0183] The output competency missing matrix D OBj is:

[0184]

[0185] Wherein, the element OB i,j is exactly the competency missing of the pilot during the EBT training, and the output missing competency is also the target competency for the next EBT training.

[0186] The above-described embodiments are only descriptions of the preferred embodiments of the present invention, and do not limit the scope of the present invention. Without departing from the design spirit of the present invention, various deformations and improvements made by those of ordinary skill in the art to the technical solutions of the present invention shall fall within the protection scope determined by the claims of the present invention.

Claims

1. A method for identifying the competency of pilots in EBT training driven by the fusion of eye movement / flying data, characterized in that, The method includes: S0: Obtain the eye movement data during the EBT training process of the pilot, construct a GDETC network to achieve spatio-temporal trajectory clustering of the eye movement data, and extract the spatio-temporal gaze trajectory of the pilot; S1: Obtain the trained EBT competency assessment worksheet, flight data, standard baseline flight data, and parameter thresholds, construct a multi-source data fusion hierarchical extraction network, and extract the flight data stream associated with error spatio-temporal points, the OBj data stream associated with scores and competencies; S2: Construct a VAE-STRF network, fuse the data extracted in S0 and S1, extract the pilot's key competency assessment score matrix, establish a pilot's key competency assessment model, and output the target competency of the EBT training.

2. The method according to claim 1, wherein In S0, obtaining the eye movement data during the EBT training process of the pilot, constructing a GDETC network to achieve spatio-temporal trajectory clustering of the eye movement data, and extracting the spatio-temporal gaze trajectory of the pilot includes: S0.1: Collect the eye movement data of the pilot during the EBT training process, perform regional segmentation on the cockpit visual space, associate the in-cockpit visual space information with the eye movement data, and extract the spatio-temporally associated eye movement feature set; S0.2: Slice the eye movement feature set through the sliding dynamic time window technology, align the non-linearly changing eye movement data in different time segments through the dynamic time warping method, and perform similarity calculation through point-by-point matching to ensure data accuracy and capture the tiny differences between trajectories, forming the eye movement spatio-temporal trajectory similarity matrix D DTW_0 , set the similarity threshold θ DTW Filter out invalid features to form the trajectory similarity matrix D for the next step DTW_new ; S0.3: Slice the eye movement feature G k and the gaze trajectory similarity matrix D DTW_new into the constructed GDETC network, calculate the density of the trajectories, cluster all the time segment trajectories into multiple clusters, where each cluster represents a gaze behavior pattern. Among them, the low-density trajectory cluster represents atypical behavior, and the high-density trajectory cluster represents the gaze behavior pattern. The feature data passing through the center of the high-density trajectory cluster constitutes the gaze spatio-temporal trajectory feature F gaze ; Among them, the set of spatio-temporal correlated eye movement features is: G EBT = D EBT ∪ A cockpit , where A cockpit is the cockpit eye movement visual space feature, and D EBT is the eye movement data; Trajectory similarity matrix D DTW_new is as follows: The retained element D in the formula DTW_new (m,n) represents pairs of spatio-temporal saccade segments with relatively high similarity, and more fixation trajectory patterns are retained between these segments; Gaze spatio-temporal trajectory feature F gaze is: In the formula, k represents the trajectory segment number, i represents the trajectory point number, and j represents the region number, represents the coordinates of the i-th fixation point of the k-th trajectory, represents the eye movement feature vector of the i-th fixation point of the k-th trajectory, represents the time point of the i-th fixation point of the k-th trajectory, represents the characteristic vector of the i-th fixation point of the k-th trajectory itself, represents the i-th fixation point vector of the k-th trajectory, represents the j-th visual region associated with the i-th fixation point of the k-th trajectory, represents the total residence time of the i-th fixation point of the k-th trajectory in the region ; represents the number of fixations of the i-th fixation point of the k-th trajectory in the region ; represents the average fixation duration of the i-th fixation point of the k-th trajectory in the region ; represents the region switching frequency of the i-th fixation point of the k-th trajectory.

3. The method according to claim 2, wherein In S1, obtaining the trained EBT competency assessment worksheet, flight data, standard baseline flight data, and parameter thresholds, constructing a multi-source data fusion hierarchical extraction network, and extracting the flight data stream associated with error spatio-temporal points, the OBj data stream associated with scores and competencies includes: S1.1: Determine the deviation threshold ∈ of flight parameters by referring to the standards and based on the experts max_deviation and the standard baseline flight data Q base , and obtain the simulator flight data Q fobs and the trained EBT competency assessment worksheet D FSTD ; S1.2: Construct the deviation mapping architecture I of the SFHE network and extract the flight data stream Q of the structured data associated with the error time and space points error ; S1.3: Construct the structure data extraction architecture II of the SFHE network, and input the FSTD data stream D FSTD , and extract the OBj data stream C of the associated score and competency in the EBT assessment work sheet score ; Among them, the trained EBT competency assessment worksheet D FSTD is: D FSTD = {D1, D2,..., D n}, where n is the total number of FSTD worksheets included, and D n is the FSTD text data for a single copy; Flight data stream Q error is: In the formula, t is time, a is a parameter, j is a position, R j is the cockpit space mapping function, T error is the set of operation error times, P error is the set of operation error parameters, P error (t) is the set of error positions, t n represents the time point associated with the nth error point in the flight data stream Q error ; a dn represents the parameter a with an error that occurs at the nth error point in the flight data stream Q error ; R d represents the cockpit space position R associated with the error that occurs at the nth error point in the flight data stream Q jn ; error ; j ; OBj data stream C score is as follows: In the formula, C i represents the i-th type of competency, OB i,j represents the j-th sub-item of the i-th type of competency, S i,j represents the score corresponding to the j-th sub-item under the i-th type of competency, S i,j ∈[1, 5], where i is the competency category index with a total of 9 categories, and j is the sub-item index under the corresponding category.

4. The method according to claim 3, characterized in that, In S2, constructing a VAE-STRF network, fusing the data extracted in S0 and S1, extracting the pilot's key competency assessment score matrix, establishing a pilot's key competency assessment model, and outputting the target competency of the EBT training includes: S2.1: Based on the variational autoencoder VAE, the gaze spatio-temporal trajectory feature F gaze , the OBj data stream Q of the correlation score and competency error , the flight data stream C of the correlated error spatio-temporal points score are feature-fused to form an eye movement feature label dataset F containing competency information G ; S2.2: Establish a score matrix E for describing competency evaluation comp , construct a scoring network with the spatio-temporal random forest network STRF as the kernel, and input the eye movement feature F containing the competency information label G , and output the predicted competency evaluation score matrix E comp ; S2.3: Establish a pilot key competency assessment model with the competency assessment score matrix E comp as the core, set a scoring threshold to identify the competency deficiency items OB i,j , and generate a competency deficiency matrix D OBj , realize the identification of competency deficiencies, and output the target competencies for EBT training; Among them, the eye movement feature label data set F G is as follows: Where, f Gn represents a feature vector in the dataset, x n , y n , t n represent the spatio-temporal positions of eye movement features, f n , p n , n n represent the latent variables fused by the encoder, f n is the eye movement feature vector, p n is the feature vector of the fixation point, n n is the fixation point vector, R j represents the region label related to competency, C i , OB i,j , S i,j represent the competency score label; Competency Assessment Score Matrix E comp is as follows: In the formula, matrix E comp The elements of represent the score of the j-th sub-item of the i-th type of competency, where i is the competency category index with a total of 9 categories, and j is the sub-item index under the corresponding category, j ∈ [j1, j9], E comp (i, j) ∈ [1, 5]; Output competency deficiency matrix D OBj is as follows: In the formula, the element OB i,j is the competency missing in the pilot's EBT training process, and the output missing competency is also the target competency for the next EBT training.

5. A method and system for identifying the pilot EBT training target competency driven by the fusion of eye movement / flying data, the system is used to implement the method described in any one of claims 1-4, characterized in that, The system includes: a first extraction module, a second extraction module, and an evaluation module; The first extraction module is used to obtain the eye movement data during the EBT training process of the pilot, construct a GDETC network to achieve spatio-temporal trajectory clustering of the eye movement data, and extract the spatio-temporal gaze trajectory of the pilot; The second extraction module is used to obtain the trained EBT competency assessment worksheet, flight data, standard baseline flight data, and parameter thresholds, construct a multi-source data fusion hierarchical extraction network, and extract the flight data stream associated with error spatio-temporal points, the OBj data stream associated with scores and competencies; The evaluation module is used to construct a VAE-STRF network, fuse the data extracted in the first extraction module and the second extraction module, extract the pilot's key competency assessment score matrix, establish a pilot's key competency assessment model, and output the target competency of the EBT training.

6. The system according to claim 5, wherein The first extraction module includes: an eye movement feature set extraction unit, a trajectory similarity matrix generation unit, and a gaze spatio-temporal trajectory feature extraction unit; The eye movement feature set extraction unit is used to collect the eye movement data of the pilot during the EBT training process, perform regional segmentation on the cockpit visual space, associate the in-cockpit visual space information with the eye movement data, and extract the spatio-temporally associated eye movement feature set; The trajectory similarity matrix generation unit is used to slice the eye movement feature set through the sliding dynamic time window technology, align the non-linearly changing eye movement data in different time segments through the dynamic time warping method, and perform similarity calculation through point-by-point matching to ensure data accuracy and capture the tiny differences between trajectories, forming the eye movement spatio-temporal trajectory similarity matrix D DTW_0 , set the similarity threshold θ DTW Filter out invalid features to form the trajectory similarity matrix D for the next step DTW_new ; The gaze spatio-temporal trajectory feature extraction unit is used to slice the eye movement feature G k and the gaze trajectory similarity matrix D DTW_new into the constructed GDETC network, calculate the density of the trajectory, cluster all the time segment trajectories into multiple clusters, each cluster representing a gaze behavior pattern, where the low-density trajectory cluster represents atypical behavior, and the high-density trajectory cluster represents the gaze behavior pattern. The feature data passing through the center of the high-density trajectory cluster constitutes the gaze spatio-temporal trajectory feature F gaze ; Among them, the set of spatio-temporal correlated eye movement features is: G EBT = D EBT ∪ A cockpit , where A cockpit is the cockpit eye movement visual space feature, and D EBT is the eye movement data; Trajectory similarity matrix D DTW_new is as follows: In the formula, the retained element D DTW_new (m,n) represents pairs of spatio-temporal saccade segments with relatively high similarity, and more fixation trajectory patterns are retained between these segments; Gaze spatio-temporal trajectory feature F gaze is: In the formula, k represents the trajectory segment number, i represents the trajectory point number, and j represents the region number. represents the coordinates of the i-th fixation point of the k-th trajectory segment. represents the eye movement feature vector of the i-th fixation point of the k-th trajectory segment. represents the time point of the i-th fixation point of the k-th trajectory segment. represents the characteristic vector of the i-th fixation point itself of the k-th trajectory segment. represents the i-th fixation point vector of the k-th trajectory segment. represents the j-th visual region associated with the i-th fixation point of the k-th trajectory segment. represents the i-th fixation point of the k-th trajectory segment in the region total residence time. represents the i-th fixation point of the k-th trajectory segment in the region number of fixations. represents the i-th fixation point of the k-th trajectory segment in the region average fixation duration. represents the region switching frequency of the i-th fixation point of the k-th trajectory segment.

7. The system according to claim 6, wherein The second extraction module includes: a flight data and assessment worksheet acquisition unit, a flight data stream extraction unit, and an OBj data stream extraction unit; The flight data and evaluation worksheet acquisition unit is used to obtain the simulator flight data Q max_deviation and the standard baseline flight data Q base by referring to the standard and based on the deviation threshold ∈ fobs of the flight parameters determined by experts, and the EBT competency evaluation worksheet D FSTD after training; The flight data stream extraction unit is used to construct the deviation mapping architecture I of the SFHE network and extract the flight data stream Q of the structured data associated error spatio-temporal points error ; The OBj data stream extraction unit is used to construct the structure data extraction architecture II of the SFHE network, and inputs the FSTD data stream D FSTD , and extracts the OBj data stream C of the associated score and competency in the EBT assessment work sheet score ; Among them, the trained EBT competency assessment worksheet D FSTD is: D FSTD = {D1, D2,..., D n}, where n is the total number of FSTD worksheets included, and D n is the FSTD text data for a single copy; Flight Data Stream Q error for: In the formula, t is time, a is parameter, j is position, R j is the cockpit space mapping function, T error is the set of operation error time, P error is the set of operation error parameters, P error (t) is the error position set, t n Indicates the flight data flow Q error The time point associated with the nth error point in dn Indicates the flight data flow Q error The error parameter a associated with the nth error point in d , R jn Indicates the flight data flow Q error The cockpit spatial position R associated with the error at the nth error point in j ; OBj data stream C score is: In the formula, C i represents the i-th type of competency, OB i,j represents the j-th sub-item of the i-th type of competency, S i,j represents the score corresponding to the j-th sub-item under the i-th type of competency, S i,j ∈[1, 5], where i is the competency category index with a total of 9 categories, and j is the sub-item index under the corresponding category.

8. The system according to claim 7, wherein The evaluation module includes: an eye movement feature label data set generation unit, a competency evaluation score matrix generation unit, and a competency evaluation score matrix generation unit; The eye movement feature label dataset generation unit is used to generate the eye movement feature label dataset F containing competency information by performing feature fusion on the gaze spatio-temporal trajectory feature F gaze , the OBj data stream Q of the association score and competency error , the flight data stream C of the associated error spatio-temporal points score ; G ; The competency evaluation score matrix generation unit is used to establish a score matrix E for describing competency evaluation comp , construct a scoring network with a spatio-temporal random forest network STRF as the kernel, and input the eye movement feature F containing the competency information label G , and output the predicted competency evaluation score matrix E comp ; The competency evaluation score matrix generation unit is used to establish a key pilot competency evaluation model with the competency evaluation score matrix E comp as the core, set a scoring threshold to determine the competency missing item OB i,j , and generate a competency missing matrix D OBj , realize the identification of competency missing, and output the target competency for EBT training; Among them, the eye movement feature label data set F G is as follows: where, f Gn represents a feature vector in the dataset, x n , y n , t n represent the spatio-temporal positions of eye movement features, f n , p n , n n represent the latent variables fused by the encoder, f n is the eye movement feature vector, p n is the feature vector of the fixation point, n n the fixation point vector, R j represents the region label related to competency, C i , OB i,j , S i,j represent the competency score label; Competency Assessment Score Matrix E comp is as follows: In the formula, matrix E comp The elements of represent the scores of the j-th sub-item of the i-th type of competency, where i is the competency category index with a total of 9 categories, and j is the sub-item index under the corresponding category, j ∈ [j1, j9], E comp (i, j) ∈ [1, 5]; The output competency deficiency matrix D OBj is as follows: In the formula, the element OB i,j is the competency that the pilot lacks during the EBT training process, and the output lacking competency is also the target competency for the next EBT training.

Images