Pilot control ability evaluation method and system based on multimodal data and online learning

Abstract

The present invention discloses a pilot control ability evaluation method and system based on multimodal data and online learning, including: determining an input parameter set, establishing a flight visual temporal fusion network, and extracting each feature of the parameter set. Feature fusion of multimodal data is completed through attention mechanism and global aggregation. A pilot control ability evaluation architecture is constructed, and an offline pilot control ability evaluation model is generated by designing a loss function based on comprehensive scoring error and sub-scoring consistency constraints. A sliding window mechanism is used to segment and preprocess the flight data and eye movement data streams generated in real time. A lightweight model based on pruning and parameter quantization technology is designed for real-time scoring requirements, and the feature expression capability in the offline pilot control ability evaluation model is migrated to a lightweight online evaluation model. The present invention can be used to improve pilot training programs, optimize the human-machine interface design of the cockpit, and improve the efficiency and safety of flight operations.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology and flight data application, and specifically relates to a pilot control ability evaluation method and system based on multimodal data and online learning. Background Art

[0002] Pilot competency assessment is a crucial component of aviation safety and pilot training optimization. Its scientific nature and real-time nature directly impact the safety and efficiency of flight operations. However, existing pilot competency assessment methods still have many deficiencies in accuracy, dynamic adaptability, and multi-source data utilization, primarily manifested in the following aspects:

[0003] First, there are the limitations of single-modal data. Traditional assessment methods often rely on a single data source (such as flight status parameters or simulation scores), making it difficult to capture the complex dynamic interactions between pilot behavior and flight status. The lack of effective integration of multimodal data results in incomplete and inaccurate assessment results.

[0004] Secondly, there is a lack of dynamic learning and real-time performance. Existing methods are usually based on pre-designed static models or rules, and lack the ability to dynamically capture data changes during flight operations. Especially in real-time scenarios, it is difficult to adapt to the rapid changes in different operating conditions.

[0005] Finally, the refinement and quantification of assessment results are imperfect. Traditional assessment methods often provide only a simple overall score, lacking a multi-dimensional analysis of pilots' specific operational capabilities (such as stability, attention allocation, and abnormal behavior). This single-scoring model has limitations in guiding pilot training optimization and flight safety management.

[0006] Therefore, a method that combines multimodal data fusion, dynamic learning and real-time evaluation technology is needed to comprehensively and dynamically evaluate the pilot's operational capabilities, thereby providing a scientific basis for pilot training optimization and cockpit design, and further improving flight safety. Summary of the Invention

[0007] To address the shortcomings of existing pilot capability assessment methods in multimodal data fusion, dynamic learning, and real-time evaluation, this paper provides a pilot control capability assessment method and system based on multimodal data and online learning. This method combines pilot eye movement data with flight data, using multimodal data fusion and dynamic learning to quantitatively score and analyze pilot control capabilities. Furthermore, through a lightweight model and online training mechanism, comprehensive, dynamic, and real-time assessment of pilot control capabilities is achieved.

[0008] To achieve the above object, the present invention provides the following solutions:

[0009] A pilot control ability assessment method based on multimodal data and online learning, the method comprising:

[0010] S0: Determine the flight data parameter set and eye movement data parameter set related to the pilot's control, establish a flight visual temporal fusion network, extract the features of the flight data parameter set and the eye movement data parameter set, and perform multimodal fusion of the features of the flight data and eye movement data through attention mechanism and global aggregation to generate a global comprehensive feature set for control ability assessment;

[0011] S1: Based on the global comprehensive feature set, a pilot control ability evaluation framework is constructed. By designing a loss function based on the comprehensive score error and sub-score consistency constraints, the AdamW optimization algorithm is used to complete the training, generate an offline pilot control ability evaluation model, and output the pilot control ability comprehensive score and corresponding sub-scores;

[0012] S2: Based on the comprehensive score of the pilot's control ability and the corresponding sub-scores, a sliding window mechanism is used to segment and preprocess the real-time flight data and eye movement data streams. A lightweight model based on pruning and parameter quantization technology is designed to meet the real-time scoring requirements. The feature expression capabilities of the offline pilot control ability evaluation model are transferred to a lightweight online evaluation model, and the pilot's control ability score is output in real time.

[0013] Preferably, in said S0, determining the flight data parameter set and the eye movement data parameter set related to the pilot's manipulation includes: defining the flight data parameter set as: S = S control ∪S state ∪S anomaly , the eye movement data parameter set is defined as: E = E fixation ∪E saccade ∪E pupil ;

[0014] Among them, the aircraft operating parameter set includes the operating parameters that reflect the pilot's control of the aircraft: S control ={s thrust ,s aileron ,s rudder ,s elevator ,s stick ,s pedal} Among them, s thrust Indicates thrust lever operation, s aileron Indicates aileron operation, s rudder Indicates rudder operation, s elevator Indicates elevator operation, s stick Indicates side stick operation, s pedal Indicates pedal operation;

[0015] The flight status parameter set contains parameters that reflect the real-time status of the aircraft: Sstate ={s altitude ,s airspecd ,s pitch ,s roll ,s yaw ,s track ,s vert_speed ,s mach} Among them, s altitude Indicates the standard atmospheric altitude, s airspecd Indicates airspeed, s pitch represents the pitch angle, s roll represents the roll angle, s yaw represents the yaw angle, s track represents the track angle, s vert_speed Indicates vertical speed, s mach represents the Mach number;

[0016] The flight abnormality parameter set contains parameters that reflect abnormal status: S anomaly ={s overspeed ,s stall ,s vibration ,s wind} Among them, s overspeed Indicates speeding, s stall Indicates stall, s vibration Indicates vibration, s wind Indicates wind shear;

[0017] The gaze behavior parameter set contains parameters that reflect the pilot's gaze behavior: fixation ={e fix_point ,e fix_duration ,e heatmap} Among them, e fix_point represents the gaze point position, e fix_duration represents the fixation duration, e heatmap represents the gaze heat map;

[0018] The scanning behavior parameter set contains parameters that reflect the pilot's scanning behavior: saccade ={e sac_path ,e sac_speed ,e sac_angle} Among them, e sac_path represents the scanning path, e sac_speed represents the scanning speed, e sac_angle Indicates the scanning angle;

[0019] The pupil characteristic parameter set includes pupil characteristic parameters that reflect the pilot's psychological load: E pupil ={e pupil_size ,e pupil_response} Among them, e pupil_size represents the pupil diameter, e pupil_responseIndicates pupillary reaction to light.

[0020] Preferably, in S0, establishing a flight visual temporal fusion network and extracting features of a flight data parameter set and an eye movement data parameter set includes:

[0021] For flight data, a time series feature extraction module based on one-dimensional convolutional neural network (1D-CNN) and long short-term memory network (LSTM) is established to extract the short-term dynamic features F at any moment through 1D-CNN. 1D-CNN , and use LSTM to capture the global dependency of time series and generate the global feature set F LSTM ;

[0022] For eye movement data, a spatial-temporal feature extraction module based on convolutional neural network (CNN) and temporal convolutional network (TCN) is constructed to extract the spatial distribution feature F of the gaze point through CNN. CNN , using TCN to capture the time series dynamic characteristics of the gaze trajectory, and finally forming the global feature F of the eye movement data TCN ;

[0023] Among them, F 1D-CNN ={f thrust ,f aileron ,f rudder ,f elevator ,f stick ,f pedal ,f vibration ,f overspeed ,f stall}, where f thrust ,f aileron , f rudder , f elevator , f stick , f pedal , f vibration , f overspeed , f stall Respectively represent the local dynamic features extracted by each parameter;

[0024] F LSTM ={f altitude ,f airspeed ,f pitch ,f roll ,f yaw ,f track ,f vert_speed ,f mach ,f wind}, where f altitude ,f airspeed ,

[0025] f pitch , f roll , f yaw , ftrack , f vert_speed , f mach , f wind Respectively represent the global temporal dependency features extracted by each parameter;

[0026] F CNN ={f fix_point ,f heatmap ,f sac_path}, where f fix_point ,f heatmap ,f sac_path Respectively represent the spatial characteristics of each parameter;

[0027] F TCN ={f fix_duration ,f sac_speed ,f sac_angle ,f pupil_size ,f pupil_response}, where f fix_duration ,f sac_speed , f sac_angle , f pupil_size , f pupil_response Represents the temporal dynamic characteristics extracted by each parameter.

[0028] Preferably, in S0, multimodal fusion of flight data and eye movement data features is performed through attention mechanism and global aggregation to generate a global comprehensive feature set for maneuverability assessment, including:

[0029]

[0030] in: represents the global feature set of operational stability, represents the global feature set of attention allocation, represents the global feature set of abnormal behavior, T is the total time step, represents the feature set after the operational stability dimension is fused at time t, represents the feature set after the fusion of attention allocation dimensions at time t, Represents the feature set after the abnormal behavior dimension is fused at time t.

[0031] Preferably, in S1, a pilot control ability evaluation framework is constructed based on the global comprehensive feature set, and a loss function based on comprehensive score error and sub-score consistency constraints is designed. The AdamW optimization algorithm is used to complete training and generate an offline pilot control ability evaluation model. The output of the pilot control ability comprehensive score and the corresponding sub-scores include:

[0032] S1.1: Construct a pilot control evaluation framework, specifically including: establishing a feature encoding layer with residual connections, gradually compressing and refining multimodal features through a multi-layer fully connected network; introducing a multi-head attention mechanism to capture the interactive relationship between different modal features; combining the task branch module to score operational stability, attention allocation, and abnormal behavior respectively, and finally weighting the pilot to generate a comprehensive score S total ;

[0033] S1.2: Design a loss function L that includes comprehensive rating error loss and sub-rating consistency constraints total , combined with the definitions of operational stability and reasonable distribution of gaze tracks in the flight manual, and adding constraints on operational specifications and behavioral consistency, the accuracy and consistency of the model scoring results are ensured by minimizing the loss function;

[0034] S1.3: Divide the global comprehensive feature dataset into training set D train , validation set D val and the test set D test , the data is labeled into high score, normal score and low score categories according to the flight expert scores, and the AdamW optimizer is used to complete the training to generate an offline pilot control ability evaluation model, and finally output the comprehensive score of the pilot control ability and the corresponding sub-score S sub ={S stab ,S att ,S anom}.

[0035] Preferably, in S1.2, a loss function L including comprehensive score error loss and sub-score consistency constraint is designed. total In combination with the definitions of operational stability and reasonable distribution of gaze tracks in the flight manual, we added constraints on operational specifications and behavioral consistency, and ensured the accuracy and consistency of the model scoring results by minimizing the loss function. These include:

[0036] M is used to represent the pilot control evaluation model, θ is used to represent the corresponding model parameters, and the quantitative score value output by the network is recorded as The input data is any fusion feature sequence The actual rating is S total,true , given the data set participating in network training, the model is trained by minimizing the loss function, and the loss function is in the form of:

[0037]

[0038] In the formula, the first term is the mean square error between the predicted value and the true value of the comprehensive score; represents the comprehensive score of the i-th sample predicted by the model, represents the true comprehensive score of the i-th sample, i.e., the expert score; the second term is the weighted consistency of the constrained comprehensive score and the sub-scores; represents the j-th subtask score of the i-th sample predicted by the model, namely, operational stability, attention allocation, and abnormal behavior; w j represents the weight of the subtask score, indicating the relative contribution of different subtasks to the overall score; γ: the weight of the weighted consistency constraint;

[0039]

[0040] Among them, L stab represents the error loss that combines operational stability and control input smoothness constraints, Δu i is the control input change in the time step, Δu max The reasonable range specified in the flight manual; L att Indicates entropy (P gace,i ) combines the error loss of attention allocation and the entropy constraint of gaze distribution to form the gaze distribution entropy, H optimal The reasonable entropy range defined for the flight manual; L anom Denotes the error loss that combines abnormal behavior and the KL divergence constraint of behavior pattern, D KL (P action,i ||P ref ) represents the deviation of the behavior distribution from the reference behavior distribution;

[0041] When the detected comprehensive score is too high, that is, exceeds the specified maximum score, or too low, that is, is less than the minimum reasonable score threshold, a loss value that meets the preset requirements is assigned, and the mean square error (MSE) and the constraint satisfaction rate are used as model performance indicators; when the mean square error reaches the preset range and the constraint satisfaction rate is higher than 95%, the model training is considered complete.

[0042] Preferably, in S1.3, the global comprehensive feature dataset is divided into a training set D train , validation set D val and the test set D test , the data is labeled into high score, normal score and low score categories according to the flight expert scores, and the AdamW optimizer is used to complete the training to generate an offline pilot control ability evaluation model, and finally output the comprehensive score of the pilot control ability and the corresponding sub-score S sub ={S stab ,S att ,S anom}include:

[0043] First, divide the dataset into D train , D val , D test, respectively for training, validation and testing, each data set is based on the comprehensive score S of flight experts total,true Divide the data into high-score data, normal data and low-score data;

[0044] Data annotation is performed based on the percentile range of the flight expert's comprehensive score. The specific rules are as follows: S total,true When it is in the top 30% of the score, it is marked as high-scoring data; when S total,true When the score is between 30% and 70%, it is marked as normal data; when S total,true When it is in the bottom 30% of the score, it is marked as low-scoring data;

[0045] Under the above rules, D train The number of samples of medium and high score data is N1, the number of samples of normal data is N2, the number of samples of low score data is N3, and D val and D test The number of samples in each category are M1, M2, and M3 respectively;

[0046] During the model training process, from the training set D train Randomly extract batch data B for gradient update, B contains high-scoring data, normal data and low-scoring data, and the number of samples in each category is B1, B2, and B3 respectively;

[0047] During the forward propagation process, the model input is multimodal fusion features The output is the comprehensive score prediction value and sub-score prediction value Through the comprehensive scoring loss function L total Calculate the total loss value of the current batch and use the AdamW optimizer to update the model parameters. The update rule is:

[0048]

[0049] Among them, λ is the learning rate and η is the weight decay coefficient;

[0050] After each round of training, on the validation set D val Evaluate the model performance, calculate the mean square error (MSE) and consistency error, and dynamically adjust the learning rate; when the MSE of the validation set is in the continuous T stop When there is no significant decrease in the round, stop training and output the optimized model parameters θ * , finally, on the test set D test Evaluate the model performance and record the test error.

[0051] Preferably, in S2, slicing and preprocessing the real-time generated flight data and eye movement data streams using a sliding window mechanism includes:

[0052] The real-time generated flight data and eye movement data are divided into continuous time segments using a fixed-length sliding window, and the length of each segment is T. w , the sliding step is S w , the fragment is represented as: Use the pre-trained offline pilot maneuverability evaluation model to predict the comprehensive score and sub-score for each time segment: where f eval Generate annotations for each clip based on the scoring results of the pre-trained pilot maneuver evaluation model Mark as high score, Mark as normal, Marked as low score;

[0053] Finally, the labeled data segments are divided into training set and validation set in a ratio of 8:2 to ensure that each category, i.e. high score, normal score, and low score, is evenly distributed in the training set and validation set.

[0054] Preferably, in S2, designing a lightweight model based on pruning and parameter quantization technology to transfer the feature expression capability in the offline pilot control ability evaluation model to the lightweight online evaluation model includes:

[0055] Using the complete pilot maneuver evaluation model as the teacher model, the pilot maneuver online evaluation model is trained through the knowledge distillation method:

[0056] L distill =α·L hard +β·L soft

[0057] Among them, L distill To train the loss function of the student model by distillation method in the pilot control ability evaluation task, L hard is the hard label loss, L soft is the soft label loss, α is the weight coefficient of the hard label loss, and β is the weight coefficient of the soft label loss;

[0058] The error between the student model’s predictions and the teacher model’s scoring labels is:

[0059]

[0060] in, is the predicted output of the teacher model, is the predicted output of the student model, CrossEntropy is the cross entropy loss function used to measure the difference between the outputs of the teacher and student models;

[0061] Minimize the difference between the teacher model and the student model by the temperature-corrected softmax output:

[0062]

[0063] Among them, z teacher is the original prediction score of the teacher model under the given input, z student is the raw prediction score of the student model under a given input, T is the temperature parameter, Softmax is the softmax function, which converts the output value into a probability form, and the sum of the probabilities of all categories is 1. KL is the KL divergence, which is a measure of the difference between two probability distributions;

[0064] The pre-training of the pilot maneuver online evaluation model is completed on the training set, and the validation set is used to evaluate the model performance, and finally the initial parameters θ are obtained. init .

[0065] The present invention also provides a pilot control ability evaluation system based on multimodal data and online learning, the system is used to implement any one of the methods described above, the system comprising: a data set construction module, an offline evaluation model construction module, and an online evaluation model construction module;

[0066] The dataset construction module is used to determine a set of flight data parameters and a set of eye movement data parameters related to pilot control, establish a flight visual temporal fusion network, extract features of the flight data parameter set and the eye movement data parameter set, and perform multimodal fusion of the features of the flight data and eye movement data through an attention mechanism and global aggregation to generate a global comprehensive feature set for control ability assessment;

[0067] The offline evaluation model construction module is used to construct a pilot control ability evaluation framework based on the global comprehensive feature set, design a loss function based on the comprehensive score error and sub-score consistency constraints, complete training using the AdamW optimization algorithm, generate an offline pilot control ability evaluation model, and output the pilot control ability comprehensive score and corresponding sub-scores;

[0068] The online evaluation model construction module is used to segment and preprocess the real-time generated flight data and eye movement data streams based on the pilot's comprehensive control ability score and the corresponding sub-scores using a sliding window mechanism. A lightweight model based on pruning and parameter quantization technology is designed to meet the real-time scoring requirements. The feature expression capabilities of the offline pilot control ability evaluation model are transferred to the lightweight online evaluation model, and the pilot's control ability score is output in real time.

[0069] Compared with the prior art, the present invention has the following beneficial effects:

[0070] (1) Multimodal data fusion and dynamic feature extraction capabilities. Traditional pilot control ability assessment methods mostly rely on single-modal data (such as flight status parameters), which makes it difficult to fully capture the complex interactive characteristics of pilot behavior and flight status. The present invention fuses flight data and pilot eye movement data, uses temporal convolutional networks (TCNs), long short-term memory networks (LSTMs), and convolutional neural networks (CNNs), extracts short-term dynamic features and global temporal features, and combines the attention mechanism to achieve feature fusion, which significantly improves the accuracy and comprehensiveness of control ability assessment.

[0071] (2) High-precision, multi-dimensional quantitative evaluation of maneuverability. Traditional evaluation methods are usually based on a single score, which makes it difficult to analyze the pilot's specific operational performance in detail. The present invention constructs an evaluation model that includes a comprehensive score and sub-task scores (such as operational stability, attention allocation, and abnormal behavior recognition), and designs a loss function based on the comprehensive score error and sub-score consistency constraints, providing refined evaluation results and providing an important reference for pilot training, ability assessment, and flight safety management.

[0072] (3) Real-time online evaluation and lightweight model design. Existing online evaluation methods usually have high requirements for computing resources and are difficult to be effectively applied in real-time scenarios. The present invention uses lightweight model design and knowledge distillation to transfer the capabilities of high-performance offline evaluation models to lightweight online models, and combines the sliding window mechanism to achieve sharding and scoring of real-time data streams. This method not only reduces computing costs, but also enables real-time monitoring and dynamic evaluation of pilot control capabilities in scenarios with limited computing resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0074] Figure 1 This is a flowchart of the main steps S0-S2 of the embodiment of the present invention;

[0075] Figure 2 The figure is a schematic diagram of the overall process of a pilot control ability evaluation method based on multimodality and online learning in an embodiment of the present invention. DETAILED DESCRIPTION

[0076] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0077] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0078] Example 1

[0079] like Figure 1 、 Figure 2 As shown, the present invention provides a multimodal and dynamic learning driven pilot capability assessment method, comprising the following steps:

[0080] S0: Determine the input parameter set based on the flight quality monitoring standards and the flight crew manual. Construct an Aerial Vision Temporal Fusion Network (AVTFN). Use a one-dimensional convolutional neural network (1D-CNN) to extract short-term dynamic features, and a long short-term memory network (LSTM) to capture global dependency features. Simultaneously, a convolutional neural network (CNN) and a temporal convolutional network (TCN) are used to extract the spatial and temporal series features of the eye movement data, respectively. Through the steps of normalization, concatenation, attention fusion, and global aggregation, a comprehensive feature set for evaluation is formed.

[0081] S1: Construct a pilot control ability assessment architecture, including a feature encoding layer, an attention mechanism module, and a task branching module. By designing a loss function based on comprehensive scoring error and sub-score consistency constraints, and using the AdamW optimization algorithm for training, an offline pilot control ability assessment model is obtained.

[0082] S2: Utilize a sliding window mechanism to segment and preprocess the real-time flight and eye movement data streams. Design a lightweight model and employ knowledge distillation to transfer the capabilities of the offline pilot control ability assessment model to an online assessment model. Combined with online training, the model is further optimized, ultimately enabling real-time scoring and dynamic monitoring of pilot control ability.

[0083] It should be noted that by adjusting the pilot task data type or operation scenario parameters used in training, the pilot control ability evaluation method based on multimodality and online learning driven proposed in the present invention can be applied to the operation ability evaluation and real-time optimization of various flight mission scenarios.

[0084] In this embodiment, in S0, a flight visual temporal fusion network (AVTFN) is constructed for the pilot's eye movement data and the flight data simulator data to extract high-quality multimodal features, and perform standardization, splicing, attention fusion, and global aggregation. The multimodal feature fusion includes:

[0085] S0.1 Combine the flight quality monitoring standards and the flight crew manual to determine the flight data parameter set related to pilot control S = S control ∪S state ∪S anomaly and the eye movement parameter set E = E fixation ∪E saccade ∪E pupil .

[0086] S0.2 Constructing a Flight Vision Temporal Fusion Network (AVTFN) For flight data, a temporal feature extraction module based on a one-dimensional convolutional neural network (1D-CNN) and a long short-term memory network (LSTM) is established. 1D-CNN is used to extract short-term dynamic features F at any moment. 1D-CNN , and use LSTM to capture the global dependency of time series and generate the global feature set F LSTM For eye movement data, a spatial-temporal feature extraction module based on convolutional neural network (CNN) and temporal convolutional network (TCN) is constructed. The spatial distribution feature F of the gaze point is extracted by CNN. CNN , using TCN to capture the time series dynamic characteristics of the gaze trajectory, and finally forming the global feature F of the eye movement data TCN .

[0087] S0.3 uses standardization, feature concatenation, attention mechanism fusion, and global aggregation operations to perform multimodal fusion of flight data and eye movement data. Dynamic weight adjustment is used to achieve weighted fusion of features, and ultimately generate a global comprehensive feature set for maneuverability assessment.

[0088] Specific implementation process:

[0089] S0.1: Determine the input parameter set with reference to the flight quality monitoring standards and the flight crew operation manual. Specifically, it includes the following parts:

[0090] The aircraft operating parameter set includes the operating parameters that reflect the pilot's control of the aircraft: S control ={s thrust ,saileron ,s rudder ,s elevator ,s stick ,s pedal} Among them, s thrust Indicates thrust lever operation, s aileron Indicates aileron operation, s rudder Indicates rudder operation, s elevator Indicates elevator operation, s stick Indicates side stick operation, s pedal Indicates pedal operation.

[0091] The flight status parameter set contains parameters that reflect the real-time status of the aircraft: S state ={s altitude ,s airspecd ,s pitch ,s roll ,s yaw ,s track ,s vert_speed ,s mach} Among them, s altitude Indicates the standard atmospheric altitude, s airspecd Indicates airspeed, s pitch represents the pitch angle, s roll represents the roll angle, s yaw represents the yaw angle, s track represents the track angle, s vert_speed Indicates vertical speed, s mach Represents the Mach number.

[0092] The flight abnormality parameter set contains parameters that reflect abnormal status: S anomaly ={s overspeed ,s stall ,s vibration ,s wind} Among them, s overspeed Indicates speeding, s stall Indicates stall, s vibration Indicates vibration, s wind Indicates wind shear.

[0093] The gaze behavior parameter set contains parameters that reflect the pilot's gaze behavior: fixation ={e fix_point ,e fix_duration ,e heatmap} Among them, e fix_point represents the gaze point position, e fix_duration represents the fixation duration, e heatmap represents the gaze heatmap.

[0094] The scanning behavior parameter set contains parameters that reflect the pilot's scanning behavior: saccade ={esac_path ,e sac_speed ,e sac_angle} Among them, e sac_path represents the scanning path, e sac_speed represents the scanning speed, e sac_angle Indicates the scanning angle.

[0095] The pupil characteristic parameter set includes pupil characteristic parameters that reflect the pilot's psychological load: E pupil ={e pupil_size ,e pupil_response} Among them, e pupil_size represents the pupil diameter, e pupil_response Indicates pupillary reaction to light.

[0096] Combining the above parameter sets, the flight data parameter set is defined as: S = S control ∪S state ∪S anomaly The eye movement data parameter set is defined as: E = E fixation ∪E saccade ∪E pupil .

[0097] The system input parameter set is the flight data simulator data parameter set S = {s1, s2, ..., s m}、The pilot's eye movement data parameter set E={e1,e2,...,e n}, with these two parameter sets as system input parameters.

[0098] S0.2: Construct a flight vision temporal fusion network (AVTFN).

[0099] First, feature extraction is performed on the flight data parameters and eye movement parameters. The specific process is as follows:

[0100] Filter parameters with time series values ​​to form a parameter set that can capture dynamic change characteristics in a local time window

[0101] S 1D-CNN ={s thrust ,s aileron ,s rudder ,s elevator ,s stick ,s pedal ,s vibration ,s overspeed ,s stall}

[0102] Use 1D-CNN to analyze S 1D-CNNPerform local dynamic feature extraction. First, each parameter sequence is divided into subsequences with a fixed-length sliding window and input into the 1D convolution layer. The sliding operation of the convolution kernel captures the feature changes within the local time window, and the pooling layer compresses the feature dimension to retain the dynamic information. The output local dynamic feature F 1D-CNN ={f thrust ,f aileron ,f rudder ,f elevator ,f stick ,f pedal ,f vibration ,f overspeed ,f stall} is used to represent the characteristic changes of the parameter within the time window. thrust ,f aileron etc.: represent the local dynamic features extracted by each parameter respectively.

[0103] A parameter set S that has long-term dependencies and needs to capture the global timing relationship over a long time span LSTM ={s altitude ,s airspeed ,s pitch ,s roll ,s yaw ,s track ,s verspeed ,s mach ,s wind}, S LSTM The input is fed into the LSTM network, which uses its built-in memory unit to capture the long-term dependencies of the parameters. LSTM selectively remembers or forgets information on time steps through a gating mechanism, extracting the global time-dependent features of the input sequence. The global features of each parameter are extracted and aggregated to form a unified global representation. LSTM ={f altitude ,f airspeed ,f pitch ,f roll ,f yaw ,f track ,f vert_speed ,f mach ,f wind}where f altitude ,f airspeed etc. respectively represent the global temporal dependency features extracted by each parameter.

[0104] Parameter set E with spatial location and distribution CNN ={e fix_point ,e heatmap ,e sac_path}Use CNN to extract spatial feature parameters. CNNInput CNN network, extract local spatial features through convolution layer, and capture the spatial distribution pattern of parameters. Use pooling layer to reduce the dimension of features and extract global spatial distribution features F CNN ={f fix_point ,f heatmap ,f sac_path}. Among them, f fix_point ,f heatmap ,f sac_path Represents the spatial characteristics of each parameter

[0105] Select parameters whose dynamic changes have time series characteristics and need to capture the change pattern through a time sliding window

[0106] E TCN ={e fix_duration ,e sac_speed ,e sac_angle ,e pupil_size ,e pupil_response}Use TCN to process the time dynamic feature parameters. TCN Input TCN, capture the sequence features in a long time range through dilated convolution. Output reflects the time feature F of dynamic changes TCN ={f fix_duration ,f sac_speed ,f sac_angle ,f pupil_size ,f pupil_response}. Among them, f fix_duration ,f sac_speed etc. represent the temporal dynamic features extracted by each parameter.

[0107] S0.3: Multimodal feature fusion is performed on flight data features and eye movement features using the steps of splicing, attention fusion, and global aggregation. The specific process is as follows:

[0108] In order to evaluate the pilot's ability to control the flight status and operation stably, the distribution of attention to key instrument areas, and the control performance under abnormal conditions, the extracted parameters are spliced ​​in the feature dimension to construct the operational stability Attention allocation and abnormal behavior A set of feature parameters in three dimensions.

[0109] For the three concatenated feature sets The attention mechanism is applied to dynamically assign the importance weight of each feature to generate a fused feature set.

[0110]

[0111] Among them: Softmax is the activation function used to convert the MLP output into a probability distribution, MLP represents a multi-layer perceptron, w stab (t) is the weight of the i-th operational stability feature at time t, w att (t) is the weight of the i-th attention allocation feature at time t, w anom (t) is the weight of the i-th abnormal behavior feature at time t.

[0112]

[0113] in: represents the feature set after the fusion of the operational stability dimension at time t; f stab,i : the i-th feature in the operational stability dimension; represents the feature set after the fusion of attention allocation dimensions at time t; f att,i : the i-th feature in the attention allocation dimension; represents the feature set after the abnormal behavior dimension is fused at time t; f anom,i : The i-th feature in the abnormal behavior dimension

[0114] Apply dynamic weights to the standardized features and perform weighted operations on the features of each modality to obtain fused features: The importance of different modal features in specific tasks is fully considered to highlight key modal information, while retaining the correlation between different modalities to generate global fusion features for subsequent evaluation.

[0115] in: represents the weighted fusion feature sequence of the operational stability dimension during the entire flight mission cycle, represents the weighted fusion feature sequence of the attention allocation dimension during the entire flight mission cycle, Represents the weighted fusion feature sequence of abnormal behavior dimensions during the entire flight mission cycle, at each time step t i Corresponding to a weighted fusion feature.

[0116] Perform a global pooling operation on all fused features at time t=1,2,...,T to generate global features.

[0117]

[0118] in: represents the global feature set of operational stability, represents the global feature set of attention allocation, represents the global feature set of abnormal behavior, T is the total time step, etc. represent the global features extracted by each parameter.

[0119] In this embodiment, S1 includes establishing a pilot control evaluation framework, establishing a model optimization objective including a comprehensive scoring error loss and a sub-scoring error loss, and obtaining a pilot control evaluation model based on a deep learning method of an AdamW optimizer, including:

[0120] S1.1 builds a pilot control evaluation architecture, establishes a feature encoding layer with residual connections, and gradually compresses and refines multimodal features through a multi-layer fully connected network; introduces a multi-head attention mechanism to capture the interactive relationship between different modal features; combines the task branch module to score operational stability, attention allocation, and abnormal behavior respectively, and finally weights the pilot to generate a comprehensive score S total .

[0121] S1.2 Design a loss function L that includes comprehensive rating error loss and sub-rating consistency constraints total Combined with the flight manual's definitions of operational stability and reasonable gaze trajectory distribution, we incorporate constraints on operational specifications and behavioral consistency. By minimizing the loss function, we ensure the accuracy and consistency of the model's scoring results.

[0122] S1.3 Divide the dataset into training set D train , validation set D val and the test set D test , the data is labeled into high score, normal score and low score categories according to the flight expert scores, and the model training is completed using the AdamW optimizer, and the final output is the comprehensive score of the pilot's control ability and its sub-scores S sub ={S stab ,S att ,S anom}.

[0123] Specific implementation process:

[0124] S1.1: Establish a feature encoding layer of a multi-layer fully connected network with residual connections, add an attention mechanism module and a task branch module to complete the scoring of different tasks (operational stability, attention allocation, abnormal behavior), and finally generate a weighted comprehensive score to form a pilot control evaluation architecture.

[0125] The global feature set of operational stability output by the flight vision temporal fusion network Attention allocation global feature set Global feature set of abnormal behavior As the input of the feature coding layer, the first layer of encoding passes through the fully connected network and ReLU activation function to obtain the result after nonlinear transformation The second layer of code is further compressed and refined through the fully connected network. The attention mechanism module then dynamically adjusts the feature weights Followed by the task branch module output set S sub ={S stab ,S att ,S anom}, and finally pass through the aggregation layer to output the comprehensive score S total This completes the construction of the pilot control evaluation framework. The model that has not been trained with data is called the pilot control evaluation framework, and the model after training is called the offline pilot control evaluation model.

[0126] in: is the global feature set of operational stability The nonlinear feature set after encoding by the first layer of fully connected network, is the global feature set of attention allocation The nonlinear feature set after encoding by the first layer of fully connected network, is a global feature set of abnormal behavior The nonlinear feature set after encoding by the first layer of fully connected network. yes A deeper feature set after encoding by the second layer of fully connected network, yes A deeper feature set after encoding by the second layer of fully connected network, yes A deeper feature set after encoding by the second layer of fully connected network. yes The feature set after dynamically adjusting the weights through the attention mechanism, yes The feature set after dynamically adjusting the weights through the attention mechanism, yes The feature set after dynamic adjustment of weights through the attention mechanism. stab is the operational stability score; S att is the attention allocation score; S anom is the abnormal behavior score; S total It is a comprehensive rating.

[0127] S1.2: Establish a model optimization objective that includes comprehensive score error loss and sub-score error loss. At the same time, combined with the operating specifications defined in the flight manual, design a constraint mechanism for the reasonable range of comprehensive scores, consistency between sub-scores and behavioral patterns, etc.

[0128] M is used to represent the pilot control evaluation model, θ is used to represent its model parameters, and the quantitative score value output by the network is recorded as The input data is any fusion feature sequence The actual rating is S total,trueGiven a data set participating in network training, the model is trained by minimizing the following loss function. The loss function is as follows:

[0129]

[0130] The first item: measures the mean square error between the predicted value and the true value of the comprehensive score; The comprehensive score of the i-th sample predicted by the model. The true comprehensive score of the i-th sample (expert score)

[0131] The second item: constrains the weighted consistency of the overall score and sub-scores; The j-th subtask score (operation stability, attention allocation, abnormal behavior) of the i-th sample predicted by the model. j : The weight of the subtask score, which indicates the relative contribution of different subtasks to the overall score.

[0132] γ: The weight of the weighted consistency constraint.

[0133]

[0134] Where: L stab : Combining the error loss of operational stability and the control input smoothness constraint, Δu i is the control input change in the time step, Δu max The reasonable range specified in the flight manual; L att :entropy(P gace,i ) combines the error loss of attention allocation and the entropy constraint of gaze distribution to form the gaze distribution entropy, H optimal The reasonable entropy range defined for the flight manual; L anom :Combines the error loss of abnormal behavior and the KL divergence constraint of behavior pattern, D KL (P action,i ||P ref ) represents the deviation of the behavior distribution from the reference behavior distribution.

[0135] If the detected comprehensive score is too high (exceeding the specified maximum score) or too low (below the minimum reasonable score threshold), a larger loss value is assigned. Mean squared error (MSE) and constraint satisfaction rate are used as model performance indicators. Model training is considered complete when the MSE reaches the preset range and the constraint satisfaction rate is above 95%.

[0136] S1.3: The fused multimodal features are trained and optimized using an AdamW optimizer-based method. Expert ratings are used to generate comprehensive ratings and sub-rating labels for the dataset. The model parameters are dynamically updated using the AdamW optimizer. The pilot control evaluation model and its optimal parameters θ are finally obtained by combining the comprehensive rating loss function and the consistency constraint optimization model. * .

[0137] First, divide the dataset into D train , D val , D test , used for training, validation and testing respectively. Each dataset is scored based on the comprehensive score S of flight experts. total,true Divide the data into high-score data, normal data, and low-score data.

[0138] Data annotation is performed based on the percentile range of the flight expert's comprehensive score. The specific rules are as follows: S total,true When it is in the top 30% of the score, it is marked as high-scoring data; when S total,true When the score is between 30% and 70%, it is marked as normal data; when S total,true When the score is in the bottom 30%, it is marked as low-scoring data.

[0139] Under this rule, D train The number of samples of medium and high score data is N1, the number of samples of normal data is N2, and the number of samples of low score data is N3. val and D test The number of samples in each category are M1, M2, and M3 respectively.

[0140] During the model training process, from the training set D train Batch data B is randomly selected for gradient update. B contains high-scoring data, normal data, and low-scoring data. The number of samples in each category is B1, B2, and B3 respectively.

[0141] During the forward propagation process, the model input is multimodal fusion features The output is the comprehensive score prediction value and sub-score prediction value Through the comprehensive scoring loss function L total Calculate the total loss value of the current batch and use the AdamW optimizer to update the model parameters. The update rule is:

[0142]

[0143] Among them, λ is the learning rate and η is the weight decay coefficient.

[0144] After each round of training, on the validation set D valEvaluate the model performance on the validation set, calculate the mean square error (MSE) and consistency error, and dynamically adjust the learning rate. stop When there is no significant decrease in the round, stop training and output the optimized model parameters θ * Finally, on the test set D test The model performance is evaluated on the dataset and the test error is recorded to measure the generalization ability of the model.

[0145] In this embodiment, in S2, firstly, the online data is divided using a sliding window technique, then a lightweight model is designed based on the pilot manipulation evaluation model to obtain a pilot manipulation online evaluation model structure, and finally, the pilot manipulation online evaluation model is further optimized using the real-time data collected online, including:

[0146] S2.1 uses a sliding window mechanism to slice and process the real-time generated flight data and eye movement data. The data stream is divided into time segments by using a fixed-length window. The pre-trained offline pilot control ability evaluation model is used to score the time segments and generate annotations according to high scores, normal scores, and low scores.

[0147] S2.2 Design a lightweight model structure and transfer the capabilities of the offline model to the online model through the knowledge distillation method. hard Ensure that the online model can correctly predict the comprehensive score category, while using the soft label loss L soft Capture changes in scoring details and improve the scoring accuracy of online models.

[0148] S2.3 combines online data with incremental training optimization, dynamically updates online evaluation model parameters, and outputs a real-time pilot control ability score. When the score falls below a set threshold, an abnormality alarm is triggered or an abnormal event is recorded, completing the online monitoring task.

[0149] Specific implementation process:

[0150] S2.1: Re-divide the real-time multi-parameter flight data using sliding window technology, complete the annotation based on the pilot control evaluation model, and generate a dataset suitable for model training.

[0151] The real-time flight data and eye movement data are divided into continuous time segments using a sliding window of fixed length. The length of each segment is T w , the sliding step is S w , the fragment is represented as: Use the pre-trained pilot maneuver evaluation model to predict the overall score and sub-scores for each time segment: where f evalGenerate annotations for each clip based on the scoring results of the pre-trained pilot maneuver evaluation model Mark as high score, Mark as normal, Marked as low score.

[0152] Finally, the labeled data segments are divided into training set and validation set in a ratio of 8:2 to ensure that each category (high score, normal score, low score) is evenly distributed in the training set and validation set.

[0153] S2.2: Based on the pilot control evaluation model, a lightweight model is designed to obtain the online evaluation model architecture:

[0154] Feature encoding layer: simplifies the multi-layer perceptron (MLP) or convolution layer; attention mechanism module: reduces the number of attention heads or weight parameters; task branch module: reduces the complexity of the sub-scoring branch through model pruning.

[0155] Using the complete pilot maneuver evaluation model as the teacher model, the pilot maneuver online evaluation model is trained through the knowledge distillation method:

[0156] L distill =α·L hard +β·L soft (7)

[0157] Among them, L distill To train the loss function of the student model by distillation method in the pilot control ability evaluation task, L hard is the hard label loss, L soft is the soft label loss, α is the weight coefficient of the hard label loss, and β is the weight coefficient of the soft label loss.

[0158] Hard label loss: The error between the student model’s predictions and the teacher model’s score labels:

[0159]

[0160] in, is the predicted output of the teacher model, is the predicted output of the student model, and CrossEntropy is the cross entropy loss function used to measure the difference between the outputs of the teacher and student models.

[0161] Soft label loss: Minimize the difference between the teacher model and the student model through the temperature-corrected softmax output:

[0162]

[0163] Among them, z teacher is the original prediction score of the teacher model under the given input, zstudent is the original prediction score of the student model under the given input, T is the temperature parameter, Softmax is the softmax function, which converts the output value into a probability form, and the sum of the probabilities of all categories is 1. KL is the KL divergence, which is a measure of the difference between two probability distributions.

[0164] The pre-training of the pilot maneuver online evaluation model is completed on the training set, and the validation set is used to evaluate the model performance, and finally the initial parameters θ are obtained. init

[0165] S2.3: Use the real-time data collected online to further optimize the pilot control online evaluation model, and use the real-time data segment X online Generate comprehensive score labels and use small batch incremental training to update model parameters:

[0166]

[0167] Among them, θ t+1 is the parameter of the model at time t+1, θ t is the parameter of the model at time t, η is the learning rate, is the gradient of the loss function with respect to the model parameters, X online Real-time data input during online learning.

[0168] During the online training process, the pilot manipulates the online evaluation model to score the data in each sliding window in real time. According to the scoring results When the comprehensive score is lower than the threshold, the current segment is marked as abnormal and real-time feedback is provided on the pilot's control ability.

[0169] Example 2

[0170] The present invention also provides a pilot control ability evaluation system based on multimodal data and online learning, the system is used to implement any one of the methods described above, the system comprising: a data set construction module, an offline evaluation model construction module, and an online evaluation model construction module;

[0171] The dataset construction module is used to determine the flight data parameter set and eye movement data parameter set related to pilot control, establish a flight visual temporal fusion network, extract the features of the flight data parameter set and the eye movement data parameter set, and perform multimodal fusion of the flight data and eye movement data features through attention mechanism and global aggregation to generate a global comprehensive feature set for control ability assessment;

[0172] The offline evaluation model construction module is used to build a pilot control ability evaluation framework based on the global comprehensive feature set. By designing a loss function based on the comprehensive score error and sub-score consistency constraints, and using the AdamW optimization algorithm to complete training, it generates an offline pilot control ability evaluation model and outputs the pilot control ability comprehensive score and corresponding sub-scores.

[0173] The online evaluation model construction module is used to segment and preprocess the real-time generated flight data and eye movement data streams according to the comprehensive score of the pilot's control ability and the corresponding sub-scores using a sliding window mechanism. A lightweight model based on pruning and parameter quantization technology is designed to meet the real-time scoring requirements. The feature expression capabilities in the offline pilot control ability evaluation model are transferred to the lightweight online evaluation model, and the pilot's control ability score is output in real time.

[0174] The embodiments described above are merely descriptions of preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by persons skilled in the art should fall within the scope of protection defined by the claims of the present invention.

Claims

1. A pilot control ability assessment method based on multimodal data and online learning is characterized by: The method comprises: S0: Determine the flight data parameter set and eye movement data parameter set related to the pilot's control, establish a flight visual temporal fusion network, extract the features of the flight data parameter set and the eye movement data parameter set, and perform multimodal fusion of the features of the flight data and eye movement data through attention mechanism and global aggregation to generate a global comprehensive feature set for control ability assessment; S1: Based on the global comprehensive feature set, a pilot control ability evaluation framework is constructed. By designing a loss function based on the comprehensive score error and sub-score consistency constraints, the AdamW optimization algorithm is used to complete the training, generate an offline pilot control ability evaluation model, and output the pilot control ability comprehensive score and corresponding sub-scores; S2: Based on the pilot's comprehensive control ability score and corresponding sub-scores, a sliding window mechanism is used to segment and preprocess the real-time flight data and eye movement data streams. A lightweight model based on pruning and parameter quantization techniques is designed to meet the real-time scoring requirements. The feature expression capabilities of the offline pilot control ability evaluation model are transferred to a lightweight online evaluation model, and the pilot's control ability score is output in real time. Said S1 includes designing a loss function including comprehensive score error loss and sub-score consistency constraint , combined with the definitions of operational stability and reasonable distribution of gaze tracks in the flight manual, and adding constraints on operational specifications and behavioral consistency, we ensure the accuracy and consistency of the model scoring results by minimizing the loss function. Specifically: use represents the pilot control evaluation model, Represents the corresponding model parameters, and the quantitative score value output by the network is recorded as , the input data is any fusion feature sequence , the true rating value is , given the data set participating in network training, the model is trained by minimizing the loss function, and the loss function is in the form of: In the formula, the first term is the mean square error between the predicted value and the true value of the comprehensive score; The model predicts the The comprehensive score of the samples, Indicates the The true comprehensive score of each sample is the expert score; the second item is the weighted consistency between the constrained comprehensive score and the sub-scores; The model predicts the The first sample The sub-task scores are operation stability, attention allocation, and abnormal behavior; The weight of the subtask score indicates the relative contribution of different subtasks to the overall score; : The weight of the weighted consistency constraint; in, represents the error loss that combines operational stability and control input smoothness constraints, is the control input variation for the time step, Reasonable range for flight manual provisions; express The error loss of attention allocation and the entropy constraint of gaze distribution are combined into gaze distribution entropy. A reasonable entropy range defined for the flight manual; Denotes the error loss that combines abnormal behavior and the KL divergence constraint of behavior pattern, D KL (P action,i ||P ref ) represents the deviation of the behavior distribution from the reference behavior distribution; When the detected comprehensive score is too high, that is, exceeds the specified maximum score, or too low, that is, is less than the minimum reasonable score threshold, a loss value that meets the preset requirements is assigned, and the mean square error (MSE) and constraint satisfaction rate are used as model performance indicators; when the mean square error reaches the preset range and the constraint satisfaction rate is higher than 95%, the model training is considered complete.

2. The method according to claim 1, characterized in that In the S0, determining the flight data parameter set and the eye movement data parameter set related to the pilot's manipulation includes: defining the flight data parameter set as: , the eye movement data parameter set is defined as: ; The aircraft operating parameter set includes the operating parameters that reflect the pilot's control of the aircraft: in, Indicates thrust lever operation, Indicates aileron operation, Indicates rudder operation, Indicates elevator operation, Indicates side stick operation, Indicates pedal operation; The flight status parameter set contains parameters that reflect the real-time status of the aircraft: in, represents the standard atmospheric altitude, Indicates airspeed, represents the pitch angle, represents the roll angle, represents the yaw angle, represents the track angle, represents the vertical speed, represents the Mach number; The flight anomaly parameter set contains parameters that reflect the anomaly status: in, Indicates speeding, Indicates stall, Indicates vibration, Indicates wind shear; The gaze behavior parameter set contains parameters that reflect the pilot's gaze behavior: in, represents the gaze point position, represents the duration of fixation, represents the gaze heat map; The Glance Behavior Parameters collection contains parameters that reflect the pilot's glancing behavior: in, represents the scanning path, represents the scanning speed, Indicates the scanning angle; The pupil characteristic parameter set includes pupil characteristic parameters that reflect the pilot's psychological load: in, represents the pupil diameter, Indicates pupillary reaction to light.

3. The method according to claim 2, characterized in that In the above S0, establishing a flight visual temporal fusion network and extracting features of the flight data parameter set and the eye movement data parameter set include: For flight data, a time series feature extraction module based on one-dimensional convolutional neural network (1D-CNN) and long short-term memory network (LSTM) is established to extract short-term dynamic features at any moment through 1D-CNN. , and use LSTM to capture the global dependency of time series and generate a global feature set ; For eye movement data, a space-time feature extraction module based on convolutional neural network (CNN) and temporal convolutional network (TCN) is constructed to extract the spatial distribution features of the gaze point through CNN. , using TCN to capture the time series dynamic characteristics of the gaze trajectory, and finally form the global features of the eye movement data ; in, ,in, , , , , , , , Respectively represent the local dynamic features extracted by each parameter; ,in, , , , , , , , Respectively represent the global temporal dependency features extracted by each parameter; ,in, Represent the spatial characteristics of each parameter respectively; ,in, , , , Represents the temporal dynamic characteristics extracted by each parameter.

4. The method according to claim 3, characterized in that In S0, the features of flight data and eye movement data are multimodally fused through attention mechanism and global aggregation to generate a global comprehensive feature set for maneuverability assessment, including: in: represents the global feature set of operational stability, represents the global feature set of attention allocation, Represents the global feature set of abnormal behavior, is the total time step, Indicates The feature set after the fusion of the moment operation stability dimension, Indicates The feature set after the fusion of the momentary attention allocation dimension, Indicates The feature set after the fusion of abnormal behavior dimensions at each moment.

5. The method according to claim 1, wherein In S1, a pilot control ability evaluation framework is constructed based on the global comprehensive feature set. By designing a loss function based on the comprehensive score error and sub-score consistency constraints, the AdamW optimization algorithm is used to complete the training and generate an offline pilot control ability evaluation model. The output of the pilot control ability comprehensive score and the corresponding sub-scores include: S1.1: Construct a pilot control evaluation framework, specifically including: establishing a feature encoding layer with residual connections, gradually compressing and refining multimodal features through a multi-layer fully connected network; introducing a multi-head attention mechanism to capture the interactive relationship between different modal features; combining the task branch module to score operational stability, attention allocation, and abnormal behavior respectively, and finally weighting and generating a comprehensive pilot score ; S1.2: Design a loss function that includes comprehensive rating error loss and sub-rating consistency constraints , combined with the definitions of operational stability and reasonable distribution of gaze tracks in the flight manual, and adding constraints on operational specifications and behavioral consistency, the accuracy and consistency of the model scoring results are ensured by minimizing the loss function; S1.3: Divide the global comprehensive feature dataset into training sets , validation set and test set , the data is labeled into high-score, normal, and low-score categories based on the flight expert scores, and the AdamW optimizer is used to complete the training to generate an offline pilot control ability evaluation model, and finally output the pilot control ability comprehensive score and corresponding sub-scores .

6. The method according to claim 5, characterized in that In S1.3, the global comprehensive feature dataset is divided into a training set , validation set and test set , the data is labeled into high-score, normal, and low-score categories based on the flight expert scores, and the AdamW optimizer is used to complete the training to generate an offline pilot control ability evaluation model, and finally output the pilot control ability comprehensive score and corresponding sub-scores include: First, divide the dataset into , , , respectively for training, validation and testing, each dataset is scored based on the comprehensive score of flight experts Divide the data into high-score data, normal data and low-score data; Data annotation is performed based on the percentile range of the flight experts' comprehensive scores. The specific rules are as follows: When it is in the top 30% of the score, it is marked as high-scoring data; when When the score is between 30% and 70%, it is marked as normal data; when When it is in the bottom 30% of the score, it is marked as low-scoring data; Under the said rules, The number of samples of medium and high score data is , the number of samples of normal data is , the number of samples of low-scoring data is , and The number of samples in each category is , , ; During model training, the training set Randomly extract batch data For gradient update, Contains high-score data, normal data and low-score data, and the number of samples in each category is , , ; During the forward propagation process, the model input is multimodal fusion features , the output is the comprehensive score prediction value and sub-score prediction value , through the comprehensive scoring loss function Calculate the total loss value of the current batch and use the AdamW optimizer to update the model parameters. The update rule is: in, is the learning rate, is the weight attenuation coefficient; After each round of training, the validation set Evaluate the model performance, calculate the mean square error (MSE) and consistency error, and dynamically adjust the learning rate; when the MSE of the validation set is continuously When there is no significant decrease in the round, stop training and output the optimized model parameters , finally, in the test set Evaluate the model performance and record the test error.

7. The method according to claim 1, characterized in that In S2, using a sliding window mechanism to segment and preprocess the real-time generated flight data and eye movement data streams includes: The real-time generated flight data and eye movement data are divided into continuous time segments using a fixed-length sliding window. The length of each segment is , the sliding step length is , the fragment is represented as: ,Use the pre-trained offline pilot control ability evaluation model to predict the comprehensive score and sub-score for each time segment: ,in Generate annotations for each clip based on the scoring results of the pre-trained pilot maneuver evaluation model Mark as high score, Mark as normal, Marked as low score; Finally, the labeled data segments are divided into training set and validation set in a ratio of 8:2 to ensure that each category, i.e. high score, normal score, and low score, is evenly distributed in the training set and validation set.

8. The method according to claim 7, characterized in that In S2, a lightweight model based on pruning and parameter quantization technology is designed to transfer the feature expression capability of the offline pilot control ability evaluation model to the lightweight online evaluation model, including: Using the complete pilot maneuver evaluation model as the teacher model, the pilot maneuver online evaluation model is trained through the knowledge distillation method: in, To train the loss function of the student model by distillation method in the pilot control ability evaluation task, is the hard label loss, is the soft label loss, is the weight coefficient of hard label loss, is the weight coefficient of soft label loss; The error between the student model’s predictions and the teacher model’s scoring labels is: in, is the predicted output of the teacher model, is the predicted output of the student model, is the cross entropy loss function, which is used to measure the difference between the outputs of the teacher and student models; Minimize the difference between the teacher model and the student model by the temperature-corrected softmax output: in, is the original prediction score of the teacher model under the given input, is the raw prediction score of the student model under the given input, is the temperature parameter, The softmax function converts the output value into a probability form, and the sum of the probabilities of all categories is 1. is the KL divergence, which is a measure of the difference between two probability distributions; The pre-training of the pilot maneuver online evaluation model is completed on the training set, and the validation set is used to evaluate the model performance and finally obtain the initial parameters. .

9. A pilot control ability evaluation system based on multimodal data and online learning, wherein the system is used to implement the method according to any one of claims 1 to 8, characterized in that: The system includes: a data set construction module, an offline evaluation model construction module and an online evaluation model construction module; The dataset construction module is used to determine a set of flight data parameters and a set of eye movement data parameters related to pilot control, establish a flight visual temporal fusion network, extract features of the flight data parameter set and the eye movement data parameter set, and perform multimodal fusion of the features of the flight data and eye movement data through an attention mechanism and global aggregation to generate a global comprehensive feature set for control ability assessment; The offline evaluation model construction module is used to construct a pilot control ability evaluation framework based on the global comprehensive feature set, design a loss function based on the comprehensive score error and sub-score consistency constraints, complete training using the AdamW optimization algorithm, generate an offline pilot control ability evaluation model, and output the pilot control ability comprehensive score and corresponding sub-scores; The online evaluation model construction module is used to segment and preprocess the real-time generated flight data and eye movement data streams based on the pilot's comprehensive control ability score and the corresponding sub-scores using a sliding window mechanism. A lightweight model based on pruning and parameter quantization technology is designed to meet the real-time scoring requirements. The feature expression capabilities of the offline pilot control ability evaluation model are transferred to the lightweight online evaluation model, and the pilot's control ability score is output in real time.

Images