Pilot dynamic assessment method, system, device and storage medium based on TEM model

Abstract

The application relates to the technical field of TEM model, and provides a pilot dynamic evaluation method, system and device based on a TEM model and a storage medium, which solves the problems of low precision of flight training evaluation and poor adaptability of training schemes. The method comprises the following steps: obtaining flight control, physiological monitoring and cockpit voice data, forming a multi-modal data stream through time synchronization; using a TEM model to perform threat perception, error management and non-technical skill three-level evaluation, generate corresponding indexes and fuse them into a comprehensive feature vector; using a double-channel long short-term memory network to process time sequence and cognitive features respectively, and extracting deep features; using SHAP values for decision tracing to locate the ability defect nodes; and finally matching the defect types through a personalized improvement scheme recommendation mechanism, and generating adaptive training content. The application improves the precision of flight training evaluation and the adaptability of training schemes.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of TEM models, in particular to a pilot dynamic evaluation method, system and device based on a TEM model and a storage medium. BACKGROUND

[0002] In the process of flight training and examination, comprehensive and objective evaluation of the operation performance and cognitive state of pilots is the key to improving flight safety and training efficiency. The current method of relying on single flight parameter alarm or post-playback analysis has been difficult to meet the technical needs of real-time, multi-dimensional quantitative evaluation of pilots' situational awareness, decision-making quality, and load state.

[0003] There is currently an automatic evaluation scheme based on multi-source data fusion. This scheme collects flight control sequences, basic physiological indicators and cabin sound records, uses machine learning methods to build evaluation models, and realizes monitoring and scoring of pilot operation compliance and part of physiological state, which to some extent reduces the subjectivity of human evaluation. The current scheme still lacks association with training scenarios, such as identification and association of threats, errors, expected flight states and unexpected flight states in training scenarios.

[0004] However, such existing schemes have not achieved accurate synchronization and deep fusion between different modal signals at the data processing level, the evaluation dimension is still mainly operation, the cognitive state judgment under high load situation is insufficient, the explainability of the evaluation result is limited, leading to feedback lag, which is difficult to support real-time personalized training guidance.

[0005] Among them, the threat refers to an event or condition that exists in the operating environment, is beyond the direct control of the crew, but can increase the complexity of operation and reduce the safety margin. Threats can be divided into expected threats and unexpected threats. Expected threats are foreseeable, such as severe weather (thunderstorms, wind shear), airport congestion or complex tasks. The crew can brief in advance and develop coping strategies, such as remaining vigilant to other aircraft at congested airports. Unexpected threats, on the other hand, occur suddenly and without warning, such as sudden on-board equipment failure or unexpected turbulence.

[0006] Further, the TEM model also considers potential threats, which are often hidden in the organization system, such as risk culture or ambiguous policies. Errors are usually defined as the behavior or non-behavior of the flight crew that leads to a deviation from the organization or operational intent. The TEM model classifies errors according to the main interacting objects when the pilot makes an error. This includes aircraft handling errors (e.g., speed or vertical deviation), procedural errors (e.g., missing or incorrectly performing checklist items), and communication errors (e.g., incorrect callout response or misunderstanding of instructions). Notably, the consequences of errors depend on whether the flight crew can timely discover and correct the error before it evolves into an undesired aircraft state. Undesired Aircraft States (UAS): UAS is usually defined as an unintended operational condition that leads to a reduction in safety margins. When threats and / or errors are not effectively managed, they can lead to UAS. The management of UAS is considered the "last chance" to avoid serious accidents or incidents. The TEM model classifies according to the type of UAS, such as aircraft handling deviation or improper system configuration. SUMMARY

[0007] The present application provides a pilot dynamic evaluation method, system, device and storage medium based on a TEM model, to solve the problems of low precision of flight training evaluation and poor adaptability of training scheme in the prior art.

[0008] To solve the above technical problems, in a first aspect, the present application provides a pilot dynamic evaluation method based on a TEM model, comprising:

[0009] obtaining flight control data of a flight target, physiological monitoring data of a pilot, and cockpit voice data;

[0010] performing time synchronization alignment processing on the flight control data, the physiological monitoring data, and the cockpit voice data to form a multi-modal data stream;

[0011] based on the TEM model, sequentially performing threat perception evaluation, error management evaluation, and non-technical skill evaluation on the multi-modal data stream to obtain corresponding threat perception indicators, error management indicators, and non-technical skill indicators, respectively, and performing feature fusion on the threat perception indicators, the error management indicators, and the non-technical skill indicators to form a comprehensive evaluation feature vector;

[0012] inputting the comprehensive evaluation feature vector into a dual-channel long short-term memory network for deep feature extraction and fusion, wherein a spatio-temporal feature channel processes flight control time sequence features in the comprehensive evaluation feature vector, and a cognitive feature channel processes physiological response features and voice cognitive features in the comprehensive evaluation feature vector;

[0013] The SHAP value is used for decision tracing on an output result of the dual-channel long short-term memory network, and based on a decision tracing result, a capability defect node of the pilot is located;

[0014] In combination with a preset personalized improvement scheme recommendation mechanism, the capability defect node is matched with a preset core competency defect library, and corresponding adaptive training content is generated according to a matching result.

[0015] Optionally, based on the TEM model, threat perception evaluation, error management evaluation and non-technical skill evaluation are sequentially performed on the multi-modal data stream, and corresponding threat perception indicators, error management indicators and non-technical skill indicators are obtained, including:

[0016] The multi-modal data stream is input into a pre-trained TEM model, and through an evaluation framework of the TEM model, aligned flight control data, aligned physiological monitoring data and aligned cockpit voice data are extracted from the multi-modal data stream;

[0017] Through a threat management module of the TEM model, a sudden threat event and an actual flight parameter appearing in a flight environment are identified from the aligned flight control data;

[0018] According to a time when the sudden threat event occurs and a time when the pilot makes a first response to the sudden threat event, a corresponding time interval is calculated, and a threat perception indicator is generated based on the time interval;

[0019] Through an error management module of the TEM model, the actual flight parameter is compared with a preset standard flight parameter, and a parameter deviation degree is calculated according to a comparison result;

[0020] An operation sequence efficiency of the pilot for correcting the parameter deviation degree is calculated, and an error management indicator is generated based on the operation sequence efficiency;

[0021] Through a non-technical skill module of the TEM model, distribution characteristics of fixation points on flight instruments are extracted from eye movement tracking data contained in the aligned physiological monitoring data, the aligned cockpit voice data is input into a voice emotion recognition model, decision instruction frequency and response interval are identified, and a non-technical skill indicator is generated based on the decision instruction frequency, the response interval and the distribution characteristics.

[0022] Optionally, the aligned cockpit voice data is input into a voice emotion recognition model, decision instruction frequency and response interval are identified, and a non-technical skill indicator is generated based on the decision instruction frequency, the response interval and the distribution characteristics, including:

[0023] input the aligned cockpit voice data into a voice emotion recognition model, and identify a decision instruction in the aligned cockpit voice data through the voice emotion recognition model;

[0024] count a number of occurrences of the decision instruction in a preset time period to obtain a decision instruction frequency;

[0025] calculate a time difference from issuing the decision instruction to receiving an effective voice response to the decision instruction, and obtain a response interval based on the time difference;

[0026] fuse the decision instruction frequency, the response interval, and the distribution feature, and generate a non-technical skill index based on a fusion result.

[0027] Optionally, the fusing the decision instruction frequency, the response interval, and the distribution feature, and generating the non-technical skill index based on the fusion result, comprises:

[0028] assigning respective weights to the decision instruction frequency, the response interval, and the distribution feature;

[0029] performing weighted calculation on the decision instruction frequency, the response interval, and the distribution feature according to the weights to obtain a comprehensive evaluation value;

[0030] inputting the comprehensive evaluation value into a preset index mapping rule library, and converting the comprehensive evaluation value into a corresponding non-technical skill level through a mapping relationship in the index mapping rule library, the non-technical skill level being a non-technical skill index.

[0031] Optionally, the inputting the comprehensive evaluation feature vector into the dual-channel long short-term memory network for deep feature extraction and fusion comprises:

[0032] inputting flight control timing features in the comprehensive evaluation feature vector into a space-time feature channel of the dual-channel long short-term memory network, and performing deep feature extraction on the flight control timing features through the space-time feature channel to obtain timing dependency features;

[0033] inputting physiological response features and voice cognitive features in the comprehensive evaluation feature vector into a cognitive feature channel of the dual-channel long short-term memory network, and performing deep feature extraction on the physiological response features and the voice cognitive features through the cognitive feature channel to obtain context features;

[0034] performing feature splicing on the timing dependency features and the context features to form spliced features;

[0035] The concatenated features are input to a full connection layer of the dual-channel long short-term memory network for dimension reduction processing to obtain reduced dimension features, which are output results of the dual-channel long short-term memory network.

[0036] Optionally, the SHAP value is used to perform decision tracing on the output results of the dual-channel long short-term memory network, and based on the decision tracing result, a capability defect node of the pilot is located, including:

[0037] The SHAP value is used to calculate a contribution value of each input feature in the output results of the dual-channel long short-term memory network, and based on all the contribution values, a feature contribution set is generated;

[0038] A feature subset that plays a leading role in the output results is identified from the feature contribution set;

[0039] Decision tracing analysis is performed on the feature subset to obtain a decision tracing result;

[0040] The decision tracing result is mapped to a preset pilot capability dimension space, and a capability defect node that meets a preset judgment standard condition is located in the pilot capability dimension space.

[0041] Optionally, the capability defect node is matched with a preset core competency defect library in combination with a preset personalized improvement scheme recommendation mechanism, and corresponding adaptive training content is generated according to a matching result, including:

[0042] The capability defect node is similarity calculated with a defect mode in the core competency defect library by using the preset personalized improvement scheme recommendation mechanism;

[0043] Based on the similarity, a core competency defect type is determined, which is a matching result;

[0044] According to the core competency defect type, a corresponding basic training scene template is selected from a preset training scene template library;

[0045] The basic training scene template is parsed to obtain corresponding basic training scene elements;

[0046] All basic training scene elements are combined to generate adaptive training content.

[0047] In a second aspect, the application provides a pilot dynamic evaluation system based on a TEM model, including:

[0048] An acquisition module is configured to acquire flight control data of a flight target, physiological monitoring data of a pilot, and cockpit voice data;

[0049] an alignment module configured to time-synchronize and align the flight control data, the physiological monitoring data, and the cockpit voice data to form a multi-modal data stream;

[0050] an evaluation module configured to sequentially perform threat perception evaluation, error management evaluation, and non-technical skill evaluation on the multi-modal data stream based on a TEM model to obtain corresponding threat perception indicators, error management indicators, and non-technical skill indicators, respectively, and to perform feature fusion on the threat perception indicators, the error management indicators, and the non-technical skill indicators to form a comprehensive evaluation feature vector;

[0051] an input module configured to input the comprehensive evaluation feature vector into a dual-channel long short-term memory network for deep feature extraction and fusion, wherein a spatio-temporal feature channel processes flight control time sequence features in the comprehensive evaluation feature vector, and a cognitive feature channel processes physiological response features and voice cognitive features in the comprehensive evaluation feature vector;

[0052] a positioning module configured to perform decision tracing on an output result of the dual-channel long short-term memory network by using SHAP values, and to locate a capability deficiency node of the pilot based on a decision tracing result;

[0053] a matching module configured to match the capability deficiency node with a preset core competency deficiency library in combination with a preset individualized improvement scheme recommendation mechanism, and to generate corresponding adaptive training content according to a matching result.

[0054] In a third aspect, the present application provides an electronic device, comprising:

[0055] a memory configured to store a computer program;

[0056] a processor configured to implement steps of the pilot dynamic evaluation method based on a TEM model according to the first aspect when executing the computer program.

[0057] In a fourth aspect, the present application provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executable by a processor to implement steps of the pilot dynamic evaluation method based on a TEM model according to the first aspect.

[0058] In the application, a pilot dynamic evaluation method based on a TEM model is provided, which comprises the following steps: obtaining flight control data of a flight target, physiological monitoring data of a pilot and cockpit voice data; performing time synchronization alignment processing on the flight control data, the physiological monitoring data and the cockpit voice data to form a multi-modal data stream; based on the TEM model, sequentially performing threat perception evaluation, error management evaluation and non-technical skill evaluation on the multi-modal data stream to obtain corresponding threat perception indicators, error management indicators and non-technical skill indicators respectively; performing feature fusion on the threat perception indicators, the error management indicators and the non-technical skill indicators to form a comprehensive evaluation feature vector; inputting the comprehensive evaluation feature vector into a double-channel long short-term memory network for deep feature extraction and fusion, wherein a spatio-temporal feature channel processes flight control time sequence features in the comprehensive evaluation feature vector, and a cognitive feature channel processes physiological response features and voice cognitive features in the comprehensive evaluation feature vector; using SHAP values to perform decision tracing on output results of the double-channel long short-term memory network, and based on the decision tracing results, locating a capability defect node of the pilot; combining a preset individualized improvement scheme recommendation mechanism, matching the capability defect node with a preset core competency defect library, and according to a matching result, generating corresponding adaptive training content.

[0059] The technical scheme provided by the application has the following beneficial effects:

[0060] The application provides a multi-angle and all-around data basis for comprehensive evaluation of pilot performance. The time consistency problem of data from different sources is solved, providing accurate data basis for subsequent fusion analysis. The transition from single operation evaluation to multi-dimensional comprehensive evaluation covering threat perception, error management and non-technical skills is realized, improving the depth and breadth of evaluation. Deep patterns are mined from operation time sequence and cognitive state respectively, enhancing the discrimination and reliability of state representation. The explainability of the model is improved, and the key capability short board affecting the pilot performance can be accurately identified. The evaluation results are automatically converted into individualized training schemes, making the training guidance more targeted and effective.

[0061] Further, by using the threat management module, the error management module and the non-technical skill module of the pre-trained TEM model, key features are extracted from the synchronized multi-modal data, and threat perception indicators, error management indicators and non-technical skill indicators are quantitatively generated.

[0062] Moreover, this step automatically completes multi-dimensional indicator extraction and generation through a structured model, improving the objectivity and efficiency of the evaluation process, and ensuring the consistency of the calculation of indicators of different capability dimensions, laying a solid foundation for subsequent comprehensive diagnosis and individualized training recommendation.

[0063] These aspects or other aspects of the present application will be made clearer in the following description of embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0064] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiment or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0065] Figure 1 A flow chart of a pilot dynamic evaluation method based on a TEM model provided by an embodiment of the present application;

[0066] Figure 2 A specific implementation schematic diagram of a pilot dynamic evaluation method based on a TEM model provided by an embodiment of the present application;

[0067] Figure 3 A structural schematic diagram of a pilot dynamic evaluation system based on a TEM model provided by an embodiment of the present application. DETAILED DESCRIPTION

[0068] In the field of pilot capability evaluation, although the existing automatic scheme based on multi-source data fusion can realize the monitoring of basic operations and part of physiological states, it still faces limitations in actual application. When dealing with multiple types of data such as flight control, physiological monitoring and voice, this kind of scheme has not yet realized accurate cross-modal time alignment, resulting in deviation in the analysis process; the evaluation dimension is still mainly discrete operation indicators, and the cognitive state of pilots in high-load situations (such as threat perception, decision logic and attention allocation) lacks deep integration and effective quantification; in addition, the evaluation model has weak interpretability, it is difficult to accurately locate the root cause of capability defects, resulting in feedback lag, and it cannot provide effective support for real-time personalized training.

[0069] To solve the above problems, the application provides a pilot dynamic evaluation method based on a TEM model. The method first performs high-precision time synchronization on multi-source heterogeneous data to form a unified multi-modal data stream. Then, a three-level evaluation system of threat perception, error management and non-technical skills is used to generate quantitative indicators and fuse them into a comprehensive feature vector. A dual-channel network is used to process flight control sequences and cognitive behavior features to realize deep state extraction and fusion. Finally, the ability defect nodes are located through explainability analysis, and the matching adaptive training scheme is automatically generated. The method realizes the transformation from single operation evaluation to accurate cognitive ability evaluation through multi-dimensional cognitive ability quantification, multi-modal data deep fusion and defect root explainability. The method effectively solves the problems of narrow evaluation dimension, insufficient fusion depth and feedback lag in the prior art, and improves the comprehensiveness of pilot training evaluation and the timeliness of guidance.

[0070] To make the person skilled in the art better understand the application scheme, the application will be further described in detail below in combination with the drawings and specific embodiments. Obviously, the described embodiments are only part of the embodiments of the application, not all. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of the application.

[0071] The core of the application is to provide a pilot dynamic evaluation method based on a TEM model. A specific embodiment of the method is shown in the flowchart Figure 1 The method comprises the following steps.

[0072] Step 101: Obtain flight control data of a flight target, physiological monitoring data of a pilot and cockpit voice data.

[0073] In step 101, the flight control data refers to the operation record data generated by the flight target during flight, including lever quantity, rudder position, throttle parameter, etc. The physiological monitoring data refers to the physiological state data of the pilot collected by the wearable device, including heart rate, eye movement trajectory, etc. The cockpit voice data refers to the voice recording data in the cockpit, including the dialogue between the pilots and the communication content with the tower. The cockpit is the operation control center of the flight target, and the two are in a relationship of whole and core component.

[0074] In the embodiment of the application, the flight control data is obtained by a flight data recording device, the pilot's heart rate and eye movement data are collected by a wearable physiological monitoring device, and the voice data is recorded by a cockpit voice recorder. The three types of data are collected in real time and temporarily stored in a data buffer area.

[0075] For example, in a simulated flight training, the system obtains flight control data such as stick quantity and rudder position from a flight simulator, collects physiological monitoring data such as heart rate and eye movement trajectory from a smart wearable device worn by the pilot, and records all voices in the cockpit through a high-fidelity recording device to form three types of raw data.

[0076] Step 102: Time synchronization alignment processing is performed on the flight control data, the physiological monitoring data, and the cockpit voice data to form a multi-modal data stream.

[0077] In step 102, the multi-modal data stream refers to a unified data sequence formed by accurately aligning data of different sources and frequencies on the time axis.

[0078] In the embodiments of the present application, a multi-thread synchronization engine is used to mark a unified timestamp on the three types of data based on a high-precision clock, and an interpolation method is used to align data of different sampling rates to the same time granularity, and finally to converge into a multi-modal data stream with consistent timing.

[0079] For example, the flight control data has a sampling rate of 10Hz, the physiological data has a sampling rate of 60Hz, and the voice has a sampling rate of 16000Hz. The system marks a timestamp on all data samples with millisecond-level precision, interpolates and fills in low-frequency data, and down-samples and integrates high-frequency data to form a multi-modal data stream with 10Hz as the reference.

[0080] Step 103: Based on the TEM model, the multi-modal data stream is sequentially subjected to threat perception evaluation, error management evaluation, and non-technical skill evaluation to obtain corresponding threat perception indicators, error management indicators, and non-technical skill indicators, respectively. The threat perception indicators, the error management indicators, and the non-technical skill indicators are fused to form a comprehensive evaluation feature vector.

[0081] In step 103, the Threat and Error Management model (TEM) refers to a threat and error management model used for system evaluation of pilot's comprehensive response capability. The core of this system lies in its multidimensional technical architecture: first, a structured data foundation consistent with the TEM model is built, which systematically classifies threats, errors, and undesired aircraft states (UAS); second, a hybrid deep learning architecture (including convolutional neural network (CNN), long short-term memory (LSTM), and Transformer) is used to identify complex flight states in real time; third, a comprehensive evaluation of the pilot's management performance in threat situations is made by fusing multi-modal data (operation data, physiological data, and speech data); finally, the difficulty of the training scene is dynamically adjusted according to the evaluation results, forming a closed-loop personalized adaptive learning feedback mechanism. To realize the effective application of the TEM model and data-driven training, the system will build three interrelated core databases. TEM model and classification database, this database is the core knowledge base of the system, which systematically and structurally stores the threats, errors, and UAS in the TEM model. The fields in the database will include: threat type: for example, environmental threats (such as thunderstorms, wind shear), organizational threats (such as operational stress, ATC congestion), or potential threats. Error category: according to the main interactive object, it is divided into control error, procedural error or communication error. UAS type: for example, vertical or lateral deviation, unstable approach or improper system configuration. This database provides standardized classification labels for AI models, enabling them to map complex flight events into the TEM framework and providing a unified reference standard for subsequent performance evaluation. Threat-scene mapping database, this database is the core of dynamic scenario generation. It associates specific TEM threats with a series of combinable simulation scenario elements. For example, when selecting an "ATC congestion" expected threat, the database will pull relevant scenario elements such as: "ATC re-planning of route"; "non-standard terminology"; "similar call signs". Through this mapping, the system can automatically combine and generate highly realistic, dynamic and unpredictable training scenarios based on the pre-set threat. This ability surpasses traditional pre-set scripts, providing pilots with preparation for dealing with various unexpected situations, thereby improving the effectiveness of training. Pilot training and performance database, this database is used to record and store pilot performance data during the entire training period. It will record in detail the flight parameters, AI evaluation results, threat and error management, and physiological and psychological state data of each training.Key data points include: training performance records: complete logs of each simulation training, including pilot's operation input and aircraft state at each stage. TEM quantitative indicators: record pilot's scores in threat management, error management, and UAS management. Longitudinal performance trends: track changes in pilot's performance at different training stages, identify their individualized weaknesses and strengths. Threat perception indicators quantify the speed of pilot's response to sudden threats. Error management indicators reflect the efficiency of pilot's operation correction. Non-technical skill indicators evaluate decision-making and attention allocation capabilities. The comprehensive evaluation feature vector is a multi-dimensional feature representation formed by the weighted fusion of the three types of indicators.

[0082] In the embodiments of the present application, first, flight control data is extracted from the multi-modal data stream, sudden threat events are identified and response delay time is calculated, and threat perception indicators are generated. Then, actual flight parameters are extracted, compared with standard parameters to obtain deviation, and correction operation efficiency is calculated, and error management indicators are generated. At the same time, instruction frequency and response interval are extracted from voice data, and gaze distribution features are extracted from eye movement data, and non-technical skill indicators are generated. Finally, the three types of indicators are fused into a comprehensive evaluation feature vector according to the weight.

[0083] For example, the time interval of 3.2 seconds between the occurrence time of the wind shear event and the first response time of the pilot is calculated as the threat perception indicator. The deviation of the actual height from the standard height is 20 meters, and the correction operation efficiency is 0.8 as the error management indicator. The non-technical skill indicator is 0.75, which is weighted and fused from the instruction frequency of 5 per minute, the average response interval of 2 seconds, and the eye movement entropy value of 1.2. Finally, the three types of indicators are fused into a 10-dimensional comprehensive evaluation feature vector.

[0084] Step 104: input the comprehensive evaluation feature vector into a dual-channel long short-term memory network for deep feature extraction and fusion, wherein the spatio-temporal feature channel processes the flight control time series features in the comprehensive evaluation feature vector, and the cognitive feature channel processes the physiological response features and voice cognitive features in the comprehensive evaluation feature vector.

[0085] In step 104, the dual-channel long short-term memory network is a deep learning network structure containing two processing channels for extracting time sequence features and cognitive features respectively. In the pre-training stage, more than 5000 hours of airline historical training data is used, and knowledge graph technology is introduced to identify corresponding behavior indicators, expected states, associated competency assessment items and their scoring rules according to different training scenarios, so as to extract and fuse deep features of the time sequence mode of flight manipulation and the cognitive state of the pilot respectively. The space-time feature channel is specially used to process features related to time sequence and operation sequence. The cognitive feature channel processes features related to physiological state and voice interaction. Flight manipulation time sequence features are extracted from flight operation data sequences, including roll rate overshoot, glide path tracking error integral and other specific operation parameters, which are used to quantify the accuracy, stability and timeliness of flight operation. Physiological response features are derived from pilot physiological monitoring data, including heart rate variability low / high frequency power ratio (reflecting stress load level) and saccade path entropy value (reflecting visual search efficiency), which are used to evaluate the physiological state and workload of the pilot. Voice cognitive features are extracted from cockpit voice data, including instruction response delay, completeness of repetition and semantic role labeling information, which are used to analyze the communication efficiency, situational understanding ability and decision-making process of the pilot. Manipulation time sequence feature extraction architecture details: a CNN-LSTM model with self-attention mechanism of transformer is used. The CNN layer will first act as a "layered feature detector" to extract local and multi-scale features from the original time sequence data. For example, identify mutations or specific patterns in pitch, roll or speed flight parameters, which are key features for identifying a sharp turn or pull-up action. The LSTM network will process the feature sequence extracted by CNN. The gating mechanism of LSTMs enables it to capture and remember "long-term dependencies" and "sequential patterns" in time series. Self-attention mechanism of transformer is introduced. Transformer can model "complex temporal, spatial and long-distance dependencies" in data

[0086] In the embodiments of the present application, the part of the comprehensive evaluation feature vector representing the operation sequence is input into the space-time feature channel, and the time sequence dependent feature is output; the part representing the physiology and voice is input into the cognitive feature channel, and the context feature is output; the output features of the two channels are spliced, and then dimensionally reduced through a fully connected layer to obtain the final feature representation.

[0087] For example, the first 4 dimensions (operation sequence related) of a 10-dimensional vector are input into the space-time channel, and 128-dimensional time sequence features are output; the last 6 dimensions (physiology and voice related) are input into the cognitive channel, and 128-dimensional cognitive features are output; after splicing into 256-dimensional features, the dimension is reduced to 128-dimensional output vector through a fully connected layer.

[0088] Step 105: Decision tracing is performed on the output of the dual-channel long short-term memory network using SHAP values, and based on the decision tracing result, a capability deficiency node of the pilot is located.

[0089] In step 105, the Shapley Additive exPlanations (SHAP) value is a numerical value used to explain the model prediction result, indicating the contribution of each input feature to the final output. The capability deficiency node refers to a specific weak link identified in the pilot's capability dimension. The capability deficiency node in the actual scenario of the present application refers to the specific capability short board or deficiency link exposed by the pilot in the competency evaluation. It does not refer to the structural node in the dual-channel long short-term memory network, but refers to the specific weak point associated with the pilot's operation performance, cognitive load or decision-making process identified through analysis of the network output and SHAP value tracing. These weak points correspond to the actual skill defects of the pilot.

[0090] In the embodiments of the present application, the SHAP value of each feature in the network output is calculated to obtain a feature contribution set; a feature subset with prominent contribution is identified; the behavior or cognitive link corresponding to these features is analyzed; and the defect node is located in the preset capability dimension space.

[0091] For example, the SHAP values of the features in the 128-dimensional output are calculated, and it is found that the eye movement entropy value (SHAP=0.15) and the response interval (SHAP=-0.22) contribute; analysis shows that attention distraction and slow instruction response are the main problems; and mapping to the capability dimension space locates two defect nodes of "attention allocation" and "decision speed".

[0092] Step 106: The capability deficiency node is matched with a preset core competency deficiency library by combining a preset personalized improvement scheme recommendation mechanism, and corresponding adaptive training content is generated according to the matching result.

[0093] In step 106, the personalized improvement scheme recommendation mechanism is a decision logic based on rules and similarity calculation, constructed based on the Evidence-Based Training (EBT) method, which generates targeted training improvement schemes for each pilot by analyzing the differences between the pilot's actual performance data and the standard competency model, combined with historical training evidence and expert rules. The core competency deficiency library is a database containing typical defect patterns and their corresponding training schemes. Adaptive training content is dynamically synthesized by Generative Adversarial Network (GAN), which can combine multiple specific threat conditions (such as engine failure and low visibility complex situations) to form highly realistic and controllable training scenarios to meet individual training needs. The corresponding adaptive training content explanation: traditional training relies on "static and preset exercises", which makes pilots unprepared when facing unpredictable challenges in the real world. The system proposed in this scheme will overcome this limitation, realizing "dynamic and realistic scenarios" and adjusting in real time according to "pilot's actions and decisions". To this end, the following AI technologies will be used: Reinforcement Learning (RL): RL will be used as the core engine of the system to achieve Dynamic Difficulty Adjustment (DDA). The "environment" of the RL agent is the flight simulator, the "state" is the pilot's real-time performance (e.g. cognitive load, operation accuracy), the "action" is to adjust the scene parameters (such as weather, traffic density, system failure), and the "reward" is the pilot's performance score on the TEM indicators. RL is particularly suitable for decision-making in simulated environments, and its goal is to keep pilots in the "flow channel" of skill and challenge balance, avoiding "cognitive overload" or boredom. Generative AI: In addition to RL, generative AI will also be used to create more immersive and unpredictable training scenarios. The generative model can generate new and unique event combinations based on existing threat and error data, such as a previously unseen combination of wind shear and system failure, thereby enhancing the realism of training. The core goal of this system is to ensure that the training difficulty matches the pilot's skill level. According to the pilot's management results, if effective management is performed and the aircraft reaches the desired state, it means that the pilot performs well in this threat subject, if there is no management or the aircraft appears new threats, it means that the pilot performs poorly in this subject. If the performance is not good, the difficulty will be reduced in the next flight scene generation, for example, in the case of "low visibility approach with engine fire warning". If the performance is not good, the low visibility will not be triggered in the next scene generation. If the performance is good, the difficulty will be increased in the next flight scene generation.Take the example of "engine fire during low-visibility approach". If the performance is good, the next time the scenario is generated, the threat of strong crosswind is increased. At the same time, the cognitive state of the pilot will be evaluated in real time using multi-modal data. For example, if the AI model detects signs of "distraction" in the pilot through physiological data, the system can automatically reduce the complexity of other tasks to avoid "cognitive overload". Conversely, if the pilot shows the ability to easily master the task, the system will gradually increase the difficulty and introduce new threats or failures.

[0094] In the embodiments of the present application, the defect node is matched with the patterns in the defect library for similarity, the defect type is determined, the corresponding scene template is selected from the training template library according to the matching result, and the template elements are parsed and dynamically combined into new training content.

[0095] For example, the defect node "attention allocation" has a similarity of 0.85 with "insufficient situational monitoring" in the defect library, and the defect node "decision speed" has a similarity of 0.78 with "instruction response delay"; the multi-instrument monitoring template and the intensive instruction response template are selected; the template elements are parsed and combined into a composite training scene containing multi-instrument abnormalities and high-frequency instructions.

[0096] The method realizes comprehensive, dynamic and interpretable evaluation of the operation and cognitive ability of the pilot by means of synchronous fusion of multi-modal data, three-level capability evaluation, double-channel deep network feature extraction, interpretable defect positioning and personalized training generation, and can automatically generate targeted training programs, effectively improving the scientificity and effectiveness of flight training.

[0097] In order to solve how to comprehensively, objectively and interpretably evaluate the comprehensive ability of the pilot in a complex situation, in some embodiments, step 103: based on the TEM model, the multi-modal data stream is sequentially subjected to threat perception evaluation, error management evaluation and non-technical skill evaluation, respectively obtaining corresponding threat perception indexes, error management indexes and non-technical skill indexes, as shown in the following table: Figure 2 As shown in the table, it includes:

[0098] Step 201: input the multi-modal data stream into the pre-trained TEM model, and extract the aligned flight control data, the aligned physiological monitoring data and the aligned cockpit voice data from the multi-modal data stream through the evaluation framework of the TEM model.

[0099] In step 201, the aligned flight control data refers to flight operation record data after time synchronization processing, including parameters such as lever amount and rudder position. The aligned physiological monitoring data refers to physiological signals such as heart rate and eye movement after time synchronization processing. The aligned cockpit voice data refers to cockpit conversation audio after time synchronization processing.

[0100] In the embodiment of the present application, the pre-trained TEM model evaluation framework receives a multi-modal data stream and separates three types of data that have been time-aligned therefrom, providing preparation for subsequent module evaluation.

[0101] Step 202: identifying a sudden threat event occurring in a flight environment and an actual flight parameter from the aligned flight control data through a threat management module of the TEM model.

[0102] In step 202, the sudden threat event refers to a dangerous situation that suddenly occurs in the flight environment, such as wind shear. The actual flight parameter refers to the flight state data actually generated by the flight target under the operation of the pilot, such as altitude airspeed, etc.

[0103] In the embodiment of the present application, the threat management module of the TEM model processes the aligned flight control data, identifies the change pattern representing the sudden threat, and records the current flight state parameter at the same time.

[0104] Step 203: calculating a corresponding time interval according to the time when the sudden threat event occurs and the time when the pilot responds to the sudden threat event for the first time, and generating a threat awareness index based on the time interval.

[0105] In step 203, the time interval refers to the time difference between the occurrence of the threat event and the start of the pilot's control action.

[0106] In the embodiment of the present application, the time difference between the identified occurrence time of the threat event and the operation time of the flight control data responding to the event for the first time is calculated, and the time difference is converted into a threat awareness index.

[0107] Step 204: comparing the actual flight parameter with a preset standard flight parameter through an error management module of the TEM model, and calculating a parameter deviation degree according to the comparison result.

[0108] In step 204, the parameter deviation degree refers to the difference between the actual flight parameter and the preset standard flight parameter.

[0109] In the embodiment of the present application, the error management module of the TEM model compares the actual flight parameter with the standard parameter, calculates the deviation between them, and obtains the parameter deviation degree.

[0110] Step 205: calculating the operation sequence efficiency of the pilot in correcting the parameter deviation degree, and generating an error management index based on the operation sequence efficiency.

[0111] In step 205, the operation sequence efficiency refers to the comprehensive measurement of the speed and effect of a series of operations performed by the pilot to correct the flight state deviation.

[0112] In the embodiments of the present application, a series of operation actions of the pilot after the parameter deviation is analyzed, the completion time and the correction effect of the actions are calculated, the operation sequence efficiency is comprehensively obtained, and the error management index is generated accordingly.

[0113] Step 206: Extracting the distribution characteristics of the fixation points on the flight instruments from the eye tracking data contained in the aligned physiological monitoring data through the non-technical skill module of the TEM model, inputting the aligned cockpit voice data into the voice emotion recognition model to identify the decision instruction frequency and the response interval, and generating the non-technical skill index based on the decision instruction frequency, the response interval and the distribution characteristics.

[0114] In step 206, the distribution characteristics of the fixation points refer to the statistical characteristics of the positions and times of the pilot's eye fixation points on different instruments in the cockpit. The decision instruction frequency refers to the number of decision-making instructions issued by the pilot per unit time. The response interval refers to the time difference from issuing the instruction to receiving the effective reply.

[0115] In the embodiments of the present application, the non-technical skill module of the TEM model extracts the distribution characteristics of the fixation points from the eye tracking data, and at the same time, the voice data is input into the voice emotion recognition model to obtain the decision instruction frequency and the response interval, and finally these characteristics are fused to generate the non-technical skill index.

[0116] The following is a specific example:

[0117] In a simulated flight training scenario, the system inputs the formed multi-modal data stream to the pre-trained TEM model, and through the evaluation framework of the model, extracts the aligned flight control data, the aligned physiological monitoring data and the aligned cockpit voice data from the data stream; the threat management module identifies the sudden threat event appearing in the flight environment from the aligned flight control data as wind shear, and records the actual flight parameter height as 280 meters; according to the time interval between the wind shear event occurrence time and the pilot's first manipulation response time, the time interval is calculated as 3.2 seconds, and the threat perception index 3.2 is generated based on the time interval; the error management module compares the actual flight parameter height 280 meters with the preset standard flight parameter height 300 meters, and calculates the parameter deviation degree as 20 meters; the pilot's subsequent push rod and throttle adjustment operation sequence is analyzed, and the operation sequence efficiency is calculated as 0.8 through the time required to complete the effective correction action and the rationality of the action, and the error management index 0.8 is generated based on the operation sequence efficiency; the non-technical skill module extracts the distribution characteristics of the fixation points on the flight instruments from the eye movement tracking data contained in the aligned physiological monitoring data, calculates the distribution entropy value as 1.2, and inputs the aligned cockpit voice data into the voice emotion recognition model, identifies the decision instruction frequency as 5 per minute, and the average response interval as 2 seconds; based on the decision instruction frequency 5, the response interval 2 seconds and the distribution feature entropy value 1.2, the non-technical skill index is generated through weighted fusion calculation, wherein the weighted calculation formula is , wherein S is the non-technical skill index, F is the decision instruction frequency, T is the average response interval, E is the eye movement distribution entropy value, and the weights are respectively set as 0.4, 0.3 and 0.3, and the calculation process is , and the non-technical skill index 2.51 is finally generated; the index is used together with the threat perception index 3.2 and the error management index 0.8 for subsequent feature fusion and comprehensive analysis.

[0118] In the embodiments of the present application, through the above three-level evaluation process, the pilot performance can be quantified comprehensively from the three dimensions of threat response, error correction and non-technical skill, and the interpretable quantitative index is generated, which provides accurate basis for subsequent ability diagnosis and individualized training.

[0119] Specific examples, supplementary explanation of the evaluation process:

[0120] With the identification of the threat state as the starting point of time, continue to analyze the flight data, including the pilot's operation data, flight state data, pilot's physiological data, and cabin voice data. Through the fusion of physiological data, voice data, operation data and other multi-modal data, it is judged whether the pilot has identified the threat (voice instructions, operations such as engine fire extinguishing operations, and whether to enter a stress state combined with EEG and HRV), whether the threat has been identified in time (the time spent from the appearance of the threat to the identification of the threat). Continuously identify the state of the aircraft to determine whether the state of the aircraft has changed (reach the expected state, new threats appear, and others). If the expected state is reached, it means that the pilot's management of the aircraft is effective, and the pilot's operation, physiological data, and voice instructions during the period from the appearance of the threat to the achievement of the expected state are recorded. Identify whether the pilot has performed management and what management has been performed to form a structured data with timestamps and management content (such as no, sometimes the aircraft crew does not manage or does not perform effective management, and the aircraft can sometimes automatically recover to the expected state). As shown in Table 1:

[0121] Table 1: Example of structured data

[0122] Time Type Content XX:XX:XX.XXX T Engine fire XX:XX:XX.XXX Physiological data Rising stress (possible perceived threat) XX:XX:XX.XXX Voice Instruction to handle fault recognized XX:XX:XX.XXX M Firefighting operation performed XX:XX:XX.XXX Desired state Fire alarm disappeared

[0123] Or the aircraft appears a new threat, then take the new threat as the end of the structured data, and so on until the training is completed, recording the entire training process of the pilot, as shown in Table 2:

[0124] Table 2: Example of new threat as structured data

[0125] Time Type Content XX:XX:XX.XXX T Engine fire XX:XX:XX.XXX Physiological data Rising stress (possible perceived threat) XX:XX:XX.XXX Voice Instruction to handle fault recognized XX:XX:XX.XXX M Firefighting operation performed XX:XX:XX.XXX New threat Airplane attitude exceeds safety margin

[0126] To solve how to quantify the decision-making and communication efficiency of the pilot from the cockpit voice data, in some embodiments, step 206: input the aligned cockpit voice data into a voice emotion recognition model to identify decision instruction frequency and response interval, and generate non-technical skill indicators based on the decision instruction frequency, the response interval and the distribution characteristics, including:

[0127] Step 301: input the aligned cockpit voice data into a voice emotion recognition model to identify decision instructions in the aligned cockpit voice data through the voice emotion recognition model.

[0128] In step 301, the decision instruction refers to a voice segment issued by the pilot in the cockpit environment that contains an explicit operation intention or action instruction.

[0129] In the embodiment of the present application, the aligned cockpit voice data is input into a voice emotion recognition model, which automatically identifies and labels all voice segments containing decision instructions by analyzing the spectral features and semantic content of the voice signals.

[0130] Step 302: Count the number of occurrences of the decision instructions in a preset time period to obtain the decision instruction frequency.

[0131] In the embodiment of the present application, a fixed statistical time period is set, and all decision instructions identified by the voice emotion recognition model in this time period are counted to obtain the decision instruction frequency.

[0132] Step 303: Calculate the time difference from the issuance of the decision instruction to the receipt of the effective voice response to the decision instruction, and obtain the response interval based on the time difference.

[0133] In step 303, effective voice response refers to voice reply to the issued decision instruction that meets the aviation communication specification and has correct content, which means that after excluding invalid noise, irrelevant dialogue or incorrect repetition, it can be identified as an effective confirmation or execution feedback to the original instruction.

[0134] In the embodiment of the present application, for each identified decision instruction, its corresponding effective voice response is searched in the subsequent voice data, the time difference between the starting points of the two voice segments is accurately calculated to obtain the response interval of the instruction, and the average response interval can be calculated. The specific process of calculating the time difference is as follows: first, determine the issuance time point of the decision instruction on the time axis, then identify and locate the starting time point of the effective voice response to the instruction, and finally calculate the difference between the two time points to obtain the response interval.

[0135] Step 304: Fuse the decision instruction frequency, the response interval and the distribution feature, and generate a non-technical skill indicator based on the fusion result.

[0136] In step 304, fusion refers to the process of combining multiple features from different sources into a comprehensive value according to predetermined rules.

[0137] In the embodiment of the present application, the decision instruction frequency, the average response interval and the gaze point distribution feature obtained from the physiological data are weighted and calculated, and the calculation result is mapped to the final non-technical skill indicator.

[0138] The following is a specific example:

[0139] In the simulated flight training scene, the system inputs the aligned cockpit voice data into a voice emotion recognition model, which identifies decision instructions containing explicit operation intentions, such as checking airspeed and maintaining heading, from continuous voice by analyzing the spectral and semantic features of the voice signal; the system counts that the identified decision instructions occur 5 times in 1 minute, obtaining a decision instruction frequency of 5 times per minute; for each decision instruction, the model locates its time of issuance and finds valid voice responses that meet the aviation communication specifications in the subsequent voice stream, such as finding the starting time of the response "airspeed is normal" for the instruction "check airspeed", calculating the time difference from issuance to valid response for each instruction, and obtaining an average response interval of 2 seconds by taking the arithmetic mean of the time differences of all instructions; the extracted gaze point distribution feature entropy value 1.2 from the eye movement data is fused with the decision instruction frequency 5 times per minute and the average response interval 2 seconds, and a weighted sum formula is used to calculate the non-technical skill index, Finally, a non-technical skill index 2.51 is generated, which is used to quantitatively represent the comprehensive performance of the pilot in decision-making communication and attention allocation.

[0140] In the embodiments of the present application, through the above steps, key communication and decision-making features can be automatically extracted from voice interaction, and combined with visual attention distribution, a quantitative non-technical skill index is generated, providing a reliable basis for objectively evaluating the team cooperation and situational management ability of pilots.

[0141] In order to further improve the standardization and interpretability of the generation of non-technical skill index, in some embodiments, step 304: the decision instruction frequency, the response interval and the distribution feature are fused, and based on the fusion result, a non-technical skill index is generated, comprising:

[0142] Step 401: assigning corresponding weights to the decision instruction frequency, the response interval and the distribution feature respectively.

[0143] In step 401, the weight refers to the importance coefficient preset for the decision instruction frequency, the response interval and the distribution feature respectively, which is used to control the contribution proportion of each feature in the final index.

[0144] In the embodiments of the present application, according to the domain prior knowledge or historical data analysis, fixed weight values are assigned to the decision instruction frequency, the response interval and the distribution feature, and these weights remain unchanged in the evaluation process.

[0145] Step 402: performing weighted calculation on the decision instruction frequency, the response interval and the distribution feature according to the weights, to obtain a comprehensive evaluation value.

[0146] In step 402, the weighted calculation refers to a mathematical operation process of multiplying each feature by its corresponding weight and then adding them up. The comprehensive evaluation value is a preliminary numerical result obtained by weighted calculation.

[0147] In the embodiment of the present application, the decision instruction frequency is multiplied by the first weight, the response interval is multiplied by the second weight after taking the reciprocal, and the distribution feature is multiplied by the third weight. Then, the three product results are added to obtain the comprehensive evaluation value.

[0148] Step 403: input the comprehensive evaluation value into a preset index mapping rule library, and convert the comprehensive evaluation value into a corresponding non-technical skill level through the mapping relationship in the index mapping rule library. The non-technical skill level is a non-technical skill index.

[0149] In step 403, the index mapping rule library is a database containing a plurality of numerical interval and level correspondence relationships. The non-technical skill level is a grading result obtained by mapping the interval in which the comprehensive evaluation value is located, and is the final non-technical skill index.

[0150] In the embodiment of the present application, the comprehensive evaluation value calculated is compared with the pre-defined numerical interval in the index mapping rule library to determine the interval range to which the value belongs, and the level identifier corresponding to the interval is output.

[0151] The following is a specific example:

[0152] In the simulated flight training scene, the system respectively assigns corresponding weights to the decision instruction frequency, the response interval, and the distribution feature, wherein the decision instruction frequency weight is 0.4, the response interval weight is 0.3, and the distribution feature weight is 0.3. The comprehensive evaluation value is obtained by weighted calculation according to the weights, and the calculation formula is wherein V represents the comprehensive evaluation value as a dimensionless value, the decision instruction frequency weight of 0.4 is dimensionless, and F represents the decision instruction frequency of 5 times per minute, the response interval weight of 0.3 is dimensionless, and T represents the average response interval of 2 seconds, the distribution feature weight of 0.3 is dimensionless, and E represents the fixation point distribution feature entropy value of 1.2, which is dimensionless. The numerical calculation process is ; input the comprehensive evaluation value 2.51 into a preset index mapping rule library, the rule library including a mapping relationship between a numerical interval and a level, wherein an interval 0 to 1.5 is mapped to a non-technical skill level one, an interval 1.5 to 2.5 is mapped to a level two, an interval 2.5 to 3.5 is mapped to a level three, and an interval 3.5 or above is mapped to a level four; since 2.51 is in the interval 2.5 to 3.5, it is converted to a non-technical skill level three through the mapping relationship, and the non-technical skill level is the final non-technical skill index.

[0153] In the embodiments of the present application, through the three steps of weight distribution, weighted calculation and level mapping, the multi-dimensional features are fused into a unified grading index, so that the evaluation result is more standardized and easy to understand, and the standardized evaluation of the pilot's non-technical skills is effectively supported.

[0154] In order to further improve the extraction and fusion ability of the deep features of the pilot's state, in some embodiments, step 104: inputting the comprehensive evaluation feature vector into a double-channel long short-term memory network for deep feature extraction and fusion, comprises:

[0155] Step 501: input the flight control time sequence feature in the comprehensive evaluation feature vector into the space-time feature channel of the double-channel long short-term memory network, and extract deep features from the flight control time sequence feature through the space-time feature channel to obtain time sequence dependent features.

[0156] In step 501, the time sequence dependent feature refers to a high-level feature extracted from the flight control time sequence feature, which can reflect the front and rear dependent relationship and change mode between operation actions.

[0157] In the embodiments of the present application, the part related to the flight control sequence in the comprehensive evaluation feature vector is input into the space-time feature channel of the network, which is composed of long short-term memory units, and the long-term and short-term dependent relationship in the operation data is learned through the gating mechanism thereof, and the time sequence dependent feature representing the operation mode is output.

[0158] Step 502: input the physiological reaction feature and the speech cognitive feature in the comprehensive evaluation feature vector into the cognitive feature channel of the double-channel long short-term memory network, and extract deep features from the physiological reaction feature and the speech cognitive feature through the cognitive feature channel to obtain context features.

[0159] In step 502, the context feature refers to a high-level feature extracted from the physiological reaction feature and the speech cognitive feature, which can reflect the cognitive load, emotional state and communication situation of the pilot.

[0160] In the embodiments of the present application, the part of the comprehensive evaluation feature vector related to the physiological state and the voice interaction is input into a cognitive feature channel of the network, which is also composed of long short-term memory units. By analyzing the context information in the sequence data, the context feature representing the cognitive state is output.

[0161] Step 503: The time sequence dependent feature is spliced with the context feature to form a spliced feature.

[0162] In step 503, the spliced feature refers to a merged feature vector formed by connecting the time sequence dependent feature and the context feature in the feature dimension.

[0163] In the embodiments of the present application, the time sequence dependent feature output by the space-time feature channel and the context feature output by the cognitive feature channel are connected at the head and tail according to the feature dimension to form a fusion feature vector with higher dimension.

[0164] Step 504: The spliced feature is input into a full connection layer of the double-channel long short-term memory network for dimension reduction processing to obtain a reduced dimension feature, and the reduced dimension feature is an output result of the double-channel long short-term memory network.

[0165] In step 504, the reduced dimension feature refers to a feature representation obtained by compressing the high-dimensional spliced feature to a lower dimensional space through linear transformation, which contains the main information of the original high-dimensional feature.

[0166] In the embodiments of the present application, the spliced feature is input into the full connection layer at the end of the network. The layer compresses and transforms the high-dimensional feature through the calculation of the weight matrix and the activation function, and outputs a low-dimensional and more representative feature vector as the final output result of the network.

[0167] The following is a specific example:

[0168] In the simulated flight training scene, the system inputs the flight manipulation timing feature part in the 10-dimensional comprehensive evaluation feature vector, i.e., the first 4-dimensional feature vector [0.85, 0.12, 0.93, 0.05], into the space-time feature channel of the dual-channel long short-term memory network, which is composed of multiple long short-term memory units that learn the dependency relationship in the operation sequence through the internal gating mechanism, and outputs a 128-dimensional timing dependency feature; the physiological response feature and the speech cognitive feature part in the comprehensive evaluation feature vector, i.e., the last 6-dimensional feature vector [0.75, 1.20, 0.80, 0.33, 0.90, 0.21], are input into the cognitive feature channel of the network, which is also composed of long short-term memory units, and outputs a 128-dimensional context feature by analyzing the sequence context information; the timing dependency feature and the context feature are spliced, i.e., the two 128-dimensional feature vectors are connected to form a 256-dimensional spliced feature; the 256-dimensional spliced feature is input into the fully connected layer of the dual-channel long short-term memory network for dimension reduction processing, and the fully connected layer realizes dimension reduction through weight matrix calculation, where the weight matrix has a size of 256 rows and 128 columns, and the calculation process is where Y represents the output reduced feature as a 128-dimensional vector, W represents the weight matrix as a 256x128-dimensional real matrix, X represents the input spliced feature as a 256-dimensional vector, and B represents the bias vector as a 128-dimensional vector. Through the linear transformation and the activation function processing, a 128-dimensional reduced feature is finally output, which is the output result of the dual-channel long short-term memory network and is used for subsequent decision trace analysis.

[0169] In the embodiments of the present application, the operation timing and cognitive state features are processed by the dual channels respectively, and are deeply fused and reduced, so that the deep feature representation capable of comprehensively reflecting the operation proficiency and situational awareness ability of the pilot is effectively extracted, thereby providing a high-quality data basis for accurate evaluation.

[0170] In order to further improve the explainable analysis capability of the root cause of the pilot's ability defects, in some embodiments, step 105: the output result of the dual-channel long short-term memory network is subjected to decision trace analysis by using SHAP values, and based on the decision trace analysis result, the ability defect node of the pilot is located, including:

[0171] Step 601: using SHAP values, the contribution degree values of each input feature in the output result of the dual-channel long short-term memory network are calculated, and based on all the contribution degree values, a feature contribution degree set is generated.

[0172] In step 601, the input features refer to the feature items formed after extraction and conversion of all original data input into the dual-channel long short-term memory network for processing, specifically including but not limited to threat perception indicators obtained from threat perception assessment, error management indicators obtained from error management assessment, non-technical skill indicators obtained from non-technical skill assessment, and bottom-layer operation sequence features, physiological response features and speech cognitive features directly extracted from multi-modal data, which together constitute the input of the network for representing the operation behavior, physiological state and cognitive process of the pilot. The contribution value refers to the influence degree value of each input feature on the final output result of the network, with a positive value indicating positive promotion and a negative value indicating negative inhibition. The feature contribution set is a set of contribution values of all input features.

[0173] In the embodiments of the present application, the SHAP value calculation method is used to perform reverse analysis on the output result of the dual-channel long short-term memory network, calculate the contribution degree of each feature dimension in the comprehensive evaluation feature vector to the output result, obtain the contribution value of each feature, and all these values form the feature contribution set.

[0174] Step 602: identifying a feature subset that plays a leading role in the output result from the feature contribution set.

[0175] In step 602, the leading role refers to the fact that some input features have a higher influence weight on the output result of the dual-channel long short-term memory network than other features, and the changes of these features directly determine the final output category or value of the network, representing the key factors affecting the evaluation result of the pilot. The feature subset is a set of features with a leading role selected from the feature contribution set.

[0176] In the embodiments of the present application, a contribution threshold is set, and features with a contribution absolute value greater than the threshold in the feature contribution set are selected to form the feature subset, which are considered to play a leading role in the output result.

[0177] Step 603: performing decision trace analysis on the feature subset to obtain a decision trace result.

[0178] In step 603, decision trace analysis refers to further analysis of the selected feature subset to trace the specific pilot behavior or cognitive link corresponding to the features. The decision trace result is a conclusion about the main influencing factors and sources obtained after analysis.

[0179] In the embodiments of the present application, for each feature in the feature subset, the actual meaning represented by the feature in the original comprehensive evaluation feature vector is analyzed, and the corresponding pilot operation behavior or physiological cognitive state is traced, so as to obtain the main factor analysis conclusion leading to the output result of the network.

[0180] Step 604: mapping the decision traceability result to a preset pilot capability dimension space, and locating a capability defect node meeting a preset judgment standard condition in the pilot capability dimension space.

[0181] In step 604, the pilot capability dimension space is a preset structured space containing multiple capability dimensions, each of which represents a specific pilot capability. The preset judgment standard condition refers to a threshold or rule set in advance through historical data analysis and expert experience, which is used to quantitatively judge whether the feature contribution is "low or abnormally fluctuating". The specific content includes threshold parameters such as the normal range threshold of feature contribution obtained based on a large number of normal operation samples, the expected baseline value of feature contribution under different flight scenarios, and the reasonable fluctuation interval of contribution degree over time.

[0182] In the embodiments of the present application, the decision traceability result is matched with the capability dimensions defined in the pilot capability dimension space, and according to the preset judgment standard condition, such as negative contribution degree or deviation from the normal range, a specific capability defect node is located on the corresponding dimension.

[0183] The following is a specific example:

[0184] In the simulated flight training scene, the system uses the SHAP value calculation method to analyze the 128-dimensional feature vector output by the dual-channel long short-term memory network, calculates the contribution degree value of each input feature in the comprehensive evaluation feature vector to the output result, wherein the comprehensive evaluation feature vector is composed of threat perception index 3.2, error management index 0.8, non-technical skill index 0.75 and other underlying features, the contribution degree value of the eye movement entropy value feature is -0.22, the contribution degree value of the response interval feature is -0.18, the contribution degree value of the threat perception index feature is 0.12, and the contribution degree values of the remaining features are all less than 0.1. Based on all the contribution degree values, a feature contribution degree set is generated. From the set, a feature subset that plays a leading role in the output result is identified, and a threshold for judging the leading role is set as the absolute value of the contribution degree being greater than 0.15. Therefore, the feature subset composed of eye movement entropy value -0.22 and response interval -0.18 is selected. Decision traceability analysis is performed on the feature subset, and the analysis shows that negative contribution of eye movement entropy value indicates that attention dispersion has a negative impact on the operation result, and negative contribution of response interval indicates that instruction response delay has a negative impact on the operation result, and the decision traceability result is obtained as insufficient attention allocation and slow decision speed. The decision traceability result is mapped to a preset pilot capability dimension space, which contains multiple capability dimensions such as attention allocation, decision speed, situational awareness, etc. According to the preset judgment standard condition that the contribution degree value is less than -0.15, the attention allocation dimension is located to the attention stability deficiency capability defect node, and the decision speed dimension is located to the instruction response delay capability defect node.

[0185] In the embodiments of the present application, by quantifying feature contribution, screening key factors, tracing behavior roots and mapping ability space, precise and interpretable positioning from model output to specific ability defect nodes is realized, providing a clear direction for targeted training improvement.

[0186] In order to further improve the pertinence and automation level of training content generation, in some embodiments, step 106: the combination of the pre-set personalized improvement scheme recommendation mechanism matches the ability defect node with the pre-set core competence defect library, and generates corresponding adaptive training content according to the matching result, including:

[0187] Step 701: using the pre-set personalized improvement scheme recommendation mechanism, the similarity between the ability defect node and the defect mode in the core competence defect library is calculated.

[0188] In step 701, similarity calculation refers to measuring the similarity between two objects by mathematical methods, which is used to compare the matching degree of the ability defect node and the defect mode in this scenario.

[0189] In the embodiments of the present application, the similarity value between each ability defect node and each defect mode description vector in the core competence defect library is calculated by the pre-set similarity algorithm in the personalized improvement scheme recommendation mechanism.

[0190] Step 702: based on the similarity, determine the core competence defect type, which is the matching result.

[0191] In step 702, the core competence defect type refers to the pre-defined defect classification identifier with clear meaning in the defect library, and the matching result refers to the most similar defect type found for each ability defect node.

[0192] In the embodiments of the present application, for each ability defect node, the type corresponding to the defect mode with the highest similarity value is selected, and the type is determined as the core competence defect type as the matching result of the node.

[0193] Step 703: according to the core competence defect type, select the corresponding basic training scene template from the pre-set training scene template library.

[0194] In step 703, the basic training scene template is a pre-designed training scene framework for a specific type of defect, which includes training objectives and basic scenario settings.

[0195] In the embodiments of the present application, according to the determined core competence defect type, one or more basic training scene templates associated with the type are queried from the training scene template library.

[0196] Step 704: parsing the basic training scene template to obtain corresponding basic training scene elements.

[0197] In step 704, the basic training scene elements refer to the basic parameters and conditions that constitute a training scene, such as environmental settings, task types, difficulty parameters, etc.

[0198] In the embodiments of the present application, each selected basic training scene template is parsed to extract various parameters and conditions defined therein, thereby obtaining independently configurable basic training scene elements.

[0199] Step 705: combining all basic training scene elements to generate adaptive training content.

[0200] In step 705, combining refers to integrating multiple elements into an organic whole according to certain rules.

[0201] In the embodiments of the present application, all parsed basic training scene elements are intelligently combined and parameter-adjusted to form a coherent and appropriately challenging composite training scene content.

[0202] The following is a specific example:

[0203] In the simulated flight training scene, the system uses a preset personalized improvement scheme recommendation mechanism to perform similarity calculation on the attention allocation of the ability deficiency node and the defect patterns in the core competence deficiency library, and adopts a cosine similarity algorithm, wherein the cosine similarity calculation formula is wherein Sim represents a dimensionless value of similarity, A represents a feature vector of the ability deficiency node, B represents a feature vector of the deficiency mode, A·B represents multiplication and summation of elements in corresponding positions of vectors, norm operation represents square root of summation of squares of elements of a vector, similarity of the attention allocation node to the situation monitoring deficiency mode is calculated as 0.85; similarity of the ability deficiency node decision speed to the deficiency mode is calculated, and similarity to the instruction response delay mode is 0.78; based on the highest similarity, the core competence deficiency type is determined, the attention allocation node corresponds to the situation monitoring deficiency, and the decision speed node corresponds to the instruction response delay; according to the deficiency type, a multi-instrument monitoring template and a dense instruction response template are selected from a preset training scene template library; the multi-instrument monitoring template is analyzed to obtain basic training scene elements including 3 abnormal instruments, a monitoring duration of 120 seconds, and 10 attention switching times; the dense instruction response template is analyzed to obtain basic training scene elements including 5 instructions, a response time requirement of 3 seconds, and 2 repeated verification times; all the basic training scene elements are combined to generate adaptive training content, which requires the pilot to monitor 3 abnormal instruments within 120 seconds, complete 10 attention switching times, process 5 instructions and respond within 3 seconds, and perform 2 repeated verifications, forming a composite training scene.

[0204] In the embodiments of the present application, by automatically matching specific ability deficiencies to standard deficiency types and intelligently generating a composite training scene based on the types, highly personalized training content recommendation is achieved, effectively improving the pertinence and efficiency of training.

[0205] Figure 3 A structure schematic diagram of a pilot dynamic evaluation system based on a TEM model is provided in the embodiments of the present application, and the specific implementation part describes:

[0206] The acquisition module 31 is configured to acquire flight control data of a flight target, physiological monitoring data of a pilot, and cockpit voice data.

[0207] The alignment module 32 is configured to perform time synchronization alignment processing on the flight control data, the physiological monitoring data, and the cockpit voice data to form a multi-modal data stream.

[0208] The evaluation module 33 is configured to perform threat perception evaluation, error management evaluation, and non-technical skill evaluation on the multi-modal data stream in sequence based on the TEM model to obtain corresponding threat perception indexes, error management indexes, and non-technical skill indexes, respectively, and perform feature fusion on the threat perception indexes, the error management indexes, and the non-technical skill indexes to form a comprehensive evaluation feature vector.

[0209] The input module 34 is configured to input the comprehensive evaluation feature vector into the dual-channel long short-term memory network for deep feature extraction and fusion, wherein the temporal-spatial feature channel processes the flight control time sequence feature in the comprehensive evaluation feature vector, and the cognitive feature channel processes the physiological response feature and the speech cognitive feature in the comprehensive evaluation feature vector.

[0210] The positioning module 35 is configured to perform decision tracing on the output result of the dual-channel long short-term memory network by using the SHAP value, and locate the ability defect node of the pilot based on the decision tracing result.

[0211] The matching module 36 is configured to match the ability defect node with a preset core competence defect library by combining a preset personalized improvement scheme recommendation mechanism, and generate corresponding adaptive training content according to the matching result.

[0212] The pilot dynamic evaluation system based on the TEM model according to the embodiments of the present application is used to implement the pilot dynamic evaluation method based on the TEM model as described above, and therefore the specific embodiments of the pilot dynamic evaluation system based on the TEM model can be seen from the foregoing embodiments of the pilot dynamic evaluation method based on the TEM model, and the specific embodiments can be referred to the descriptions of the corresponding embodiments of each part, which will not be described herein again.

[0213] The present application also provides an electronic device, comprising: a memory configured to store a computer program; and a processor configured to execute the computer program to implement the steps of the pilot dynamic evaluation method based on the TEM model as described above.

[0214] The present application also provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the pilot dynamic evaluation method based on the TEM model as described above.

[0215] In an exemplary embodiment, the computer readable storage medium as described above can include, but is not limited to, a U disk, a read-only memory, a random access memory, a mobile hard disk, a magnetic disk or an optical disk, and various media that can store computer programs.

[0216] The embodiments of the present application also provide a computer program product, wherein the computer program product includes a computer program, and the computer program is executed by a processor to implement the steps in the pilot dynamic evaluation method based on the TEM model as described above.

[0217] Those skilled in the art will further realize that the mere concepts, teachings, and embodiments described herein are merely meant to provide an enabling description of embodiments of the present application and are not intended to limit the scope of the present application. Accordingly, embodiments as described herein contemplate all modifications that come within the scope of the present application.

[0218] The pilot dynamic assessment method, system, device and storage medium based on the TEM model are described in detail above. The principles and implementation modes of the present application are described by applying specific examples in this paper. The above description of the embodiments is only used to help understand the method and its core idea of the present application. It should be pointed out that for ordinary skilled in the art, without departing from the principles of the present application, some improvements and modifications can be made to the present application, and these improvements and modifications also fall within the protection scope of the present application.

Claims

1. A pilot dynamic evaluation method based on a TEM model, characterized in that, include: Acquire flight control data of the target, physiological monitoring data of the pilot, and cockpit voice data; The flight control data, the physiological monitoring data, and the cockpit voice data are time-synchronized and aligned to form a multimodal data stream; Based on the TEM model, the multimodal data stream is sequentially subjected to threat perception assessment, error management assessment, and non-technical skills assessment to obtain corresponding threat perception indicators, error management indicators, and non-technical skills indicators. The threat perception indicators, error management indicators, and non-technical skills indicators are then fused to form a comprehensive assessment feature vector. The comprehensive evaluation feature vector is input into a dual-channel long short-term memory network for deep feature extraction and fusion. The spatiotemporal feature channel processes the flight control timing features in the comprehensive evaluation feature vector, and the cognitive feature channel processes the physiological response features and speech cognitive features in the comprehensive evaluation feature vector. The SHAP value is used to trace the output of the dual-channel long short-term memory network, and the pilot's capability deficiency node is located based on the tracing result. Combining a pre-set personalized improvement scheme recommendation mechanism, the capability defect nodes are matched with a pre-set core competency defect library, and corresponding adaptive training content is generated based on the matching results. Based on the TEM model, the multimodal data stream is sequentially assessed for threat perception, error management, and non-technical skills, yielding corresponding threat perception indices, error management indices, and non-technical skill indices, including: The multimodal data stream is input into a pre-trained TEM model, and aligned flight control data, aligned physiological monitoring data, and aligned cockpit voice data are extracted from the multimodal data stream through the evaluation framework of the TEM model. The threat management module of the TEM model identifies sudden threat events and actual flight parameters in the flight environment from the aligned flight control data. Based on the time of occurrence of the sudden threat event and the time of the pilot's first maneuver in response to the sudden threat event, the corresponding time interval is calculated, and a threat perception index is generated based on the time interval. The error management module of the TEM model compares the actual flight parameters with the preset standard flight parameters and calculates the degree of parameter deviation based on the comparison results. Calculate the efficiency of the pilot's operational sequence for correcting deviations from the parameters, and generate error management indicators based on the operational sequence efficiency; The non-technical skills module of the TEM model extracts the distribution features of gaze points on flight instruments from the eye-tracking data contained in the aligned physiological monitoring data. The aligned cockpit voice data is then input into the voice emotion recognition model to identify the frequency of decision commands and the response interval. Based on the frequency of decision commands, the response interval, and the distribution features, non-technical skills indicators are generated. The step of inputting the comprehensive evaluation feature vector into a dual-channel long short-term memory network for deep feature extraction and fusion includes: The flight control timing features in the comprehensive evaluation feature vector are input into the spatiotemporal feature channel of the dual-channel long short-term memory network. The flight control timing features are then extracted through the spatiotemporal feature channel to obtain the timing-dependent features. The physiological response features and speech cognition features in the comprehensive evaluation feature vector are input into the cognitive feature channel of the dual-channel long short-term memory network. Deep feature extraction is performed on the physiological response features and speech cognition features through the cognitive feature channel to obtain context features. The temporal dependency feature and the context feature are concatenated to form the concatenated feature; The spliced ​​features are input into the fully connected layer of the dual-channel long short-term memory network for dimensionality reduction, resulting in dimensionality-reduced features, which are the output of the dual-channel long short-term memory network.

2. The method according to claim 1, characterized in that, The aligned cockpit voice data is input into a voice emotion recognition model to identify the frequency of decision commands and response intervals. Based on the frequency of decision commands, the response intervals, and the distribution characteristics, non-technical skill indicators are generated, including: The aligned cockpit voice data is input into the voice emotion recognition model, and the voice emotion recognition model identifies the decision commands in the aligned cockpit voice data. The frequency of the decision instruction is obtained by counting the number of times the decision instruction appears within a preset time period. Calculate the time difference between issuing the decision instruction and receiving the effective voice response to the decision instruction, and obtain the response interval based on the time difference; The frequency of decision instructions, the response interval, and the distribution characteristics are fused together, and non-technical skill indicators are generated based on the fusion result.

3. The method according to claim 2, characterized in that, The process of fusing the frequency of decision commands, the response interval, and the distribution characteristics, and generating non-technical skill indicators based on the fusion result, includes: Assign corresponding weights to the frequency of the decision command, the response interval, and the distribution characteristics, respectively. Based on the weights, the frequency of the decision instructions, the response interval, and the distribution characteristics are weighted and calculated to obtain a comprehensive evaluation value. The comprehensive evaluation value is input into a preset indicator mapping rule library. Through the mapping relationship in the indicator mapping rule library, the comprehensive evaluation value is converted into the corresponding non-technical skill level, and the non-technical skill level is a non-technical skill indicator.

4. The method according to claim 1, characterized in that, The process of using SHAP values ​​to perform decision tracing on the output of the dual-channel long short-term memory network, and locating the pilot's capability deficiency nodes based on the decision tracing results, includes: Using the SHAP value, the contribution value of each input feature in the output of the dual-channel long short-term memory network is calculated, and a feature contribution set is generated based on all contribution values. Identify the subset of features that play a dominant role in the output result from the set of feature contributions; Decision source tracing analysis is performed on the feature subset to obtain the decision source tracing results; The decision tracing results are mapped to a preset pilot capability dimension space, and capability defect nodes that meet preset judgment criteria are located in the pilot capability dimension space.

5. The method according to claim 1, characterized in that, The aforementioned personalized improvement scheme recommendation mechanism matches the capability deficiency nodes with a preset core competency deficiency database, and generates corresponding adaptive training content based on the matching results, including: Using a pre-defined personalized improvement scheme recommendation mechanism, the similarity between the capability defect node and the defect patterns in the core competency defect library is calculated. Based on the similarity, the core competency deficiency type is determined, and the core competency deficiency type is the matching result; Based on the type of core competency deficiency, select the corresponding basic training scenario template from the preset training scenario template library; The basic training scenario template is analyzed to obtain the corresponding basic training scenario elements; All basic training scenario elements are combined to generate adaptive training content.

6. A pilot dynamic evaluation system based on a TEM model, characterized in that, include: The acquisition module is used to acquire flight control data of the flight target, physiological monitoring data of the pilot, and cockpit voice data. The alignment module is used to perform time synchronization alignment processing on the flight control data, the physiological monitoring data, and the cockpit voice data to form a multimodal data stream; The evaluation module is used to perform threat perception evaluation, error management evaluation and non-technical skills evaluation on the multimodal data stream based on the TEM model, and obtain the corresponding threat perception index, error management index and non-technical skills index respectively. The threat perception index, the error management index and the non-technical skills index are fused to form a comprehensive evaluation feature vector. The input module is used to input the comprehensive evaluation feature vector into a dual-channel long short-term memory network for deep feature extraction and fusion. The spatiotemporal feature channel processes the flight control timing features in the comprehensive evaluation feature vector, and the cognitive feature channel processes the physiological response features and speech cognitive features in the comprehensive evaluation feature vector. The positioning module is used to perform decision tracing on the output results of the dual-channel long short-term memory network using SHAP values, and to locate the pilot's capability defect node based on the decision tracing results; The matching module is used to combine the preset personalized improvement scheme recommendation mechanism to match the capability defect nodes with the preset core competency defect library, and generate corresponding adaptive training content based on the matching results. Based on the TEM model, the multimodal data stream is sequentially assessed for threat perception, error management, and non-technical skills, yielding corresponding threat perception indices, error management indices, and non-technical skill indices, including: The multimodal data stream is input into a pre-trained TEM model, and aligned flight control data, aligned physiological monitoring data, and aligned cockpit voice data are extracted from the multimodal data stream through the evaluation framework of the TEM model. The threat management module of the TEM model identifies sudden threat events and actual flight parameters in the flight environment from the aligned flight control data. Based on the time of occurrence of the sudden threat event and the time of the pilot's first maneuver in response to the sudden threat event, the corresponding time interval is calculated, and a threat perception index is generated based on the time interval. The error management module of the TEM model compares the actual flight parameters with the preset standard flight parameters and calculates the degree of parameter deviation based on the comparison results. Calculate the efficiency of the pilot's operational sequence for correcting deviations from the parameters, and generate error management indicators based on the operational sequence efficiency; The non-technical skills module of the TEM model extracts the distribution features of gaze points on flight instruments from the eye-tracking data contained in the aligned physiological monitoring data. The aligned cockpit voice data is then input into the voice emotion recognition model to identify the frequency of decision commands and the response interval. Based on the frequency of decision commands, the response interval, and the distribution features, non-technical skills indicators are generated. The step of inputting the comprehensive evaluation feature vector into a dual-channel long short-term memory network for deep feature extraction and fusion includes: The flight control timing features in the comprehensive evaluation feature vector are input into the spatiotemporal feature channel of the dual-channel long short-term memory network. The flight control timing features are then extracted through the spatiotemporal feature channel to obtain the timing-dependent features. The physiological response features and speech cognition features in the comprehensive evaluation feature vector are input into the cognitive feature channel of the dual-channel long short-term memory network. Deep feature extraction is performed on the physiological response features and speech cognition features through the cognitive feature channel to obtain context features. The temporal dependency feature and the context feature are concatenated to form the concatenated feature; The spliced ​​features are input into the fully connected layer of the dual-channel long short-term memory network for dimensionality reduction, resulting in dimensionality-reduced features, which are the output of the dual-channel long short-term memory network.

7. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the pilot dynamic evaluation method based on the TEM model as described in any one of claims 1 to 5 when executing the computer program.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, enables the implementation of the pilot dynamic evaluation method based on the TEM model as described in any one of claims 1 to 5.

Images