Pilot competency dynamic assessment method, system and device based on multi-modal data and storage medium

Abstract

The application relates to the technical field of multi-modal data, and provides a pilot competence dynamic evaluation method, system, device and storage medium based on multi-modal data, which solves the problems of low precision of pilot competence evaluation and poor pertinence of training guidance. The method comprises the following steps: collecting flight control data, physiological signal data, psychological test data, subjective scale data and international civil aviation organization core competence information; adopting a time synchronization algorithm to align and fuse the multi-source data; identifying an attention fixation mode based on a hidden Markov model; fusing a subjective scale and a physiological entropy value to construct a workload index; inputting multi-modal features into a pre-trained competence evaluation model; outputting three-level psychological evaluation indexes including a basic ability layer, a dynamic performance layer and a risk early warning layer; and finally generating a personalized training report. The application improves the precision of pilot competence evaluation and the pertinence of training guidance.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multi-modal data, in particular to a pilot competence dynamic evaluation method and system based on multi-modal data, a device and a storage medium. BACKGROUND

[0002] In the field of aviation training, with the increasing complexity of flight tasks, there is an urgent need for a comprehensive evaluation method that can capture the cognitive state, operational performance and psychological load of pilots in real time. The traditional single-dimensional evaluation system has been unable to meet the demand for precise quantification of human performance in modern aviation training, and a technical solution is needed that can integrate multi-source data and dynamically reflect the comprehensive competence of pilots.

[0003] Currently, there is an evaluation scheme based on physiological signals and operation data. This scheme collects physiological indicators such as electrocardiogram and skin galvanic response of pilots during simulated flight, combines joystick input parameters, and uses a support vector machine algorithm to build a classification model to evaluate the workload level of pilots. This scheme detects abnormalities in physiological signals by setting fixed thresholds and generates a single-dimensional load evaluation report.

[0004] However, this scheme has deficiencies in the time sequence correlation processing of physiological signals and operation data, and lacks a dynamic adaptation mechanism for different flight stage-specific correlation patterns. The classification model used in the scheme has limited depth in describing the non-linear relationships between multi-dimensional features, resulting in insufficient granularity in recognizing complex cognitive states. In addition, the evaluation results do not form a hierarchical psychological indicator output, limiting the relevance of training recommendations. SUMMARY

[0005] The present application provides a pilot competence dynamic evaluation method, system, device and storage medium based on multi-modal data to solve the problems of low precision in pilot competence evaluation and poor relevance in training guidance in the prior art.

[0006] To solve the above technical problems, in a first aspect, the present application provides a pilot competence dynamic evaluation method based on multi-modal data, comprising:

[0007] Collecting flight control data, physiological signal data, psychological test data, subjective scale data and core competence information defined by the International Civil Aviation Organization when the pilot operates the flight target;

[0008] Using a time synchronization algorithm to align the flight control data, physiological signal data and psychological test data, obtaining fused multi-modal data;

[0009] Based on a hidden Markov model, analyzing the fused multi-modal data to identify attention fixation patterns;

[0010] fusing the subjective scale data and the physiological entropy data in the physiological signal data to construct a workload index;

[0011] inputting the fused multi-modal data, the attention fixation pattern, the workload index, and the core competency information into a pre-trained competency evaluation model, outputting a pilot competency evaluation result by the competency evaluation model, the pilot competency evaluation result including three-level psychological evaluation indexes of a basic ability layer, a dynamic performance layer, and a risk early warning layer;

[0012] generating a personalized training report based on the pilot competency evaluation result.

[0013] Optionally, the inputting the fused multi-modal data, the attention fixation pattern, the workload index, and the core competency information into a pre-trained competency evaluation model, outputting a pilot competency evaluation result by the competency evaluation model, includes:

[0014] inputting the fused multi-modal data, the attention fixation pattern, the workload index, and the core competency information into a pre-trained competency evaluation model, establishing, by a mapping module of the competency evaluation model, the core competency information as an evaluation benchmark, establishing a first index mapping relationship between the evaluation benchmark and the fused multi-modal data, a second index mapping relationship between the evaluation benchmark and the attention fixation pattern, and a third index mapping relationship between the evaluation benchmark and the workload index;

[0015] performing pattern recognition and classification based on the first index mapping relationship by a pattern classifier in the competency evaluation model to obtain an operation mode classification result;

[0016] performing state prediction based on the second index mapping relationship by a time series predictor in the competency evaluation model to obtain a state prediction result, and performing load prediction based on the third index mapping relationship to obtain a load prediction result;

[0017] performing weighted fusion on the classification result, the state prediction result, and the load prediction result to generate a pilot competency evaluation result.

[0018] Optionally, the performing weighted fusion on the classification result, the state prediction result, and the load prediction result to generate a pilot competency evaluation result includes:

[0019] obtaining a set of weight coefficients of each evaluation index corresponding to a flight phase of a flight target;

[0020] According to the weight coefficient set, a first weighted calculation is performed on the classification result to obtain a technical ability score;

[0021] According to the weight coefficient set, a second weighted calculation is performed on the state prediction result to obtain an attention state score;

[0022] According to the weight coefficient set, a third weighted calculation is performed on the load prediction result to obtain a working load score;

[0023] The technical ability score, the attention state score, and the working load score are input into a preset comprehensive evaluation function to generate a basic ability layer evaluation value, a dynamic performance layer evaluation value, and a risk warning layer evaluation value;

[0024] According to a preset grade mapping rule, the basic ability layer evaluation value, the dynamic performance layer evaluation value, and the risk warning layer evaluation value are converted into corresponding psychological evaluation indicators;

[0025] Based on each psychological evaluation indicator, a pilot competence evaluation result is generated.

[0026] Optionally, the competence evaluation model is constructed, including:

[0027] An initial training sample set is obtained, and each initial sample in the training sample set includes historical fusion multi-modal data, a historical attention fixation mode, a historical working load index, historical core competence information, and a historical pilot competence evaluation result;

[0028] An analytic hierarchy process is used to determine the index weight priority of each modal data in the initial sample in different flight stages of a flight target to generate a target sample;

[0029] Based on a random forest classifier and a long short-term memory neural network, an initial evaluation model is constructed;

[0030] Based on the target sample, the initial evaluation model is trained until a preset training stop condition is met to obtain the competence evaluation model.

[0031] Optionally, the analytic hierarchy process is used to determine the index weight priority of each modal data in the initial sample in different flight stages of a flight target to generate a target sample, including:

[0032] A hierarchical structure model is constructed, a target layer of the hierarchical structure model is a flight stage competence evaluation result, a criterion layer includes the fusion multi-modal data, the attention fixation mode, the working load index, and the core competence information, and a scheme layer is different flight stages;

[0033] constructing a first judgment matrix of each element of the criterion layer to the target layer, and a second judgment matrix of each element of the scheme layer to each element of the criterion layer;

[0034] calculating a maximum eigenvalue and a corresponding eigenvector of each judgment matrix, and calculating a consistency ratio based on the maximum eigenvalue;

[0035] when the consistency ratio is less than a preset ratio threshold, performing normalization processing on the eigenvector to obtain an effective weight vector of each level;

[0036] extracting a scheme layer weight vector and a criterion layer weight vector from the effective weight vector;

[0037] extracting a corresponding stage weight coefficient from the scheme layer weight vector according to a current flight phase of the flight target;

[0038] combining and calculating the stage weight coefficient and the criterion layer weight vector to generate an index weight priority of each modality data in a specific flight phase;

[0039] performing weighted processing on a value of corresponding modality data in the initial sample according to the index weight priority to generate a target sample.

[0040] Optionally, the fusion multi-modal data is analyzed based on a hidden Markov model to identify an attention fixation mode, including:

[0041] extracting a visual attention point sequence and an operation response sequence from the fusion multi-modal data;

[0042] inputting the visual attention point sequence and the operation response sequence as an observation sequence into a hidden Markov model, and calculating a probability of being in each hidden state at each time under the observation sequence by a forward-backward algorithm in the hidden Markov model, the hidden Markov model including three hidden states of a dispersion state, a concentration state and a fixation state;

[0043] when the probability of the concentration state or the fixation state exceeds a preset probability threshold, determining that an attention fixation mode occurs.

[0044] Optionally, the individualized training report is generated based on the pilot competency evaluation result, including:

[0045] extracting a weak link indicator from the pilot competency evaluation result;

[0046] matching the weak link indicator with a flight scene in a preset historical flight database to determine a target scene that needs to be strengthened;

[0047] reconstruct an attention allocation waveform and a cognitive load curve in the target scene according to the time series data of the attention fixation mode and the time series data of the workload index;

[0048] identify an attention abnormal period satisfying a preset first abnormality identification condition from the reconstructed attention allocation waveform, and identify a cognitive load overrun period satisfying a preset second abnormality identification condition from the reconstructed cognitive load curve;

[0049] label an operation action corresponding to the attention abnormal period and the cognitive load overrun period as a to-be-improved operation item;

[0050] generate a personalized training report according to the target scene, the to-be-improved operation item, and an expected training target.

[0051] In a second aspect, the present application provides a pilot competence dynamic evaluation system based on multi-modal data, comprising:

[0052] The acquisition module is configured to acquire flight control data, physiological signal data, psychological test data, subjective scale data, and core competence information defined by the International Civil Aviation Organization when the pilot operates a flight target.

[0053] The alignment module is configured to perform alignment processing on the flight control data, the physiological signal data, and the psychological test data by using a time synchronization algorithm to obtain fused multi-modal data.

[0054] The analysis module is configured to analyze the fused multi-modal data based on a hidden Markov model to identify an attention fixation mode.

[0055] The fusion module is configured to fuse the subjective scale data and physiological entropy value data in the physiological signal data to construct a workload index.

[0056] The input module is configured to input the fused multi-modal data, the attention fixation mode, the workload index, and the core competence information into a pre-trained competence evaluation model, and output a pilot competence evaluation result by the competence evaluation model, wherein the pilot competence evaluation result includes three-level psychological evaluation indexes of a basic ability layer, a dynamic performance layer, and a risk warning layer.

[0057] The generation module is configured to generate a personalized training report based on the pilot competence evaluation result.

[0058] In a third aspect, the present application provides an electronic device, comprising:

[0059] The memory is configured to store a computer program.

[0060] The processor is configured to execute the computer program to implement the steps of the method for dynamically evaluating pilot competence based on multi-modal data according to the first aspect.

[0061] In a fourth aspect, the present application provides a computer readable storage medium, wherein a computer program is stored in the computer readable storage medium, and the computer program is configured to implement the steps of the method for dynamically evaluating pilot competence based on multi-modal data according to the first aspect when executed by a processor.

[0062] The technical scheme provided by the present application has the following beneficial effects:

[0063] The present application realizes comprehensive collection of flight control, physiological state, psychological characteristics and industry standard information, provides complete data basis for comprehensive evaluation. The time synchronization algorithm is used to solve the time sequence difference of multi-source data, form the unified fusion data in space and time, and ensure the accuracy of subsequent analysis. The hidden Markov model is used to dynamically analyze the attention state of the pilot, and the abnormal attention allocation behavior is accurately captured. The subjective scale and objective physiological entropy value are fused to form a comprehensive index of quantitative cognitive load, and the reliability of the workload evaluation is enhanced. The machine learning model is used to integrate multi-dimensional features, output hierarchical psychological evaluation indexes, and realize fine diagnosis of competence. The training scheme is automatically generated based on the evaluation result, and the accuracy and efficiency of flight training are improved.

[0064] Further, the present application also establishes a mapping relationship between multi-dimensional indexes, respectively identifies, classifies and predicts the operation mode, attention state and workload, and finally generates a comprehensive evaluation result through weighted fusion.

[0065] Moreover, the scheme realizes organic fusion and dynamic analysis of multi-source evaluation indexes, can comprehensively reflect the operation characteristics, cognitive state and load level of the pilot, and improves the systematicness and accuracy of the competence evaluation.

[0066] These aspects or other aspects of the present application will be more apparent in the following description of the embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0067] In order to more clearly illustrate the technical scheme in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiment or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0068] Figure 1 A flowchart of a method for dynamically evaluating pilot competence based on multi-modal data provided by the present application is shown in the following figure:

[0069] Figure 2 A specific implementation schematic diagram of a pilot competence dynamic evaluation method based on multi-modal data provided by an embodiment of the present application is shown in FIG. 1.

[0070] Figure 3 Another specific implementation schematic diagram of a pilot competence dynamic evaluation method based on multi-modal data provided by an embodiment of the present application is shown in FIG. 2.

[0071] Figure 4 A structural schematic diagram of a pilot competence dynamic evaluation system based on multi-modal data provided by an embodiment of the present application is shown in FIG. 3. DETAILED DESCRIPTION

[0072] In order to make the personnel in the technical field better understand the present application, the present application will be further described in detail below in combination with the drawings and specific embodiments. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by the person skilled in the art without making creative efforts belong to the scope of protection of the present application.

[0073] The core of the present application is to provide a pilot competence dynamic evaluation method based on multi-modal data, and a flowchart of a specific implementation of the method is shown in FIG. 1. Figure 1 The method comprises the following steps.

[0074] Step 101: Collecting flight control data, physiological signal data, psychological test data, subjective scale data and core competence information defined by the International Civil Aviation Organization when the pilot operates the flight target.

[0075] In step 101, the flight control data refers to the control instruction parameters generated by the pilot during the operation process and the external environment information, specifically including simulation machine control parameters such as pitch angle deviation rate, throttle response delay and other indicators reflecting operation accuracy, and external environment data such as weather visibility, air traffic control instruction density and other situational factors affecting operation difficulty. Exemplarily, the flight control data also includes pitch and roll inputs of the control stick, rudder control of the footrest, thrust adjustment of the throttle lever, landing gear handle operation, ground steering control of the turn handwheel, and state adjustment information of various system panel buttons and knobs, comprehensively reflecting the pilot's comprehensive operation behavior on the aircraft. The physiological signal data is the pilot's physiological response information collected by the biological sensor, including the gaze hotspot distribution and saccade path efficiency recorded by the eye tracker to reflect visual attention allocation, the skin conductance response and heart rate variability monitored by the wearable device to represent autonomic nervous excitability, and the brain function indicators such as frontal lobe theta wave (related to cognitive load) and alpha wave (related to attention concentration) extracted by the electroencephalogram device. Exemplarily, the physiological signal data also includes electroencephalogram signals reflecting brain activity state, electrocardiogram signals reflecting heart function, respiration signals reflecting respiration rhythm, pulse signals reflecting cardiovascular state, skin conductance signals reflecting skin electrical activity, electromyogram signals reflecting muscle tension, eye movement signals reflecting visual attention allocation, pupil diameter changes reflecting cognitive load, facial expression features reflecting emotional state, and voice features reflecting psychological stress level. The psychological assessment data is the quantitative results of cognitive ability obtained through standardized psychological paradigms, including basic ability tests such as attention allocation scores of attention network tests, emotional interference effect values of emotional Stroop tasks, and performance indicators such as decision delay time and accuracy in special situation handling scenarios (taking engine failure as an example). The subjective scale data is the load evaluation scale filled in by the pilot according to his own feelings, such as the multi-dimensional subjective scores of NASA-TLX scale. The core competence information is the ability framework that the pilot must have defined by the International Civil Aviation Organization, including standardized ability dimensions such as situational awareness, decision-making ability, and communication and cooperation.

[0076] In the embodiments of the present application, the data acquisition system deployed on the flight simulator records the pilot's control actions in real time, and the electrocardiogram, skin conductance and other signals are collected through the wearable physiological monitoring device. Standardized psychological tests are conducted before the start of the flight task to obtain basic cognitive ability data, and the pilot is guided to fill in the subjective load scale after the task is completed. Finally, the standardized ability requirement document published by the International Civil Aviation Organization is integrated to form a complete multi-source data set.

[0077] For example, in a certain airline pilot training center, when the pilot operates the B737 simulator for approach and landing training, the system collects 12 groups of pitch control data, 10 groups of roll control data, and 8 groups of throttle position data; through the smart bracelet, 1500 sample points of heart rate variability data and 1200 sample points of skin electricity reaction data are collected; before training, the attention network test is completed to obtain an attention allocation score of 85 points, and the emotional Stroop test is completed to obtain an emotional stability score of 78 points; after training, the subjective scale of NASA-TLX is filled out to obtain a subjective load score of 70 points; and 9 core competency items in the International Civil Aviation Organization (ICAO) document are integrated to form a multi-source data set of this training.

[0078] Step 102: Aligning the flight control data, the physiological signal data, and the psychological test data by using a time synchronization algorithm to obtain fused multi-modal data.

[0079] In step 102, the time synchronization algorithm is a calculation method for time axis alignment of multi-source data sequences with different sampling speeds, and different data with different sampling frequencies are made to have the same time stamp by interpolation or resampling. The fused multi-modal data refers to a multi-dimensional data set with a unified time stamp formed after time alignment, wherein each time point contains synchronized control, physiological, and psychological data.

[0080] In the embodiment of the present application, first, the time stamp information of the collected multi-source data is extracted, the highest sampling rate is determined as the reference frequency, the resampling is performed on other data sequences by using the cubic spline interpolation method, so that all data streams have the same time interval and point number, and finally the aligned data is integrated into a multi-dimensional matrix form of the fused data set according to the time stamp.

[0081] For example, the control data (sampling rate 10 Hz), physiological data (sampling rate 100 Hz), and psychological data (sampling rate 1 Hz) collected in step 101 are time-synchronized, 100 Hz is used as the reference frequency, linear interpolation is used to increase the control data from 10 Hz to 100 Hz, and spline interpolation is used to increase the psychological data from 1 Hz to 100 Hz, and finally a fused multi-modal data matrix containing 5000 time points (corresponding to a 50-second training process) and 28 dimensions per time point is generated.

[0082] Step 103: Analyzing the fused multi-modal data based on a hidden Markov model to identify an attention fixation mode.

[0083] In step 103, the hidden Markov model is a probability model describing the generation of an observation sequence by a hidden state sequence, where the hidden state represents an unobservable cognitive state (dispersion, concentration, fixation), and the observation sequence refers to the measurable sequence of eye movements, operations, etc. extracted from the fusion data. The attention fixation mode refers to an abnormal state in which the pilot's attention is excessively concentrated on a specific interface element and lasts for too long.

[0084] In the embodiments of the present application, the eye movement fixation point coordinate sequence and operation response delay sequence are extracted from the fusion data as observation inputs, and the forward-backward algorithm is used to calculate the probability of being in three hidden states at each time. When the probability of concentration or fixation state continues to exceed the threshold value, it is marked as an attention fixation event, and its start and end time and intensity value are recorded. The specific implementation process is as follows: first, the sample entropy of heart rate variability and the approximate entropy of pupil diameter fluctuation are extracted from the physiological signal data as objective physiological entropy values, and the multi-dimensional scores in the subjective scale data are standardized; then, a regression model is trained based on historical data in high-pressure scenarios such as adverse weather approach, and a mapping relationship between physiological entropy values and subjective scores is established; finally, the standardized subjective scores and physiological entropy values are combined through a weighted fusion algorithm, where the weight coefficient is dynamically adjusted through an entropy value analysis model, and when the fusion value exceeds the preset critical threshold, a situational awareness degradation warning is triggered, thereby generating a workload index that can reflect the cognitive load level in real time.

[0085] For example, the gaze instrument panel duration ratio sequence and operation response delay sequence are extracted from the fusion data of step 102, and input into the trained hidden Markov model (transition probability matrix A=[[0.7, 0.2, 0.1], [0.3, 0.6, 0.1], [0.2, 0.2, 0.6]], emission probability matrix B is Gaussian distribution), and the calculation result is that the concentration state probability at 15-18 seconds is greater than 0.85, which is determined as an attention fixation mode, with a duration of 3 seconds and an average probability of 0.88.

[0086] Step 104: fuse the subjective scale data and the physiological entropy data in the physiological signal data to construct a workload index.

[0087] In step 104, the physiological entropy value data is a complexity index calculated from the physiological signal, such as heart rate variability sample entropy, skin electric reaction approximate entropy, reflecting the degree of disorder of the autonomic nervous system, wherein the complexity index is a nonlinear characteristic parameter obtained by quantitatively analyzing multi-source physiological signals through information entropy theory and signal processing method, and the calculation process includes: using differential entropy and power spectral density method to extract energy distribution characteristics of α, β, θ frequency band for electroencephalogram signal, and using P300 potential amplitude variation characteristics for memory-related tasks; using signal decomposition methods such as variational mode decomposition to extract intrinsic mode for electroencephalogram, electrocardiogram, skin electricity, pupil diameter, eye movement scan video frequency, blink frequency and other signals; extracting local texture features, shape features and wavelet transform features for expression data, and extracting prosody features, spectral features and voice quality features for speech data; finally, the normalized multi-modal features are fused by machine learning methods such as convolutional neural network, graph neural network and Transformer to generate two-dimensional image feature vectors representing the complexity of physiological state, which comprehensively reflects the nonlinear dynamic characteristics and information processing efficiency of the physiological system. The workload index is a standardized load quantitative value formed by combining subjective scale scores and objective physiological entropy values.

[0088] In the embodiments of the present application, sample entropy is calculated for heart rate variability signal, approximate entropy is calculated for skin electric response signal, the two entropy values are z-score standardized and then weighted and fused, and then linearly combined with the standardized subjective scale score, and finally mapped to the 0-100 interval by sigmoid function to form the workload index. The specific process is to quantitatively process the subjective scale data to obtain a subjective load score; extract physiological entropy value data from the physiological signal data, the physiological entropy value data including a heart rate sequence and a skin electric response signal sequence; calculate the sample entropy value of the heart rate sequence as a first physiological entropy value, and calculate the sample entropy value of the skin electric response signal sequence as a second physiological entropy value; linearly weight and fuse the subjective load score, the first physiological entropy value and the second physiological entropy value; and interval map and standardize the fused value to generate a workload index.

[0089] For example, the sample entropy value of the heart rate variability data in the calculation step 101 is 1.32, the approximate entropy value of the skin electric response data is 0.87, and the objective load value is 0.68 after standardization; the subjective scale score is 70, and the standardized value is 0.72; the fusion value is 0.696 according to the weights 0.6 (objective) and 0.4 (subjective); and the workload index is 82.3, wherein the x fusion value is 0.696, according to the formula

[0090] ​Step 105: inputting the fusion multi-modal data, the attention fixation mode, the workload index, and the core competency information into a pre-trained competency evaluation model, outputting a pilot competency evaluation result by the competency evaluation model, the pilot competency evaluation result including three-level psychological evaluation indexes of a basic ability layer, a dynamic performance layer, and a risk early warning layer.

[0091] In step 105, the pre-trained competency evaluation model refers to a multi-input and multi-output prediction model trained by a method of machine learning such as a convolutional neural network (CNN), a graph neural network (GNN), a long short-term memory network (LSTM), and the like, which inputs include multi-modal features, attention modes, load indexes, and ability standards, and outputs three-level evaluation indexes. Exemplarily, the competency evaluation model further includes dimensions, different physiological data used in different dimensions, and comprehensive evaluation by integrating physiological data and operation data in different dimensions, specifically including attention dimensions using pupil diameter, brain electrical alpha / theta waves, heart rate variability, and flight control data, memory dimensions using flight control data, voice features, eye movement patterns, brain electrical theta waves, and P300 potentials, cognitive dimensions using flight control data, brain electrical theta waves, and heart rate variability, motion perception dimensions using brain electrical alpha waves, flight control data, eye movement saccade area detection, and electromyographic signals, risk perception dimensions using heart rate variability, skin electric response, eye movement features, pupil diameter, and brain electrical beta / theta waves, decision-making dimensions using brain electrical theta waves, P300 potentials, pupil diameter, and heart rate variability, stress dimensions using skin electric response, heart rate variability, brain electrical alpha / beta waves, and respiratory waveform, emotional dimensions using heart rate variability, brain electrical alpha waves, pupil diameter, respiratory features, voice signals, and facial expression data, and precise evaluation of each dimension ability is realized by multi-modal data fusion analysis. The basic ability layer evaluates static psychological characteristics, the dynamic performance layer evaluates real-time operation efficiency, and the risk early warning layer evaluates potential risk levels.

[0092] In the embodiments of the present application, the fusion data, the attention mode, the load index, and the ability standard are input into the trained random forest + long short-term memory network (LSTM) hybrid model. First, the operation mode type is identified by the random forest classifier, then the load and attention trend are predicted by the LSTM network, and finally all outputs are dynamically weighted and fused according to the flight phase to generate three-level evaluation values.

[0093] For example, the fusion data (28 dimensions x 5000 points) obtained by the preceding step, the attention fixation mode (duration 3 seconds, intensity 0.88), the workload index 82.3, and the ICAO capability standard input model are input into the random forest to output the operation mode risk probability 0.76, the LSTM outputs the attention deterioration trend 0.62 and the load rising trend 0.85, which are weighted according to the approach phase weight (operation 0.5, attention 0.3, load 0.2) to obtain 0.76x0.5+0.62x0.3+0.85x0.2=0.756, which is mapped to the basic ability layer 78 points, the dynamic performance layer 65 points, and the risk warning layer 82 points.

[0094] Step 106: generating a personalized training report based on the pilot competency evaluation result.

[0095] In step 106, the personalized training report is a targeted training program customized according to the weaknesses in the evaluation result, including the ability dimensions that need to be strengthened, the training subjects, the training duration, and the expected targets.

[0096] In the embodiments of the present application, the ability indicators below the threshold value are extracted from the evaluation result, the training case library is queried to match the training program of a similar scene, the operation actions that need to be improved are marked in combination with the specific abnormal period of the attention mode and the load curve, and finally a text report containing the training subject, the training intensity, and the improvement target is generated.

[0097] For example, according to the dynamic performance layer 65 points (threshold value 70) and the risk warning layer 82 points (threshold value 75) in the output of step 105, it is determined that the instrument scanning and load management capabilities need to be improved, the "adverse weather approach" training scene in the database is matched, and the throttle operation delay problem corresponding to the 15-18 second attention fixation period is combined to generate a report containing 5 hours of instrument identification training and 3 hours of load control training, and the expected target is to improve the dynamic performance layer to more than 70 points.

[0098] The method realizes comprehensive and dynamic monitoring of the cognitive state, operation performance, and psychological load of the pilot through synchronous acquisition and fusion processing of multi-modal data; accurately identifies the attention abnormal mode and the workload change trend through the combination and application of the hidden Markov model and the machine learning algorithm; and finally generates three-level evaluation indicators and a personalized training report, which provides a precise and targeted scientific training basis for the improvement of the pilot's ability, and improves the quality and efficiency of aviation training.

[0099] In order to solve the problem of lack of dynamic fusion mechanism for multi-dimensional evaluation indicators in the prior art, in some embodiments, step 105: the fusion multi-modal data, the attention fixation pattern, the workload index, and the core competency information are input into a pre-trained competency evaluation model, and a pilot competency evaluation result is output through the competency evaluation model, as shown in Figure 2

[0100] Step 201: input the fusion multi-modal data, the attention fixation pattern, the workload index, and the core competency information into a pre-trained competency evaluation model, and through a mapping module of the competency evaluation model, the core competency information is taken as an evaluation benchmark, and the evaluation benchmark is respectively mapped with the fusion multi-modal data to establish a first index mapping relationship, mapped with the attention fixation pattern to establish a second index mapping relationship, and mapped with the workload index to establish a third index mapping relationship.

[0101] In step 201, the mapping module is a component responsible for establishing an index correlation relationship in the competency evaluation model, which takes the core competency information defined by the International Civil Aviation Organization as a benchmark framework. The first index mapping relationship refers to the establishment of a quantitative correlation between the 9 core competencies defined by the International Civil Aviation Organization (including situational awareness, decision-making ability, communication ability, etc.) and the operation characteristic indicators (such as pitch angle deviation rate, throttle response delay, etc.) in the fusion multi-modal data through mathematical modeling. Through the calculation of the similarity measure of the multi-dimensional feature vector and each competency dimension, a numerical correlation model reflecting the matching degree of operation behavior and standard ability framework is formed. The second index mapping relationship refers to the establishment of a dynamic correlation between the characteristic parameters of the attention fixation pattern (such as duration, intensity index, and occurrence frequency) and the cognitive ability dimension (such as situational awareness and work management) in the core competency. Through a state transition probability model, the influence degree of the attention abnormal pattern on a specific ability dimension is quantified. The third index mapping relationship refers to the establishment of a functional relationship between the numerical change of the workload index and the performance dimension (such as decision-making efficiency and operation precision) in the core competency. Through a regression analysis model, the nonlinear mapping rule between the load level and the ability performance is described, which is used to evaluate the stability of the competency under different load states.

[0102] In the embodiments of the present application, the mapping module first analyzes the text description of the core competency information, extracts the key ability dimension as a benchmark vector, then calculates the cosine similarity of the fusion multi-modal data feature vector and the benchmark vector to form the first index mapping relationship, calculates the Euclidean distance of the attention fixation pattern duration and intensity index and the benchmark behavior dimension to form the second index mapping relationship, and calculates the relative deviation of the workload index value and the benchmark load dimension to form the third index mapping relationship. Finally, three mapping relationship matrices are output. ​

[0103] Step 202: performing pattern recognition and classification based on the first index mapping relationship by a pattern classifier in the competency evaluation model to obtain an operation pattern classification result.

[0104] In step 202, the pattern classifier is a classification component constructed based on a machine learning algorithm in the competency evaluation model, which receives the first index mapping relationship matrix as input, recognizes the operation behavior pattern through multiple layers of decision rules, outputs an operation pattern classification result containing three types of operation patterns, i.e., normal operation, risky operation and critical operation, and gives the probability distribution of each category. The operation pattern classification result refers to the classification label and its probability distribution output by the machine learning algorithm after recognizing the pilot operation behavior characteristics, which is used to judge whether the current operation belongs to the normal operation, risky operation or critical operation mode, and reflects the operation standardization and risk tendency of the pilot.

[0105] In the embodiments of the present application, the pattern classifier adopts an ensemble learning algorithm, first performs feature dimension reduction processing on the first index mapping relationship matrix, then parallelly calculates the likelihood of different operation patterns through multiple decision tree models, and finally integrates the outputs of all decision trees through a voting mechanism to obtain the final operation pattern classification result and the corresponding confidence score.

[0106] Step 203: performing state prediction based on the second index mapping relationship by a time series predictor in the competency evaluation model to obtain a state prediction result, and performing load prediction based on the third index mapping relationship to obtain a load prediction result.

[0107] In step 203, the time series predictor is a prediction component in the competence evaluation model specially processing time series data, which respectively receives the second index mapping relationship and the third index mapping relationship as input, learns the historical sequence change law through a recurrent neural network structure, and outputs the state prediction result of the attention state in the future time period and the load prediction result of the workload change trend. The state prediction result refers to the prediction value of the attention state change trend in the future time period based on the time series data, which is used to warn the state abnormalities such as attention fixation, dispersion or transfer, and reflects the dynamic evolution law of the cognitive state of the pilot. The load prediction result refers to the estimated value of the future workload level by analyzing the historical load data, which is used to predict the rising or falling trend of the cognitive load, and reflects the change of the psychological resource consumption of the pilot. The specific implementation process is as follows: the time series predictor first extracts the historical attention pattern features based on the attention state time series data in the second index mapping relationship, learns the state transition law through the gating mechanism, predicts the state probability distribution of attention fixation, concentration or dispersion in the future time period, and generates a state warning result when the attention loss probability of consecutive multiple time steps exceeds the threshold; at the same time, based on the load time series data in the third index mapping relationship, the load change trend is analyzed using a sliding window, combined with the eye movement hotspot drift rate feature (when the gaze point deviates from the key instrument area for 3 seconds, an abnormal mark is triggered), the rising or falling trajectory of the workload is predicted, and finally the comprehensive prediction result containing attention state warning and load overrun warning is output.

[0108] In the embodiments of the present application, the time series predictor adopts a recurrent neural network with a gating mechanism, first divides the second index mapping relationship sequence into a sliding window, learns the attention state evolution law through hidden state transmission, and outputs the state prediction value of the future time steps; at the same time, the third index mapping relationship sequence is decomposed in time series, the trend feature and the period feature are extracted, and the prediction trajectory of the workload is output.

[0109] Step 204: Weighted fusion of classification results, state prediction results and load prediction results to generate pilot competence evaluation results.

[0110] In step 204, the weighted fusion is a process of integrating multi-dimensional prediction results through a dynamic weight distribution algorithm, which assigns different weight coefficients to the operation mode classification result, the state prediction result and the load prediction result according to the characteristics of the current flight phase, and generates three-level evaluation results including basic ability score, dynamic performance score and risk warning score through linear weighted calculation.

[0111] In this embodiment, the weighted fusion module first reads the weight combination corresponding to the current flight phase from the preset weight configuration library, then standardizes the probability value of the operation mode classification result, the value of the state prediction result, and the value of the load prediction result, respectively, and finally performs weighted summation according to the weight coefficients, and maps it to the standard scoring interval through the activation function to generate the final three-level pilot competency assessment result.

[0112] Here is a specific example:

[0113] In the approach and landing training scenario at the A Airlines pilot training center, the 28-dimensional × 5000 time points of the fused multimodal data obtained in the aforementioned embodiment, the duration of attention fixation mode (3 seconds) and intensity (0.88), the workload index (82.3), and the ICAO core competency information were input into a pre-trained competency assessment model. First, the mapping module used the nine core competencies as assessment benchmarks. The cosine similarity between the pitch angle deviation rate sequence and the situational awareness dimension in the fused multimodal data was calculated, yielding a first index mapping relationship value of 0.85. The calculation formula is as follows: ,in Indicates the first Pitch angle deviation rate values ​​at each time point (unit: degrees / second). The first vector representing the baseline vector of the context awareness dimension Several numerical values; the Euclidean distance between the intensity of attention fixation mode and the decision-making ability dimension yields a second index mapping value of 0.35, calculated using the following formula: ,in The first feature vector representing attention fixation One parameter, The first vector representing the baseline vector of decision-making ability dimension The numerical values; the relative deviation between the workload index and the work management dimension yields a third indicator mapping value of 0.18, calculated using the following formula: Where R represents the current workload index of 82.3 (dimensionless), and S represents the baseline value of the work management dimension of 70 (dimensionless). Subsequently, a pattern classifier performs pattern recognition based on the first indicator mapping value of 0.85, outputting an operation mode classification result of a high-risk landing mode probability of 0.76. A time series predictor predicts a deterioration trend in attention state of 0.62 in the next 10 seconds based on the second indicator mapping value of 0.35, and a load increase trend value of 0.85 based on the third indicator mapping value of 0.18. Finally, the three results are weighted and fused, calculated as follows: Comprehensive Score = Operation Mode Weight 0.5 × Classification Result 0.76 + Attention Weight 0.3 × State Prediction Result 0.62 + Load Weight 0.2 × Load Prediction Result 0.85. Substituting the values... The value is converted into a three-level evaluation result through a linear mapping function, wherein the basic ability layer score = 50 + 50*0.736 = 86.8, the dynamic performance layer score = 30 + 30*0.736 = 52.08, and the risk warning layer score = 20 + 20*0.736 = 34.72, and finally a pilot competence evaluation result containing the three values is generated.

[0114] In the embodiments of the present application, the accurate conversion from multi-source data to evaluation results is realized through a multi-stage dynamic fusion mechanism, a complete technical chain of index correlation analysis, pattern recognition, trend prediction and decision fusion is established, and the accuracy and practicability of the evaluation results are improved, thereby providing a scientific basis for pilot competence training.

[0115] To solve the problem of insufficient fusion precision of multi-dimensional evaluation results, in some embodiments, step 204: the classification result, the state prediction result and the load prediction result are weighted and fused to generate a pilot competence evaluation result, as shown in Figure 3 , which includes:

[0116] Step 301: obtaining a weight coefficient set of each evaluation index corresponding to the flight phase of the flight target.

[0117] In step 301, the weight coefficient set refers to the index importance configuration parameter preset for different flight phases, which contains the weight distribution proportion of the three types of indexes of operation mode, attention state and workload. The set is determined by expert evaluation and historical data analysis together, and reflects the relative importance of each evaluation index in a specific flight phase.

[0118] In the embodiments of the present application, the system first identifies the current flight phase type, then queries the weight coefficient combination corresponding to the phase from the preset weight configuration database, and the combination contains three decimal values corresponding to the three types of evaluation indexes, and finally loads the weight coefficient set into the memory for subsequent calculation.

[0119] Step 302: performing first weighted calculation on the classification result according to the weight coefficient set to obtain a technical ability score.

[0120] In step 302, the first weighted calculation refers to the process of weighting the classification result using the operation mode weight in the weight coefficient set, which multiplies the probability value of the operation mode classification result by the corresponding weight to obtain a quantitative score reflecting the technical ability level. The technical ability score refers to the quantitative value obtained by weighting the operation mode classification result, which reflects the operation standardization and technical proficiency of the pilot in a specific flight phase. The higher the score, the greater the operation risk or the more obvious the technical defects.

[0121] In the embodiment of the present application, the operation mode weight value is extracted from the weight coefficient set, the high-risk operation probability in the classification result is multiplied by the weight value, and a technical ability score is generated. The higher the score value is, the greater the operation risk is.

[0122] Step 303: According to the weight coefficient set, a second weighted calculation is performed on the state prediction result to obtain an attention state score.

[0123] In step 303, the second weighted calculation refers to a process of performing weighted processing on the state prediction result by using the attention state weight in the weight coefficient set. The attention state prediction value is multiplied by the corresponding weight to obtain a quantitative score reflecting the attention state. The attention state score refers to a quantitative value obtained by weighting the attention state prediction result, representing the rationality and stability of the pilot's attention distribution. The higher the score is, the more serious the attention abnormality is or the worse the attention management ability is.

[0124] In the embodiment of the present application, the attention state weight value is extracted from the weight coefficient set, the attention deterioration trend value in the state prediction result is multiplied by the weight value, and an attention state score is generated. The higher the score value is, the worse the attention state is.

[0125] Step 304: According to the weight coefficient set, a third weighted calculation is performed on the load prediction result to obtain a working load score.

[0126] In step 304, the third weighted calculation refers to a process of performing weighted processing on the load prediction result by using the working load weight in the weight coefficient set. The working load prediction value is multiplied by the corresponding weight to obtain a quantitative score reflecting the load level. The working load score refers to a quantitative value obtained by weighting the working load prediction result, representing the intensity and change trend of the pilot's cognitive load. The higher the score is, the greater the psychological pressure is or the weaker the load regulation ability is.

[0127] In the embodiment of the present application, the working load weight value is extracted from the weight coefficient set, the load rising trend value in the load prediction result is multiplied by the weight value, and a working load score is generated. The higher the score value is, the greater the load pressure is.

[0128] Step 305: The technical ability score, the attention state score, and the working load score are input into a preset comprehensive evaluation function to generate a basic ability layer evaluation value, a dynamic performance layer evaluation value, and a risk warning layer evaluation value.

[0129] In step 305, the comprehensive evaluation function refers to a mathematical function for converting the three types of weighted scores into three levels of evaluation values, which maps the technical ability score, attention state score and workload score to the evaluation value interval of the basic ability layer, dynamic performance layer and risk warning layer respectively through a linear combination formula. The basic ability layer evaluation value refers to the quantitative result of the basic psychological characteristics obtained by the comprehensive evaluation function, reflecting the inherent cognitive ability and emotional stability level of the pilot. The dynamic performance layer evaluation value refers to the quantitative result of the real-time operation efficiency obtained by the comprehensive evaluation function, embodying the actual operation performance and quality of the pilot in specific flight tasks. The risk warning layer evaluation value refers to the quantitative result of the potential risk obtained by the comprehensive evaluation function, indicating the risk degree of operation failure or state abnormality of the pilot in subsequent tasks.

[0130] In the embodiments of the present application, the technical ability score is input into the basic ability layer conversion function, the attention state score is input into the dynamic performance layer conversion function, and the workload score is input into the risk warning layer conversion function, and the corresponding three-level evaluation values are generated through three parallel linear calculation formulas.

[0131] Step 306: According to the preset grade mapping rule, the basic ability layer evaluation value, the dynamic performance layer evaluation value and the risk warning layer evaluation value are converted into corresponding psychological evaluation indexes.

[0132] In step 306, the grade mapping rule refers to the corresponding relationship table for converting numerical evaluation values into graded psychological evaluation indexes, which defines the mapping relationship between different numerical intervals and grade identifiers, and is used to discretize continuous numerical values into graded indexes with semantic interpretation. The psychological evaluation index refers to the graded evaluation result obtained by converting the numerical evaluation value through the grade mapping rule, which includes the semantic description of three dimensions of basic ability grade, dynamic performance grade and risk warning grade, providing intuitive reference for training improvement.

[0133] In the embodiments of the present application, the basic ability layer evaluation value, the dynamic performance layer evaluation value and the risk warning layer evaluation value are compared with the preset grade threshold value respectively, and the corresponding grade identifier is determined according to the interval to generate three-level psychological evaluation indexes.

[0134] Step 307: Based on each psychological evaluation index, a pilot competence evaluation result is generated.

[0135] In the embodiments of the present application, the three grade identifiers are combined according to the standard format, and the corresponding grade description text is added to generate a pilot competence evaluation result document containing complete evaluation conclusions.

[0136] The following is a specific example:

[0137] In the approach and landing training scene of the A airline pilot training center, based on the operation mode classification result 0.76, the state prediction result 0.62 and the load prediction result 0.85 obtained in the foregoing embodiment, first, the weight coefficient set corresponding to the approach stage is obtained as [0.5, 0.3, 0.2], wherein the three values respectively represent the weights of the operation mode, the attention state and the workload; according to the weight coefficient set, first weighting calculation is performed, using the operation mode weight 0.5 multiplied by the classification result 0.76 to obtain the technical ability score 0.38, the calculation formula is technical ability score = operation mode weight x classification result; second weighting calculation is performed, using the attention state weight 0.3 multiplied by the state prediction result 0.62 to obtain the attention state score 0.186; third weighting calculation is performed, using the workload weight 0.2 multiplied by the load prediction result 0.85 to obtain the workload score 0.17; input the three scores into the comprehensive evaluation function, wherein the basic ability layer evaluation value = 50 + 50 x technical ability score = 50 + 50 x 0.38 = 69, the dynamic performance layer evaluation value = 30 + 30 x attention state score = 30 + 30 x 0.186 = 35.58, and the risk warning layer evaluation value = 20 + 20 x workload score = 20 + 20 x 0.17 = 23.4; according to the threshold range 60-80 for good, 30-60 for improvement and 20-40 for low risk set in the grade mapping rule, the basic ability layer evaluation value 69 is mapped to the good grade, the dynamic performance layer evaluation value 35.58 is mapped to the improvement grade, and the risk warning layer evaluation value 23.4 is mapped to the low risk grade; finally, based on the three psychological evaluation indexes, a formal pilot competence evaluation result report containing grade description and specific improvement suggestions is generated.

[0138] In the embodiments of the present application, through the dynamic weight distribution and multi-level conversion mechanism, accurate mapping from the original prediction data to the structured evaluation result is realized, ensuring that the evaluation result not only retains numerical accuracy but also has grade interpretation, providing a clear and explicit improvement direction for pilot ability improvement.

[0139] In order to solve the problem of insufficient matching degree between model training data and real scene, in some embodiments, step 105: constructing a competence evaluation model, comprising:

[0140] Step 401: obtaining an initial training sample set, each initial sample in the training sample set comprising: historical fused multi-modal data, historical attention fixation mode, historical workload index, historical core competence information and historical pilot competence evaluation result.

[0141] In step 401, the historical fusion multi-modal data refers to a multi-source data set that has been time-synchronized in past training. The historical attention fixation pattern refers to a recognized attention abnormality state record. The historical workload index refers to a calculated load quantification value. The historical core competency information refers to a past version of the International Civil Aviation Organization standard. The historical pilot competency assessment result refers to a past generated evaluation conclusion containing three-level indicators.

[0142] In the embodiment of the present application, historical training records are extracted from the training database, each record containing all multi-modal data collected during a complete training period, attention patterns and workload indices obtained through analysis, competency standard documents used at the time, and finally generated assessment result reports, which together constitute an initial training sample set.

[0143] Step 402: Using the analytic hierarchy process, determine the index weight priority of each modality data in the initial sample in different flight stages of the flight target to generate a target sample.

[0144] In step 402, the index weight priority refers to the importance ranking of different data types in a specific flight stage determined through expert evaluation and mathematical calculation, which reflects the contribution difference of each modality data to the evaluation result, and is used for weighted processing of the original sample. The target sample refers to the weighted sample generated by applying weight priority to each modality data in the initial training sample through the analytic hierarchy process. It assigns appropriate weight coefficients to data of different importance, strengthens the influence of key features in the training process, and thus improves the sensitivity and learning efficiency of the model to important features.

[0145] In the embodiment of the present application, the analytic hierarchy process is used to construct a structure model containing target layer, criterion layer and scheme layer. A judgment matrix is established through expert scoring, the consistency ratio is calculated and verified, then the weight vector corresponding to each flight stage is extracted, and the different modality data in the initial sample is weighted to generate a target sample that strengthens important features.

[0146] Step 403: Based on the random forest classifier and the long short-term memory neural network, an initial assessment model is constructed.

[0147] In step 403, the initial assessment model refers to a machine learning model framework that has not been trained, which contains a tree ensemble algorithm for pattern classification and a recurrent neural network structure for time series prediction, and has the ability to process multi-dimensional input and output three-level evaluation results.

[0148] In the embodiments of the present application, the number of trees and the depth parameters of the random forest classifier are configured first, then the hidden layer structure and the number of nodes of the long short-term memory neural network are set, and finally the two model components are integrated to construct an initial model architecture that can simultaneously process classification and prediction tasks. The specific implementation process is as follows: first, the dynamic weight priorities of each evaluation index are determined for different flight stages using the analytic hierarchy process, and on this basis, a hybrid model architecture is constructed by integrating the random forest classifier and the long short-term memory neural network, wherein the random forest classifier is configured with multiple decision trees for identifying high-risk maneuvering modes such as landing bounce trends and other operation characteristics, and the long short-term memory neural network is designed with a gating mechanism for processing time series data such as electrodermal signals to predict the emotional stability decline trajectory. Finally, the outputs of the two types of models are integrated through a weighted fusion layer to form a multifunctional evaluation model that can simultaneously process classification and prediction tasks.

[0149] Step 404: Based on the target sample, the initial evaluation model is trained until the preset training stopping condition is met, and a competency evaluation model is obtained.

[0150] In step 404, the preset training stopping condition refers to the termination criteria set during the model training process, including the maximum number of iterations and the loss function threshold, etc. indicators, which are used to control the training process and stop at the appropriate time to avoid overfitting.

[0151] In the embodiments of the present application, the target sample is input into the initial evaluation model for forward propagation calculation to predict the output, the difference between the predicted result and the true label is compared to calculate the loss value, the model parameters are updated through the back propagation algorithm, and when the training round reaches the set value and the loss value is lower than the threshold, the training is stopped to obtain a competency evaluation model that can be put into use.

[0152] The following is a specific example:

[0153] In the model building process of the A airline pilot training center, first, an initial training sample set containing 1000 groups of historical training samples is obtained, each group of samples contains a 28-dimensional x 5000 time point matrix of historical fusion multi-modal data in the approach phase, a historical attention fixation mode record containing a duration of 3 seconds and an intensity of 0.88, a historical workload index of 82.3, historical core competence information, that is, ICAO standard documents, and historical pilot competence evaluation results containing basic ability layer 78 points, dynamic performance layer 65 points and risk early warning layer 82 points; the index weight priority of each modal data in the approach phase is determined by using the analytic hierarchy process, and the judgment matrix is calculated by expert scoring to obtain the weight of the control data 0.4, the weight of the physiological data 0.3, the weight of the attention data 0.2, and the weight of the load data 0.1. The data in the initial sample is weighted to generate a target sample according to these weights, wherein the control data is multiplied by 0.4, the physiological data is multiplied by 0.3, the attention data is multiplied by 0.2, and the load data is multiplied by 0.1. Based on the random forest classifier, 100 decision trees are configured to identify high-risk control patterns, and based on the long short-term memory neural network, 128 hidden units are configured to predict emotional stability trends, and an initial evaluation model is built. The initial evaluation model is trained using the target sample, and the maximum number of iterations is set to 200 rounds. When the loss function value drops to 0.05, the training is stopped, wherein L represents the loss function value, N represents the number of samples, represents the true label value of the th sample, represents the predicted value of the th sample, and finally a competence evaluation model with a verification set accuracy of 0.92 is obtained. The model can accurately output three-level evaluation indicators consistent with the historical evaluation results.

[0154] In the embodiments of the present application, by historical data weighting processing and hybrid model training, an evaluation model that can accurately reflect the characteristics of different flight phases is built, which not only retains historical experience but also integrates the importance difference of multi-dimensional features, and improves the accuracy and reliability of the model in actual application.

[0155] In order to solve the rationality problem of multi-modal data weight distribution, in some embodiments, step 402: the analytic hierarchy process is used to determine the index weight priority of each modal data in the initial sample in different flight phases of the flight target to generate a target sample, including:

[0156] Step 501: build a hierarchical structure model, and the target layer of the hierarchical structure model is the flight phase competence evaluation result, the criterion layer contains the fusion multi-modal data, the attention fixation mode, the workload index and the core competence information, and the scheme layer is different flight phases.

[0157] In step 501, the hierarchical structure model refers to a tree structure that decomposes a complex decision-making problem into a target layer, a criterion layer, and a scheme layer, wherein the target layer represents the final goal flight phase competency evaluation result to be achieved, the criterion layer includes multi-dimensional factor fusion multi-modal data, attention fixation mode, workload index, and core competency information that affect the target, and the scheme layer includes all possible flight phase options.

[0158] In the embodiment of the present application, firstly, it is determined that the evaluation target accurately reflects the flight phase competency level, then four criterion layer elements, i.e., multi-modal data comprehensiveness, attention mode characteristics, load level index, and capability standard compliance, are determined, and finally all flight phase types that need to be supported are listed to form a complete hierarchical structure model.

[0159] Step 502: constructing a first judgment matrix of each element of the criterion layer with respect to the target layer, and a second judgment matrix of each element of the scheme layer with respect to each element of the criterion layer.

[0160] In step 502, the first judgment matrix refers to an importance comparison matrix of each element of the criterion layer with respect to the target layer. The second judgment matrix refers to an importance comparison matrix of each flight phase of the scheme layer with respect to each element of the criterion layer, both of which are constructed in a pairwise comparison manner.

[0161] In the embodiment of the present application, a plurality of flight experts are invited to use the 1-9 scale method to perform pairwise importance comparison on the four elements of the criterion layer to generate the first judgment matrix, and to compare the importance of each flight phase with respect to each element of the criterion layer to generate a plurality of second judgment matrices.

[0162] Step 503: calculating the maximum eigenvalue and the corresponding eigenvector of each judgment matrix, and calculating a consistency ratio based on the maximum eigenvalue.

[0163] In step 503, the maximum eigenvalue refers to the principal eigenvalue of the judgment matrix. The eigenvector is a vector corresponding to the maximum eigenvalue. The consistency ratio is a ratio obtained by comparing the consistency index of the judgment matrix with the random consistency index, which is used to check the consistency degree of the judgment logic.

[0164] In the embodiment of the present application, the eigenvalue and the eigenvector of each judgment matrix are calculated, the maximum eigenvalue is selected and the corresponding consistency index is calculated, and then the consistency ratio value is obtained by comparing the consistency index with the random consistency index of the same order matrix.

[0165] Step 504: when the consistency ratio is less than a preset ratio threshold, performing normalization processing on the eigenvector to obtain an effective weight vector of each layer.

[0166] In step 504, the preset ratio threshold is an acceptable upper limit of consistency level set in advance. The effective weight vector is the normalized feature vector after passing the consistency test, representing the reasonable weight distribution of each element in the corresponding level.

[0167] In the embodiment of the present application, when the calculated consistency ratio is less than 0.1, the feature vector is normalized to make the sum of each component 1, and an effective weight vector reflecting the relative importance of each element is obtained.

[0168] Step 505: Extracting a scheme layer weight vector and a criterion layer weight vector from the effective weight vector.

[0169] In step 505, the scheme layer weight vector refers to the weight set of each flight phase in the scheme layer relative to the total target, and the criterion layer weight vector refers to the weight set of each element in the criterion layer relative to the total target, both of which are standardized weight values extracted from the effective weight vector.

[0170] In the embodiment of the present application, the scheme layer weight vector representing the weight of the flight phase and the criterion layer weight vector representing the weight of the criterion element are extracted from the weight vector passing the test, providing a basis for subsequent combination calculation.

[0171] Step 506: According to the current flight phase of the flight target, extracting a corresponding phase weight coefficient from the scheme layer weight vector.

[0172] In step 506, the phase weight coefficient refers to the specific weight value corresponding to the current processed flight phase in the scheme layer weight vector, reflecting the relative importance of the flight phase among all phases.

[0173] In the embodiment of the present application, according to the type of the current flight phase to be processed, the corresponding weight value is searched from the scheme layer weight vector as the importance coefficient of the phase.

[0174] Step 507: Combining the phase weight coefficient with the criterion layer weight vector to generate the index weight priority of each modality data in a specific flight phase.

[0175] In step 507, the combination calculation refers to the process of multiplying or weighting the phase weight coefficient with the criterion layer weight vector, and the generated result represents the optimal weight priority of each modal data in a specific flight phase. The specific flight phase refers to all flight phase types (such as take-off, climb, cruise, approach, landing, etc.) predefined in the analytic hierarchy process scheme layer, which is a static set containing all possible phases; while the "current flight phase" refers to a specific flight phase in which the flight target is currently located, which is an instance in the "specific flight phase" set; the two are in the relationship of abstract definition and specific instance, and the scheme layer weight vector contains the weight data of all "specific flight phases", and the corresponding specific weight value needs to be extracted from this vector according to the "current flight phase" during the combination calculation.

[0176] In the embodiment of the present application, the phase weight coefficient is multiplied by each component in the criterion layer weight vector in turn to obtain the combined weight value of each criterion element for the current flight phase, forming a complete weight priority sequence.

[0177] Step 508: According to the index weight priority, the numerical values of the corresponding modal data in the initial sample are weighted to generate a target sample.

[0178] In step 508, the weighting process refers to the process of coefficient adjustment of the numerical values of the corresponding modal data in the initial sample according to the weight priority, so as to generate an optimized target sample by emphasizing important features and weakening secondary features.

[0179] In the embodiment of the present application, the different types of data in the initial sample are multiplied by the corresponding weight coefficients according to the calculated weight priority, to generate a target sample set that emphasizes important features.

[0180] The following is a specific example:

[0181] In the model construction process of the pilot training center of A airline, the analytic hierarchy process is used to determine the index weight priority of each modal data in the approach phase. First, a hierarchical structure model is constructed, in which the target layer is the flight phase competence evaluation result, the criterion layer contains four elements of fusion multi-modal data, attention fixation mode, workload index and core competence information, and the scheme layer contains five flight phases including the approach phase; five flight experts are invited to construct the first judgment matrix M 1 [[1, 3, 5, 7], [1 / 3, 1, 4, 6], [1 / 5, 1 / 4, 1, 3], [1 / 7, 1 / 6, 1 / 3, 1]] represents the importance comparison of each element in the criterion layer to the target layer, and the second judgment matrix M 2[[1, 2, 4, 5], [1 / 2, 1, 3, 4], [1 / 4, 1 / 3, 1, 2], [1 / 5, 1 / 4, 1 / 2, 1]] represents the importance comparison of each element of the approach phase alignment criterion layer; the maximum eigenvalue of the first judgment matrix is calculated = 4.12 and the corresponding eigenvector V1 = [0.58, 0.28, 0.10, 0.04], wherein the maximum eigenvalue calculation formula is , and the eigenvector is obtained by solving ; the consistency index is calculated , wherein CI represents the consistency index, n represents the order of the judgment matrix, M represents the judgment matrix, the numerical value is substituted , the random consistency index RI = 0.89 is obtained by looking up the table, the consistency ratio , the numerical value is substituted , and 0.045 is less than the preset threshold value 0.1; the eigenvector V 1 is normalized to obtain the effective weight vector of the criterion layer = [0.58, 0.28, 0.10, 0.04]; the scheme layer weight vector W s = [0.46, 0.28, 0.16, 0.10] is also calculated from the second judgment matrix; according to the current flight phase being the approach phase, the corresponding phase weight coefficient 0.46 is extracted from W s ; the phase weight coefficient 0.46 is combined with the criterion layer weight vector to calculate the index weight priority of each modal data in the approach phase ; according to the weight priority, the values of the corresponding modal data in the initial sample are weighted, wherein the fusion multi-modal data is multiplied by 0.2668, the attention fixation mode data is multiplied by 0.1288, the workload index is multiplied by 0.046, and the core competence information is multiplied by 0.0184, and finally the target sample emphasizing important features is generated for subsequent model training.

[0182] In the embodiments of the present application, the weight priority of each modal data in different flight phases is scientifically determined by the analytic hierarchy process, the rationalization and standardization of data weighting are realized, the target sample with clear feature importance distinction is provided for model training, and the accuracy and reliability of the subsequent evaluation model are improved.

[0183] In order to solve the problem of insufficient attention state recognition accuracy, in some embodiments, step 103: the fusion multi-modal data is analyzed based on a hidden Markov model to identify the attention fixation mode, comprising:

[0184] Step 601: Extracting a visual attention point sequence and an operation response sequence from the fusion multi-modal data.

[0185] In step 601, the visual attention point sequence refers to the coordinate record of the pilot's gaze position changing over time during the driving process, containing screen coordinates and duration information of the gaze point. The operation response sequence refers to the timestamp record of the pilot operating the control device, containing parameters such as operation type and response delay time.

[0186] In the embodiments of the present application, the gaze point coordinate sequence collected by the eye tracker and the operation lever action time sequence are extracted from the fusion multi-modal data, and the two sequences are time-aligned to ensure that each time point contains corresponding visual attention points and operation response data.

[0187] Step 602: Inputting the visual attention point sequence and the operation response sequence as observation sequences into a hidden Markov model, and calculating the probability of being in each hidden state at each time under the observation sequence by using the forward-backward algorithm in the hidden Markov model, the hidden Markov model including three hidden states of dispersion state, concentration state and fixation state.

[0188] In step 602, the forward-backward algorithm is a probability calculation method in the hidden Markov model, which is used to calculate the probability of being in different hidden states at each time under the given observation sequence. The hidden state refers to the cognitive state that cannot be directly observed, including the dispersion state representing inattention, the concentration state representing normal concentration, and the fixation state representing excessive concentration. The dispersion state, as the basic reference state in the attention state transition model, plays a core role in providing a probability comparison benchmark for identifying the concentration state and the fixation state. By continuously calculating the background probability of the system being in the dispersion state, the model can accurately determine whether the current observation sequence deviates from the benchmark state, thereby triggering the judgment of the concentration or fixation state, so the dispersion state is a necessary component for constructing the state transition probability matrix and realizing effective pattern recognition.

[0189] In the embodiments of the present application, the visual attention point sequence and the operation response sequence are input as observations, the forward algorithm is used to calculate the probability accumulation from the front to the back, the backward algorithm is used to calculate the probability accumulation from the back to the front, and the probability distribution of being in three hidden states at each time point is calculated by combining the results of the two.

[0190] Step 603: When the probability of the concentration state or the fixation state exceeds a preset probability threshold, it is determined that the attention fixation mode occurs.

[0191] In step 603, the preset probability threshold is a critical value determined according to historical data statistical analysis, which is used to determine whether the attention state is abnormal.

[0192] In the embodiments of the present application, the state probability value at each time point is monitored in real time. When the probability of the concentrated state or the fixation state exceeds a set threshold value for a plurality of time steps, it is determined that the attention fixation mode occurs, and the start time, duration and intensity index of the mode are recorded.

[0193] The following is a specific example:

[0194] In the approach and landing training scene of the A airline pilot training center, based on the fusion multi-modal data matrix containing 5000 time points and 28 dimensions generated by the foregoing embodiments, the visual attention point sequence, i.e., the gaze instrument panel duration proportion sequence, and the operation response sequence, i.e., the operation response delay sequence, are first extracted from the matrix; the two sequences are input as observation sequences into a pre-trained hidden Markov model, which includes three hidden states, a dispersed state, a concentrated state and a fixation state, wherein the transition probability matrix A = [[0.7, 0.2, 0.1], [0.3, 0.6, 0.1], [0.2, 0.2, 0.6]] represents the transition probability between states, and the emission probability matrix B adopts a Gaussian distribution to describe the relationship between the observation value and the state; the state probability at each time is calculated by the forward-backward algorithm, wherein the forward probability calculation formula is wherein represents the forward probability of the hidden state at time -1, represents the forward probability of the hidden state at time -1, represents the probability of transition from the hidden state to the hidden state , represents the probability of observing the value under the hidden state , represents the observation value at time . The backward probability calculation formula is wherein represents the backward probability of the hidden state at time , represents the forward probability of the hidden state at time +1. The concentrated state probability is 0.92, the fixation state probability is 0.05, and the dispersed state probability is 0.03 at the 15th second. When it is detected that the concentrated state probability exceeds the preset threshold value 0.85 for 4 seconds from the 15th second to the 18th second, it is determined that the attention fixation mode occurs, and the start time is recorded as the 15th second, the duration is 4 seconds, and the average probability is 0.93. The recognition result and the subsequent workload index 82.3 jointly constitute the evaluation input data.

[0195] In the embodiments of the present application, the dynamic analysis of multi-modal data is performed by using a hidden Markov model, thereby realizing accurate recognition of attention states and abnormal pattern detection, providing an effective technical means for pilot state monitoring, and improving the intelligent level of flight safety management.

[0196] To solve the problem of insufficient pertinence of training reports, in some embodiments, step 106: generating an individualized training report based on the pilot competency evaluation result, comprises:

[0197] Step 701: extracting a weak link indicator from the pilot competency evaluation result.

[0198] In step 701, the weak link indicator refers to the ability dimension score lower than the preset threshold in the evaluation result, reflecting the ability short board that needs to be improved by the pilot, including the score items that do not meet the standard in each level evaluation value and the gap with the standard value.

[0199] In the embodiments of the present application, the system automatically scans all score items in the three-level evaluation result, marks the indicators lower than the corresponding threshold as weak links, and records the difference between the specific score and the threshold.

[0200] Step 702: matching the weak link indicator with the flight scene in the preset historical flight database to determine the target scene that needs to be strengthened.

[0201] In step 702, the historical flight database refers to a database storing past training records and scene characteristics. The target scene refers to the training environment setting most matched with the current weak link, and the flight situation that needs to be trained is determined by similarity calculation.

[0202] In the embodiments of the present application, the weak link indicator is matched and calculated with the scene characteristics in the database, and the historical training scene with the highest similarity is selected as the target scene, ensuring that the training content is highly related to the ability short board.

[0203] Step 703: reconstructing the attention allocation waveform and cognitive load curve under the target scene according to the time series data of the attention fixation pattern and the time series data of the workload index.

[0204] In step 703, the time sequence data of the attention fixation mode refers to the sequence data of the pilot's attention state change recorded in time sequence based on the analysis of the hidden Markov model, which contains the continuous record of the attention state type (dispersion / concentration / fixation) and its duration. The time sequence data of the workload index refers to the sequence of cognitive load quantitative values recorded in time sequence, which is a continuous numerical sequence generated by fusing the subjective scale data and the physiological entropy value data. The attention allocation waveform refers to the curve of the change of the attention state with time, and the cognitive load curve refers to the trend graph of the change of the workload index with time, which together reflect the change process of the cognitive state of the pilot in a specific scene.

[0205] In the embodiment of the present application, the historical time sequence data of the attention fixation mode and the workload index is extracted, intercepted and reorganized according to the time period of the target scene, to generate a visual curve reflecting the change of the cognitive state in the scene.

[0206] Step 704: identifying the attention abnormal period meeting the preset first abnormal identification condition from the reconstructed attention allocation waveform, and identifying the cognitive load over-limit period meeting the preset second abnormal identification condition from the reconstructed cognitive load curve.

[0207] In step 704, the first abnormal identification condition refers to the judgment standard of the attention state abnormality, specifically, when the fixation state continuously appears in the attention allocation waveform for more than a preset time threshold, or the frequency of the concentration state is lower than a preset frequency threshold, it is identified as an attention abnormal period. The second abnormal identification condition refers to the judgment standard of the cognitive load over-limit, specifically, when the value of the cognitive load curve continuously exceeds a preset load threshold, it is identified as a cognitive load over-limit period. The attention abnormal period refers to the period of continuous attention state abnormality, and the cognitive load over-limit period refers to the period of continuous load level over-limit.

[0208] In the embodiment of the present application, the reconstructed waveform and curve are scanned and detected to identify the time period meeting the attention continuous fixation condition and the time period with the load value continuously exceeding the threshold, and the start and end time of these abnormal periods are recorded.

[0209] Step 705: labeling the operation action corresponding to the attention abnormal period and the cognitive load over-limit period as an operation item to be improved.

[0210] In step 705, the operation item to be improved refers to the specific operation action corresponding to the abnormal period, which is determined by analyzing the operation record to improve the operation details, and provides clear target content for training.

[0211] In the embodiment of the present application, the operation record is queried according to the abnormal period timestamp, and all operation actions in the time period are extracted and labeled as operation items that need to be focused on and improved in training.

[0212] Step 706: generating a personalized training report according to the target scene, the operation item to be improved, and the expected training target.

[0213] In step 706, the expected training target refers to a quantifiable ability improvement index determined by comparing the weak link index with the historical flight database, including the target achievement value and the target improvement degree, which is derived from the difference analysis of the industry standard value and the personal historical baseline value.

[0214] In the embodiments of the present application, the training target value is set according to the gap between the weak link index and the standard value, and a complete training plan including training content, training duration and standard of achievement is generated in combination with the target scene and the operation item to be improved.

[0215] The following is a specific example:

[0216] In the approach and landing training scene of the pilot training center of A airline, the pilot competence evaluation result obtained based on the foregoing embodiments contains 78 points in the basic ability layer, 65 points in the dynamic performance layer, and 82 points in the risk warning layer. First, extract 65 points in the dynamic performance layer and 82 points in the risk warning layer from the result as the weak link index, wherein the dynamic performance layer is 5 points lower than the threshold value of 70 points, and the risk warning layer is 7 points higher than the threshold value of 75 points; match these indexes with the historical flight database, and calculate the matching degree with the bad weather approach scene as 0.85 through the similarity calculation formula wherein S represents the scene matching similarity, , the weight coefficients are all taken as 0.5, the numerical value similarity, the feature similarity, determine the bad weather approach scene as the target scene; according to the state probability 0.85-0.92 of the attention fixation pattern time series data and the numerical value 75-88 of the workload index time series data in the 10th-20th second, reconstruct the attention allocation waveform in the target scene to present a continuous high value and an upward trend of the cognitive load curve; identify the period when the state probability continuously exceeds 0.9 in the 15.5th-17.5th second as the attention abnormal period from the reconstructed attention allocation waveform, and identify the period when the load value continuously exceeds 85 in the 16th-18th second as the cognitive load overrun period from the cognitive load curve; query the operation record to mark the throttle control delay of 1.5 seconds and the insufficient heading correction of 2 degrees corresponding to the two periods as the operation items to be improved; finally, generate a personalized training report including 5 hours of instrument identification training and 3 hours of load control training according to the target scene of bad weather approach, the operation items to be improved of throttle control and heading correction, and the expected training target of improving the dynamic performance layer to more than 70 points, wherein the instrument identification training focuses on improving the gaze range coverage problem, and the load control training focuses on practicing the operation stability under stress situation.

[0217] In the embodiment of the present application, the training efficiency and the ability improvement effect of the pilot are effectively improved by accurately identifying the ability short board and matching the corresponding training scene, combining the positioning and analysis of specific operation problems, and generating a highly targeted personalized training scheme.

[0218] Figure 4 The structure diagram of the pilot competence dynamic evaluation system based on multi-modal data provided in the embodiment of the present application, and the specific implementation part describes:

[0219] The acquisition module 41 is configured to acquire flight control data, physiological signal data, psychological test data, subjective scale data, and core competence information defined by the International Civil Aviation Organization when the pilot operates the flight target.

[0220] The alignment module 42 is configured to perform alignment processing on the flight control data, the physiological signal data, and the psychological test data by using a time synchronization algorithm to obtain fused multi-modal data.

[0221] The analysis module 43 is configured to analyze the fused multi-modal data based on a hidden Markov model to identify an attention fixation mode.

[0222] The fusion module 44 is configured to fuse the subjective scale data and physiological entropy value data in the physiological signal data to construct a workload index.

[0223] The input module 45 is configured to input the fused multi-modal data, the attention fixation mode, the workload index, and the core competence information into a pre-trained competence evaluation model, and output a pilot competence evaluation result by using the competence evaluation model, wherein the pilot competence evaluation result includes three-level psychological evaluation indexes of a basic ability layer, a dynamic performance layer, and a risk warning layer.

[0224] The generation module 46 is configured to generate a personalized training report based on the pilot competence evaluation result.

[0225] The pilot competence dynamic evaluation system based on multi-modal data in the embodiment of the present application is used to implement the pilot competence dynamic evaluation method based on multi-modal data described above, and therefore the specific implementation part of the pilot competence dynamic evaluation system based on multi-modal data can refer to the embodiment part of the pilot competence dynamic evaluation method based on multi-modal data described above. The specific implementation can refer to the description of the corresponding embodiment part, and will not be described here.

[0226] The application further provides an electronic device, comprising: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of the method for dynamically evaluating pilot competence based on multi-modal data.

[0227] The application further provides a computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the steps of the method for dynamically evaluating pilot competence based on multi-modal data.

[0228] In an example embodiment, the computer-readable storage medium can include, but is not limited to, a U disk, a read-only memory, a random access memory, a mobile hard disk, a magnetic disk or an optical disk, and various media capable of storing a computer program.

[0229] The embodiments of the application further provide a computer program product, the computer program product comprising a computer program, the computer program being executed by a processor to implement the steps in the method for dynamically evaluating pilot competence based on multi-modal data.

[0230] The skilled person can further realize that the units and algorithm steps of the examples described in connection with the embodiments disclosed herein can be realized in electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of the examples have been described in general terms in the above description. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the application.

[0231] The above provides a method, system, device and storage medium for dynamically evaluating pilot competence based on multi-modal data. The principles and implementation modes of the application are described by using specific examples. The above description of the examples is only used to help understand the method and core idea of the application. It should be noted that, for those skilled in the art, without departing from the principles of the application, some improvements and modifications can be made to the application, and these improvements and modifications also fall within the protection scope of the application.

Claims

1. A pilot competency dynamic assessment method based on multi-modal data, characterized in that, The method comprises the following steps: Collecting flight control data, physiological signal data, psychological test data, subjective scale data, and core competency information defined by the International Civil Aviation Organization when a pilot operates a flight target; Aligning the flight control data, physiological signal data, and psychological test data using a time synchronization algorithm to obtain fused multi-modal data; Based on a hidden Markov model, analyzing the fused multi-modal data to identify an attention fixation mode; Fusing the subjective scale data and the physiological entropy data in the physiological signal data to construct a workload index; Inputting the fused multi-modal data, the attention fixation mode, the workload index, and the core competency information into a pre-trained competency evaluation model, and outputting a pilot competency evaluation result from the competency evaluation model, which includes three-level psychological evaluation indicators of a basic ability layer, a dynamic performance layer, and a risk warning layer; Based on the pilot competency evaluation result, generating a personalized training report; The inputting of the fused multi-modal data, the attention fixation mode, the workload index, and the core competency information into the pre-trained competency evaluation model, and the outputting of the pilot competency evaluation result from the competency evaluation model, comprises: Inputting the fused multi-modal data, the attention fixation mode, the workload index, and the core competency information into the pre-trained competency evaluation model, and establishing a mapping module in the competency evaluation model, taking the core competency information as an evaluation benchmark, and establishing a first index mapping relationship between the evaluation benchmark and the fused multi-modal data, a second index mapping relationship between the evaluation benchmark and the attention fixation mode, and a third index mapping relationship between the evaluation benchmark and the workload index; Performing pattern recognition and classification based on the first index mapping relationship through a pattern classifier in the competency evaluation model to obtain an operation mode classification result; Performing state prediction based on the second index mapping relationship through a time series predictor in the competency evaluation model to obtain a state prediction result, and performing load prediction based on the third index mapping relationship to obtain a load prediction result; Weighted fusion of the classification result, the state prediction result, and the load prediction result to generate a pilot competency evaluation result; The method comprises the following steps: Extracting a visual focus point sequence and an operation response sequence from the fused multi-modal data; Inputting the visual focus point sequence and the operation response sequence as observation sequences into a hidden Markov model, and calculating the probability of being in each hidden state at each time under the observation sequences through a forward-backward algorithm in the hidden Markov model, the hidden Markov model containing three hidden states of a dispersion state, a concentration state, and a fixation state; When the probability of the concentration state or the fixation state exceeds a preset probability threshold, it is determined that an attention fixation mode occurs.

2. The method for pilot proficiency dynamic assessment based on multi-modal data according to claim 1, characterized in that, The classification result, the state prediction result and the load prediction result are weighted and fused to generate a pilot competence evaluation result, comprising: obtaining a weight coefficient set of each evaluation index corresponding to a flight phase of a flight target; performing first weighted calculation on the classification result according to the weight coefficient set to obtain a technical ability score; performing second weighted calculation on the state prediction result according to the weight coefficient set to obtain an attention state score; performing third weighted calculation on the load prediction result according to the weight coefficient set to obtain a work load score; inputting the technical ability score, the attention state score and the work load score into a preset comprehensive evaluation function to generate a basic ability layer evaluation value, a dynamic performance layer evaluation value and a risk warning layer evaluation value; according to a preset grade mapping rule, the basic ability layer evaluation value, the dynamic performance layer evaluation value and the risk warning layer evaluation value are converted into corresponding psychological evaluation indexes; based on each psychological evaluation index, a pilot competence evaluation result is generated. 3.The method for dynamically evaluating pilot competency based on multi-modal data according to claim 1, characterized in that, The competence evaluation model is constructed, comprising: obtaining an initial training sample set, each initial sample in the training sample set comprising: historical fusion multi-modal data, historical attention fixation mode, historical work load index, historical core competence information and historical pilot competence evaluation result; determining the index weight priority of each modal data in the initial sample in different flight phases of the flight target by using the analytic hierarchy process to generate a target sample; based on the random forest classifier and the long short-term memory neural network, an initial evaluation model is constructed; based on the target sample, the initial evaluation model is trained until a preset training stop condition is met to obtain the competence evaluation model.

4. The method for pilot proficiency dynamic assessment based on multi-modal data according to claim 3, characterized in that, The index weight priority of each modal data in the initial sample in different flight phases of the flight target is determined by using the analytic hierarchy process to generate a target sample, comprising: constructing a hierarchical structure model, the target layer of the hierarchical structure model being a flight phase competence evaluation result, the criterion layer comprising the fusion multi-modal data, the attention fixation mode, the work load index and the core competence information, and the scheme layer being different flight phases; constructing a first judgment matrix of each element of the criterion layer to the target layer and a second judgment matrix of each element of the scheme layer to each element of the criterion layer; calculating the maximum eigenvalue of each judgment matrix and the corresponding eigenvector, and calculating a consistency ratio based on the maximum eigenvalue; when the consistency ratio is less than a preset ratio threshold, normalizing the eigenvector to obtain an effective weight vector of each level; extracting a scheme layer weight vector and a criterion layer weight vector from the effective weight vector; according to the current flight phase of the flight target, extracting a corresponding phase weight coefficient from the scheme layer weight vector; combining and calculating the phase weight coefficient and the criterion layer weight vector to generate the index weight priority of each modal data in a specific flight phase; according to the index weight priority, the numerical value of the corresponding modal data in the initial sample is weighted to generate a target sample. 5.The method for dynamically evaluating pilot competency based on multi-modal data according to claim 1, wherein, The personalized training report is generated based on the pilot competency evaluation result, including: extracting a weak link indicator from the pilot competency evaluation result; matching the weak link indicator with a flight scene in a preset historical flight database to determine a target scene that needs to be strengthened; reconstructing an attention allocation waveform and a cognitive load curve under the target scene according to the timing data of the attention fixation mode and the timing data of the workload index; identifying an attention abnormal period that meets a preset first abnormal identification condition from the reconstructed attention allocation waveform, and identifying a cognitive load over-limit period that meets a preset second abnormal identification condition from the reconstructed cognitive load curve; labeling an operation action corresponding to the attention abnormal period and the cognitive load over-limit period as an operation item to be improved; generating a personalized training report according to the target scene, the operation item to be improved, and an expected training target.

6. A pilot competency dynamic assessment system based on multi-modal data, characterized in that, It includes: The acquisition module is used for collecting flight control data, physiological signal data, psychological test data, subjective scale data and core competency information defined by the International Civil Aviation Organization when the pilot operates the flight target; The alignment module is used for aligning the flight control data, the physiological signal data and the psychological test data by using a time synchronization algorithm to obtain fused multi-modal data; The analysis module is used for analyzing the fused multi-modal data based on a hidden Markov model to identify an attention fixation mode; The fusion module is used for fusing the subjective scale data and the physiological entropy data in the physiological signal data to construct a workload index; The input module is used for inputting the fused multi-modal data, the attention fixation mode, the workload index and the core competency information into a pre-trained competency evaluation model, outputting a pilot competency evaluation result by the competency evaluation model, and the pilot competency evaluation result including three-level psychological evaluation indicators of a basic ability layer, a dynamic performance layer and a risk warning layer; The generation module is used for generating a personalized training report based on the pilot competency evaluation result; The input module is used for inputting the fused multi-modal data, the attention fixation mode, the workload index and the core competency information into a pre-trained competency evaluation model, outputting a pilot competency evaluation result by the competency evaluation model, and the pilot competency evaluation result including three-level psychological evaluation indicators of a basic ability layer, a dynamic performance layer and a risk warning layer; The generation module is used for generating a personalized training report based on the pilot competency evaluation result; The input module is used for inputting the fused multi-modal data, the attention fixation mode, the workload index and the core competency information into a pre-trained competency evaluation model, outputting a pilot competency evaluation result by the competency evaluation model, and the pilot competency evaluation result including three-level psychological evaluation indicators of a basic ability layer, a dynamic performance layer and a risk warning layer; The input module is used for inputting the fused multi-modal data, the attention fixation mode, the workload index and the core competency information into a pre-trained competency evaluation model, outputting a pilot competency evaluation result by the competency evaluation model, and the pilot competency evaluation result including three-level psychological evaluation indicators of a basic ability layer, a dynamic performance layer and a risk warning layer; The input module is used for inputting the fused multi-modal data, the attention fixation mode, the workload index and the core competency information into a pre-trained competency evaluation model, outputting a pilot competency evaluation result by the competency evaluation model, and the pilot competency evaluation result including three-level psychological evaluation indicators of a basic ability layer, a dynamic performance layer and a risk warning layer; The time series predictor in the competency evaluation model performs state prediction based on the second index mapping relationship to obtain a state prediction result, and performs load prediction based on the third index mapping relationship to obtain a load prediction result; The classification result, the state prediction result, and the load prediction result are fused by weighting to generate a pilot competency evaluation result; The fusion multi-modal data is analyzed based on the hidden Markov model to identify the attention fixation mode, including: A visual focus point sequence and an operation response sequence are extracted from the fusion multi-modal data; The visual focus point sequence and the operation response sequence are input as observation sequences into the hidden Markov model, and a forward-backward algorithm in the hidden Markov model is used to calculate probabilities of being in each hidden state at each time under the observation sequences, the hidden Markov model including three hidden states of a dispersion state, a concentration state, and a fixation state; When the probability of the concentration state or the fixation state exceeds a preset probability threshold, it is determined that the attention fixation mode occurs.

7. An electronic device, comprising: The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the steps of the pilot competency dynamic evaluation method based on multi-modal data according to any one of claims 1 to 5. The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the steps of the pilot competency dynamic evaluation method based on multi-modal data according to any one of claims 1 to 5. ​ 8. A computer-readable storage medium, characterized in that, ​

Images