Pilot competency dynamic evaluation method, system and equipment based on multi-modal data and storage medium

Abstract

The invention relates to the technical field of multi-modal data, provides a pilot competency dynamic evaluation method, system and device based on multi-modal data and a storage medium, and solves the problems of low accuracy of pilot competency evaluation and poor pertinence of training guidance. The method comprises the steps of collecting flight control data, physiological signal data, psychological assessment data, subjective scale data and international civil aviation organization core competency information, performing alignment fusion on multi-source data by adopting a time synchronization algorithm, and identifying an attention fixation mode based on a hidden Markov model. And constructing a workload index by fusing a subjective scale and a physiological entropy value, inputting the multi-modal features into a pre-trained competency assessment model, outputting three levels of psychological assessment indexes including a basic ability layer, a dynamic presentation layer and a risk early warning layer, and finally generating a personalized training report. According to the invention, the accuracy of pilot competency evaluation and the pertinence of training guidance are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multi-modal data, in particular to a pilot competence dynamic evaluation method and system based on multi-modal data, a device and a storage medium. BACKGROUND

[0002] In the field of aviation training, with the increasing complexity of flight tasks, there is an urgent need for a comprehensive evaluation method that can capture the cognitive state, operational performance and psychological load of pilots in real time. The traditional single-dimensional evaluation system has been difficult to meet the demand for precise quantification of human performance in modern aviation training, and a technical solution is needed that can integrate multi-source data and dynamically reflect the comprehensive competence of pilots.

[0003] Currently, there is an evaluation scheme based on physiological signals and operation data. This scheme collects physiological indicators such as electrocardiogram and skin electric response of pilots during simulated flight, combines joystick input parameters, and uses support vector machine algorithm to build a classification model to evaluate the workload level of pilots. This scheme detects physiological signals by setting a fixed threshold and generates a single-dimensional load evaluation report.

[0004] However, this scheme has deficiencies in the time sequence correlation processing of physiological signals and operation data, and lacks a dynamic adaptation mechanism for different flight stage-specific correlation patterns. The classification model used in the scheme has limited depth in describing the non-linear relationship between multi-dimensional features, resulting in insufficient granularity in recognizing complex cognitive states. In addition, the evaluation results do not form a hierarchical psychological indicator output, which limits the relevance of training recommendations. SUMMARY

[0005] The present application provides a pilot competence dynamic evaluation method, system, device and storage medium based on multi-modal data to solve the problems of low precision in pilot competence evaluation and poor relevance in training guidance in the prior art.

[0006] To solve the above technical problems, in a first aspect, the present application provides a pilot competence dynamic evaluation method based on multi-modal data, comprising: Collecting flight control data, physiological signal data, psychological test data, subjective scale data and core competence information defined by the International Civil Aviation Organization when the pilot operates the flight target; Aligning the flight control data, physiological signal data and psychological test data using a time synchronization algorithm to obtain fused multi-modal data; Based on a hidden Markov model, analyzing the fused multi-modal data to identify attention fixation patterns; fusing the subjective scale data and physiological entropy data in the physiological signal data to construct a workload index; inputting the fused multi-modal data, the attention fixation pattern, the workload index, and the core competency information into a pre-trained competency evaluation model, outputting a pilot competency evaluation result by the competency evaluation model, the pilot competency evaluation result including three-level psychological evaluation indexes of a basic ability layer, a dynamic performance layer, and a risk early warning layer; generating a personalized training report based on the pilot competency evaluation result.

[0007] Optionally, the inputting the fused multi-modal data, the attention fixation pattern, the workload index, and the core competency information into a pre-trained competency evaluation model, and outputting a pilot competency evaluation result by the competency evaluation model, includes: inputting the fused multi-modal data, the attention fixation pattern, the workload index, and the core competency information into a pre-trained competency evaluation model, and establishing, by a mapping module of the competency evaluation model, the core competency information as an evaluation benchmark, establishing a first index mapping relationship between the evaluation benchmark and the fused multi-modal data, a second index mapping relationship between the evaluation benchmark and the attention fixation pattern, and a third index mapping relationship between the evaluation benchmark and the workload index; performing pattern recognition and classification based on the first index mapping relationship by a pattern classifier in the competency evaluation model to obtain an operation mode classification result; performing state prediction based on the second index mapping relationship by a time series predictor in the competency evaluation model to obtain a state prediction result, and performing load prediction based on the third index mapping relationship to obtain a load prediction result; performing weighted fusion on the classification result, the state prediction result, and the load prediction result to generate a pilot competency evaluation result.

[0008] Optionally, the performing weighted fusion on the classification result, the state prediction result, and the load prediction result to generate a pilot competency evaluation result includes: obtaining a set of weight coefficients of each evaluation index corresponding to a flight phase of a flight target; performing first weighted calculation on the classification result according to the set of weight coefficients to obtain a technical ability score; performing second weighted calculation on the state prediction result according to the set of weight coefficients to obtain an attention state score; performing third weighted calculation on the load prediction result according to the set of weight coefficients to obtain a workload score; inputting the technical ability score, the attention state score and the workload score into a preset comprehensive evaluation function to generate a basic ability layer evaluation value, a dynamic performance layer evaluation value and a risk early warning layer evaluation value; According to the preset grade mapping rule, the basic ability layer evaluation value, the dynamic performance layer evaluation value and the risk early warning layer evaluation value are converted into corresponding psychological evaluation indexes; Based on each psychological evaluation index, a pilot competence evaluation result is generated.

[0009] Optionally, the competence evaluation model is constructed, including: An initial training sample set is obtained, and each initial sample in the training sample set includes: historical fusion multi-modal data, historical attention fixation mode, historical workload index, historical core competence information and historical pilot competence evaluation result; An analytic hierarchy process is used to determine the index weight priority of each modal data in the initial sample in different flight stages of the flight target to generate a target sample; Based on a random forest classifier and a long short-term memory neural network, an initial evaluation model is constructed; Based on the target sample, the initial evaluation model is trained until a preset training stop condition is met, and a competence evaluation model is obtained.

[0010] Optionally, the analytic hierarchy process is used to determine the index weight priority of each modal data in the initial sample in different flight stages of the flight target to generate a target sample, including: A hierarchical structure model is constructed, the target layer of the hierarchical structure model is a flight stage competence evaluation result, the criterion layer includes the fusion multi-modal data, the attention fixation mode, the workload index and the core competence information, and the scheme layer is different flight stages; A first judgment matrix of each element of the criterion layer to the target layer and a second judgment matrix of each element of the scheme layer to each element of the criterion layer are constructed; The maximum eigenvalue and the corresponding eigenvector of each judgment matrix are calculated, and based on the maximum eigenvalue, a consistency ratio is calculated; When the consistency ratio is less than a preset ratio threshold, the eigenvector is normalized to obtain an effective weight vector of each level; The scheme layer weight vector and the criterion layer weight vector are extracted from the effective weight vector; According to the current flight stage of the flight target, a corresponding stage weight coefficient is extracted from the scheme layer weight vector; The stage weight coefficient and the criterion layer weight vector are combined and calculated to generate the index weight priority of each modal data in a specific flight stage. According to the index weight priority, the values of the corresponding modal data in the initial sample are weighted to generate a target sample.

[0011] Optionally, the fusion multi-modal data is analyzed based on a hidden Markov model to identify the attention fixation mode, including: extracting a visual attention point sequence and an operation response sequence from the fusion multi-modal data; inputting the visual attention point sequence and the operation response sequence as observation sequences into a hidden Markov model, and calculating the probability of being in each hidden state at each time under the observation sequences by a forward-backward algorithm in the hidden Markov model, the hidden Markov model including three hidden states of dispersion state, concentration state and fixation state; when the probability of the concentration state or the fixation state exceeds a preset probability threshold, determining that the attention fixation mode occurs.

[0012] Optionally, the personalized training report is generated based on the pilot competence evaluation result, including: extracting a weak link indicator from the pilot competence evaluation result; matching the weak link indicator with a flight scene in a preset historical flight database to determine a target scene requiring intensive training; reconstructing an attention allocation waveform and a cognitive load curve under the target scene according to the time series data of the attention fixation mode and the time series data of the workload index; identifying an attention abnormal period that meets a preset first abnormal identification condition from the reconstructed attention allocation waveform, and identifying a cognitive load over-limit period that meets a preset second abnormal identification condition from the reconstructed cognitive load curve; labeling an operation action corresponding to the attention abnormal period and the cognitive load over-limit period as an operation item to be improved; generating a personalized training report according to the target scene, the operation item to be improved and an expected training target.

[0013] In a second aspect, the present application provides a pilot competence dynamic evaluation system based on multi-modal data, including: a collection module configured to collect flight control data, physiological signal data, psychological test data, subjective scale data and core competence information defined by the International Civil Aviation Organization when a pilot operates a flight target; an alignment module configured to perform alignment processing on the flight control data, the physiological signal data and the psychological test data by using a time synchronization algorithm to obtain fusion multi-modal data; an analysis module configured to analyze the fusion multi-modal data based on a hidden Markov model to identify an attention fixation mode; a fusion module configured to fuse the subjective scale data and the physiological entropy value data in the physiological signal data to construct a workload index; an input module configured to input the fusion multi-modal data, the attention fixation mode, the workload index, and the core competency information into a pre-trained competency evaluation model, and output a pilot competency evaluation result by the competency evaluation model, the pilot competency evaluation result including three-level psychological evaluation indexes of a basic ability layer, a dynamic performance layer, and a risk early warning layer; a generation module configured to generate an individualized training report based on the pilot competency evaluation result.

[0014] In a third aspect, the present application provides an electronic device, comprising: a memory configured to store a computer program; a processor configured to implement the steps of the pilot competency dynamic evaluation method based on multi-modal data according to the first aspect when executing the computer program.

[0015] In a fourth aspect, the present application provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executable by a processor to implement the steps of the pilot competency dynamic evaluation method based on multi-modal data according to the first aspect.

[0016] The technical scheme provided by the present application has the following beneficial effects: The present application realizes comprehensive collection of flight control, physiological state, psychological characteristics, and industry standard information, provides complete data basis for comprehensive evaluation. The time synchronization algorithm is used to solve the time sequence difference problem of multi-source data, form time-space unified fusion data, and ensure the accuracy of subsequent analysis. The hidden Markov model is used to dynamically analyze the attention state of pilots, and accurately capture abnormal attention allocation behavior. The subjective scale and objective physiological entropy value are fused to form a quantitative cognitive load comprehensive index, and the reliability of workload evaluation is enhanced. The machine learning model is used to integrate multi-dimensional features, output layered psychological evaluation indexes, and realize fine diagnosis of competency. The evaluation result is used to automatically generate a targeted training scheme, and the accuracy and efficiency of flight training are improved.

[0017] Further, the present application further establishes a multi-dimensional index mapping relationship, respectively identifies, classifies, and predicts the operation mode, attention state, and workload, and finally generates a comprehensive evaluation result by weighted fusion.

[0018] And the scheme realizes organic fusion and dynamic analysis of multi-source evaluation indexes, can comprehensively reflect the operation characteristics, cognitive state and load level of the pilot, and improves the systematization and precision of the competency evaluation.

[0019] These aspects or other aspects of the present application will be more apparent in the following description of embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0021] Figure 1 A flow chart of a pilot competency dynamic evaluation method based on multi-modal data provided by an embodiment of the present application; Figure 2 A specific implementation schematic diagram of a pilot competency dynamic evaluation method based on multi-modal data provided by an embodiment of the present application; Figure 3 Another specific implementation schematic diagram of a pilot competency dynamic evaluation method based on multi-modal data provided by an embodiment of the present application; Figure 4 A structural schematic diagram of a pilot competency dynamic evaluation system based on multi-modal data provided by an embodiment of the present application. DETAILED DESCRIPTION

[0022] In order to make the person skilled in the art better understand the present application, the present application will be further described in detail below in combination with the drawings and specific embodiments. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0023] The core of the present application is to provide a pilot competency dynamic evaluation method based on multi-modal data, and a flow schematic diagram of a specific implementation mode thereof is shown in Figure 1 The method comprises: Step 101: collecting flight control data, physiological signal data, psychological test data, subjective scale data and core competency information defined by the International Civil Aviation Organization when the pilot operates the flight target.

[0024] In step 101, the flight control data refers to the control instruction parameters generated by the pilot during the operation process and the external environment information, specifically including simulation machine control parameters such as pitch angle deviation rate, throttle response delay and other indicators reflecting operation accuracy, and external environment data such as weather visibility, air traffic control instruction density and other situational factors affecting operation difficulty. Exemplarily, the flight control data also includes pitch and roll inputs of the control stick, rudder control of the footrest, thrust adjustment of the throttle lever, landing gear handle operation, ground steering control of the turn handwheel, and state adjustment information of various system panel buttons and knobs, comprehensively reflecting the pilot's comprehensive operation behavior on the aircraft. The physiological signal data is the pilot's physiological response information collected by the biological sensor, including the gaze hotspot distribution and saccade path efficiency recorded by the eye tracker to reflect visual attention allocation, the skin conductance response and heart rate variability monitored by the wearable device to represent autonomic nervous excitability, and the brain function indicators such as frontal lobe theta wave (related to cognitive load) and alpha wave (related to attention concentration) extracted by the electroencephalogram device. Exemplarily, the physiological signal data also includes electroencephalogram signals reflecting brain activity state, electrocardiogram signals reflecting heart function, respiration signals reflecting respiration rhythm, pulse signals reflecting cardiovascular state, skin conductance signals reflecting skin electrical activity, electromyogram signals reflecting muscle tension, eye movement signals reflecting visual attention allocation, pupil diameter changes reflecting cognitive load, facial expression features reflecting emotional state, and voice features reflecting psychological stress level. The psychological assessment data is the quantitative results of cognitive ability obtained through standardized psychological paradigms, including basic ability tests such as attention allocation scores of attention network tests, emotional interference effect values of emotional Stroop tasks, and performance indicators such as decision delay time and accuracy in special situation handling scenarios (taking engine failure as an example). The subjective scale data is the load evaluation scale filled in by the pilot according to his own feelings, such as the multi-dimensional subjective scores of NASA-TLX scale. The core competence information is the ability framework that the pilot must have defined by the International Civil Aviation Organization, including standardized ability dimensions such as situational awareness, decision-making ability, and communication and cooperation.

[0025] In the embodiments of the present application, the data acquisition system deployed on the flight simulator records the pilot's control actions in real time, and the electrocardiogram, skin conductance and other signals are collected through the wearable physiological monitoring device. Standardized psychological tests are conducted before the start of the flight task to obtain basic cognitive ability data, and the pilot is guided to fill in the subjective load scale after the task is completed. Finally, the standardized ability requirement document published by the International Civil Aviation Organization is integrated to form a complete multi-source data set.

[0026] For example, in a certain airline pilot training center, when the pilot operates the B737 simulator for approach and landing training, the system collects 12 groups of pitch control data, 10 groups of roll control data, and 8 groups of throttle position data; through the smart bracelet, 1500 sample points of heart rate variability data and 1200 sample points of skin electricity reaction data are collected; before training, the attention network test is completed to obtain an attention allocation score of 85 points, and the emotional Stroop test is completed to obtain an emotional stability score of 78 points; after training, the subjective scale of NASA-TLX is filled out to obtain a subjective load score of 70 points; and 9 core competency items in the International Civil Aviation Organization (ICAO) document are integrated to form a multi-source data set of this training.

[0027] Step 102: Aligning the flight control data, the physiological signal data, and the psychological test data by using a time synchronization algorithm to obtain fused multi-modal data.

[0028] In step 102, the time synchronization algorithm is a calculation method for time axis alignment of multi-source data sequences with different sampling speeds, and different data with different sampling frequencies are made to have the same time stamp by interpolation or resampling. The fused multi-modal data refers to a multi-dimensional data set with a unified time stamp formed after time alignment, wherein each time point contains synchronized control, physiological, and psychological data.

[0029] In the embodiment of the present application, first, the time stamp information of the collected multi-source data is extracted, the highest sampling rate is determined as the reference frequency, the resampling is performed on other data sequences by using the cubic spline interpolation method, so that all data streams have the same time interval and point number, and finally the aligned data is integrated into a multi-dimensional matrix form of the fused data set according to the time stamp.

[0030] For example, the control data (sampling rate 10 Hz), physiological data (sampling rate 100 Hz), and psychological data (sampling rate 1 Hz) collected in step 101 are time-synchronized, 100 Hz is used as the reference frequency, linear interpolation is used to increase the control data from 10 Hz to 100 Hz, and spline interpolation is used to increase the psychological data from 1 Hz to 100 Hz, and finally a fused multi-modal data matrix containing 5000 time points (corresponding to a 50-second training process) and 28 dimensions per time point is generated.

[0031] Step 103: Based on a hidden Markov model, the fused multi-modal data is analyzed to identify an attention fixation mode.

[0032] In step 103, the hidden Markov model is a probability model describing the generation of an observation sequence by a hidden state sequence, where the hidden state represents an unobservable cognitive state (dispersion, concentration, fixation), and the observation sequence refers to the measurable sequence of eye movement, operation, etc. extracted from the fusion data. The attention fixation mode refers to an abnormal state in which the pilot's attention is excessively concentrated on a specific interface element and lasts for too long.

[0033] In the embodiments of the present application, the eye movement fixation point coordinate sequence and operation response delay sequence are extracted from the fusion data as observation inputs, and the forward-backward algorithm is used to calculate the probability of being in three hidden states at each time. When the probability of concentration or fixation state continues to exceed the threshold value, it is marked as an attention fixation event, and its start and end time and intensity value are recorded. The specific implementation process is as follows: first, the sample entropy of heart rate variability and the approximate entropy of pupil diameter fluctuation are extracted from the physiological signal data as objective physiological entropy values, and the multi-dimensional scores in the subjective scale data are standardized; then, a regression model is trained based on historical data in high-pressure scenarios such as bad weather approach, and a mapping relationship between physiological entropy values and subjective scores is established; finally, the standardized subjective scores and physiological entropy values are combined through a weighted fusion algorithm, where the weight coefficient is dynamically adjusted through an entropy value analysis model, and when the fusion value exceeds the preset critical threshold, a situational awareness degradation warning is triggered, thereby generating a workload index that can reflect the cognitive load level in real time.

[0034] For example, the gaze instrument panel duration ratio sequence and operation response delay sequence are extracted from the fusion data of step 102, and input into the trained hidden Markov model (transition probability matrix A=[[0.7, 0.2, 0.1], [0.3, 0.6, 0.1], [0.2, 0.2, 0.6]], emission probability matrix B is Gaussian distribution), and the calculation result is that the concentration state probability of 15-18 seconds is greater than 0.85, which is determined as an attention fixation mode, with a duration of 3 seconds and an average probability of 0.88.

[0035] Step 104: fuse the subjective scale data and the physiological entropy data in the physiological signal data to construct a workload index.

[0036] In step 104, the physiological entropy value data is a complexity index calculated from the physiological signal, such as heart rate variability sample entropy, skin electric reaction approximate entropy, reflecting the degree of disorder of the autonomic nervous system, wherein the complexity index is a nonlinear characteristic parameter obtained by quantitatively analyzing multi-source physiological signals through information entropy theory and signal processing method, and the calculation process includes: using differential entropy and power spectral density method to extract energy distribution characteristics of α, β, θ frequency band for electroencephalogram signal, and using P300 potential amplitude variation characteristics for memory-related tasks; using signal decomposition methods such as variational mode decomposition to extract intrinsic mode for electroencephalogram, electrocardiogram, skin electricity, pupil diameter, eye movement scan video frequency, blink frequency and other signals; extracting local texture features, shape features and wavelet transform features for expression data, and extracting prosody features, spectral features and voice quality features for speech data; finally, through machine learning methods such as convolutional neural network, graph neural network and Transformer, the normalized multi-modal features are fused to generate a two-dimensional image feature vector representing the complexity of the physiological state, which comprehensively reflects the nonlinear dynamic characteristics and information processing efficiency of the physiological system. The workload index is a standardized load quantitative value formed by combining the subjective scale score and the objective physiological entropy value.

[0037] In the embodiments of the present application, sample entropy is calculated for heart rate variability signal, approximate entropy is calculated for skin electric response signal, the two entropy values are z-score standardized and then weighted and fused, and then linearly combined with the standardized subjective scale score, and finally mapped to the 0-100 interval through the sigmoid function to form the workload index. The specific process is to quantitatively process the subjective scale data to obtain a subjective load score; extract physiological entropy value data from the physiological signal data, the physiological entropy value data including a heart rate sequence and a skin electric response signal sequence; calculate the sample entropy value of the heart rate sequence as a first physiological entropy value, and calculate the sample entropy value of the skin electric response signal sequence as a second physiological entropy value; linearly weight and fuse the subjective load score, the first physiological entropy value and the second physiological entropy value; and interval map and standardize the fused value to generate a workload index.

[0038] For example, the sample entropy value of the heart rate variability data in the calculation step 101 is 1.32, the approximate entropy value of the skin electric response data is 0.87, and the objective load value is 0.68 after standardization; the subjective scale score is 70, and the standardized value is 0.72; the fusion value is 0.696 according to the weights 0.6 (objective) and 0.4 (subjective); and the workload index is 82.3, wherein the x fusion value is 0.696, according to the formula

[0039] ​Step 105: inputting the fusion multi-modal data, the attention fixation mode, the workload index, and the core competency information into a pre-trained competency evaluation model, outputting a pilot competency evaluation result by the competency evaluation model, the pilot competency evaluation result including three-level psychological evaluation indexes of a basic ability layer, a dynamic performance layer, and a risk early warning layer.

[0040] In step 105, the pre-trained competency evaluation model refers to a multi-input and multi-output prediction model trained by a method of machine learning such as a convolutional neural network (CNN), a graph neural network (GNN), a long short-term memory network (LSTM), and the like, which inputs include multi-modal features, attention modes, load indexes, and ability standards, and outputs three-level evaluation indexes. Exemplarily, the competency evaluation model further includes dimensions, different physiological data used in different dimensions, and comprehensive evaluation by integrating physiological data and operation data in different dimensions, specifically including attention dimensions using pupil diameter, brain electrical alpha / theta waves, heart rate variability, and flight control data, memory dimensions using flight control data, voice features, eye movement patterns, brain electrical theta waves, and P300 potentials, cognitive dimensions using flight control data, brain electrical theta waves, and heart rate variability, motion perception dimensions using brain electrical alpha waves, flight control data, eye movement saccade area detection, and electromyographic signals, risk perception dimensions using heart rate variability, skin electric response, eye movement features, pupil diameter, and brain electrical beta / theta waves, decision-making dimensions using brain electrical theta waves, P300 potentials, pupil diameter, and heart rate variability, stress dimensions using skin electric response, heart rate variability, brain electrical alpha / beta waves, and respiratory waveform, emotional dimensions using heart rate variability, brain electrical alpha waves, pupil diameter, respiratory features, voice signals, and facial expression data, and precise evaluation of each dimension ability is realized by multi-modal data fusion analysis. The basic ability layer evaluates static psychological characteristics, the dynamic performance layer evaluates real-time operation efficiency, and the risk early warning layer evaluates potential risk levels.

[0041] In the embodiments of the present application, the fusion data, attention mode, load index, and ability standard are input into the trained random forest + long short-term memory network (LSTM) hybrid model. First, the operation mode type is identified by the random forest classifier, then the load and attention trend are predicted by the LSTM network, and finally all outputs are dynamically weighted and fused according to the flight phase to generate three-level evaluation values.

[0042] For example, the fusion data (28 dimensions x 5000 points) obtained by the preceding step, the attention fixation mode (duration 3 seconds, intensity 0.88), the workload index 82.3, and the ICAO capability standard input model are input into the random forest to output the operation mode risk probability 0.76, the LSTM outputs the attention deterioration trend 0.62 and the load rising trend 0.85, which are weighted according to the approach phase weight (operation 0.5, attention 0.3, load 0.2) to obtain 0.76x0.5+0.62x0.3+0.85x0.2=0.756, which is mapped to the basic ability layer 78 points, the dynamic performance layer 65 points, and the risk warning layer 82 points.

[0043] Step 106: generating a personalized training report based on the pilot competency evaluation result.

[0044] In step 106, the personalized training report is a targeted training program customized according to the weaknesses in the evaluation result, including the ability dimensions that need to be strengthened, the training subjects, the training duration, and the expected targets.

[0045] In the embodiments of the present application, the ability indicators below the threshold value are extracted from the evaluation result, the training case database is queried to match the training program of a similar scene, the operation actions that need to be improved are marked in combination with the specific abnormal period of the attention mode and the load curve, and finally a text report containing the training subject, the training intensity, and the improvement target is generated.

[0046] For example, according to the dynamic performance layer 65 points (threshold value 70) and the risk warning layer 82 points (threshold value 75) in the output of step 105, it is determined that the instrument scanning and load management capabilities need to be improved, the "adverse weather approach" training scene in the database is matched, and the throttle operation delay problem corresponding to the 15-18 second attention fixation period is combined to generate a report containing 5 hours of instrument identification training and 3 hours of load control training, and the expected target is to improve the dynamic performance layer to more than 70 points.

[0047] The method realizes comprehensive and dynamic monitoring of the cognitive state, operation performance, and psychological load of the pilot through synchronous acquisition and fusion processing of multi-modal data; accurately identifies the attention abnormal mode and the workload change trend through the combination and application of the hidden Markov model and the machine learning algorithm; and finally generates three-level evaluation indicators and a personalized training report, which provides a precise and targeted scientific training basis for the improvement of the pilot's ability, and improves the quality and efficiency of aviation training.

[0048] In order to solve the problem of lack of dynamic fusion mechanism for multi-dimensional evaluation indicators in the prior art, in some embodiments, step 105: the fusion multi-modal data, the attention fixation pattern, the workload index, and the core competency information are input into a pre-trained competency evaluation model, and a pilot competency evaluation result is output through the competency evaluation model, as shown in Figure 2 The method comprises the following steps: Step 201: input the fusion multi-modal data, the attention fixation pattern, the workload index, and the core competency information into a pre-trained competency evaluation model, and through a mapping module of the competency evaluation model, the core competency information is taken as an evaluation benchmark, and the evaluation benchmark is respectively mapped to the fusion multi-modal data to establish a first index mapping relationship, mapped to the attention fixation pattern to establish a second index mapping relationship, and mapped to the workload index to establish a third index mapping relationship.

[0049] In step 201, the mapping module is a component responsible for establishing an index correlation relationship in the competency evaluation model, which takes the core competency information defined by the International Civil Aviation Organization as a benchmark framework. The first index mapping relationship refers to the establishment of a quantitative correlation between the 9 core competencies (including situational awareness, decision-making ability, communication ability, etc.) defined by the International Civil Aviation Organization and the operation characteristic indicators (such as pitch angle deviation rate, throttle response delay, etc.) in the fusion multi-modal data through mathematical modeling. Through the calculation of the similarity measure of the multi-dimensional feature vector and each competency dimension, a numerical correlation model reflecting the matching degree of operation behavior and standard ability framework is formed. The second index mapping relationship refers to the establishment of a dynamic correlation between the characteristic parameters (such as duration, intensity index, and occurrence frequency) of the attention fixation pattern and the cognitive ability dimension (such as situational awareness and work management) in the core competency, and the influence degree of the attention abnormal pattern on the specific ability dimension is quantified through a state transition probability model. The third index mapping relationship refers to the establishment of a functional relationship between the numerical change of the workload index and the performance dimension (such as decision-making efficiency and operation precision) in the core competency, and the nonlinear mapping rule between the load level and the ability performance is described through a regression analysis model, which is used to evaluate the stability of the competency under different load states.

[0050] In the embodiments of the present application, the mapping module first analyzes the text description of the core competency information, extracts the key ability dimension as a benchmark vector, then calculates the cosine similarity of the fusion multi-modal data feature vector and the benchmark vector to form the first index mapping relationship, calculates the Euclidean distance of the attention fixation pattern duration and intensity index and the benchmark behavior dimension to form the second index mapping relationship, and calculates the relative deviation of the workload index value and the benchmark load dimension to form the third index mapping relationship. Finally, three mapping relationship matrices are output.

[0051] Step 202: performing pattern recognition and classification based on the first index mapping relationship through a pattern classifier in the competency evaluation model to obtain an operation pattern classification result.

[0052] In step 202, the pattern classifier is a classification component constructed based on a machine learning algorithm in the competency evaluation model, which receives the first index mapping relationship matrix as input, recognizes the operation behavior pattern through multi-layer decision rules, outputs an operation pattern classification result containing three types of operation patterns, i.e., normal operation, risk operation and critical operation, and gives the probability distribution of each category. The operation pattern classification result refers to the classification label and its probability distribution output by the machine learning algorithm after recognizing the pilot operation behavior characteristics, which is used to judge whether the current operation belongs to the normal operation, risk operation or critical operation mode type, and reflects the operation standardization and risk tendency of the pilot.

[0053] In the embodiments of the present application, the pattern classifier adopts an ensemble learning algorithm, first performs feature dimension reduction processing on the first index mapping relationship matrix, then parallelly calculates the likelihood of different operation patterns through multiple decision tree models, and finally integrates the outputs of all decision trees through a voting mechanism to obtain the final operation pattern classification result and the corresponding confidence score.

[0054] Step 203: performing state prediction based on the second index mapping relationship through a time series predictor in the competency evaluation model to obtain a state prediction result, and performing load prediction based on the third index mapping relationship to obtain a load prediction result.

[0055] In step 203, the time series predictor is a prediction component in the competence evaluation model specially processing time series data, which respectively receives the second index mapping relationship and the third index mapping relationship as input, learns the historical sequence change law through a recurrent neural network structure, and outputs the state prediction result of the attention state in the future time period and the load prediction result of the workload change trend. The state prediction result refers to the prediction value of the attention state change trend in the future time period based on the time series data, which is used to warn the state abnormalities such as attention fixation, dispersion or transfer, and reflects the dynamic evolution law of the cognitive state of the pilot. The load prediction result refers to the estimated value of the future workload level by analyzing the historical load data, which is used to predict the rising or falling trend of the cognitive load, and reflects the change of the psychological resource consumption of the pilot. The specific implementation process is as follows: the time series predictor first extracts the historical attention pattern features based on the attention state time series data in the second index mapping relationship, learns the state transition law through the gating mechanism, predicts the state probability distribution of attention fixation, concentration or dispersion in the future time period, and generates a state warning result when the attention loss probability of consecutive multiple time steps exceeds the threshold; at the same time, based on the load time series data in the third index mapping relationship, the load change trend is analyzed using a sliding window, combined with the eye movement hotspot drift rate feature (when the gaze point deviates from the key instrument area for 3 seconds, an abnormal mark is triggered), the rising or falling trajectory of the workload is predicted, and finally the comprehensive prediction result containing attention state warning and load overrun warning is output.

[0056] In the embodiments of the present application, the time series predictor adopts a recurrent neural network with a gating mechanism, first divides the second index mapping relationship sequence into a sliding window, learns the attention state evolution law through hidden state transmission, and outputs the state prediction value of the future time steps; at the same time, the third index mapping relationship sequence is decomposed in time series, the trend feature and the period feature are extracted, and the prediction trajectory of the workload is output.

[0057] Step 204: Weighted fusion of classification results, state prediction results and load prediction results to generate pilot competence evaluation results.

[0058] In step 204, the weighted fusion is a process of integrating multi-dimensional prediction results through a dynamic weight distribution algorithm, which assigns different weight coefficients to the operation mode classification result, the state prediction result and the load prediction result according to the characteristics of the current flight phase, and generates three-level evaluation results including basic ability score, dynamic performance score and risk warning score through linear weighted calculation.

[0059] In this embodiment, the weighted fusion module first reads the weight combination corresponding to the current flight phase from the preset weight configuration library, then standardizes the probability value of the operation mode classification result, the value of the state prediction result, and the value of the load prediction result, respectively, and finally performs weighted summation according to the weight coefficients, and maps it to the standard scoring interval through the activation function to generate the final three-level pilot competency assessment result.

[0060] Here is a specific example: In the approach and landing training scenario at the A Airlines pilot training center, the 28-dimensional × 5000 time points of the fused multimodal data obtained in the aforementioned embodiment, the duration of attention fixation mode (3 seconds) and intensity (0.88), the workload index (82.3), and the ICAO core competency information were input into a pre-trained competency assessment model. First, the mapping module used the nine core competencies as assessment benchmarks. The cosine similarity between the pitch angle deviation rate sequence and the situational awareness dimension in the fused multimodal data was calculated, yielding a first index mapping relationship value of 0.85. The calculation formula is as follows: ,in Indicates the first Pitch angle deviation rate values ​​at each time point (unit: degrees / second). The first vector representing the baseline vector of the context awareness dimension Several numerical values; the Euclidean distance between the intensity of attention fixation mode and the decision-making ability dimension yields a second index mapping value of 0.35, calculated using the following formula: ,in The first feature vector representing attention fixation One parameter, The first vector representing the baseline vector of decision-making ability dimension The numerical values; the relative deviation between the workload index and the work management dimension yields a third indicator mapping value of 0.18, calculated using the following formula: Where R represents the current workload index of 82.3 (dimensionless), and S represents the baseline value of the work management dimension of 70 (dimensionless). Subsequently, a pattern classifier performs pattern recognition based on the first indicator mapping value of 0.85, outputting an operation mode classification result of a high-risk landing mode probability of 0.76. A time series predictor predicts a deterioration trend in attention state of 0.62 in the next 10 seconds based on the second indicator mapping value of 0.35, and a load increase trend value of 0.85 based on the third indicator mapping value of 0.18. Finally, the three results are weighted and fused, calculated as follows: Comprehensive Score = Operation Mode Weight 0.5 × Classification Result 0.76 + Attention Weight 0.3 × State Prediction Result 0.62 + Load Weight 0.2 × Load Prediction Result 0.85. Substituting the values... The value is converted into a three-level evaluation result by a linear mapping function, wherein the basic ability layer score = 50 + 50*0.736 = 86.8, the dynamic performance layer score = 30 + 30*0.736 = 52.08, and the risk warning layer score = 20 + 20*0.736 = 34.72, and finally a pilot competence evaluation result containing the three values is generated.

[0061] In the embodiments of the present application, the accurate conversion from multi-source data to evaluation results is realized through a multi-stage dynamic fusion mechanism, a complete technical chain of index correlation analysis, pattern recognition, trend prediction and decision fusion is established, and the accuracy and practicability of the evaluation results are improved, thereby providing a scientific basis for pilot competence training.

[0062] In order to solve the problem of insufficient fusion precision of multi-dimensional evaluation results, in some embodiments, step 204: the classification result, the state prediction result and the load prediction result are weighted and fused to generate a pilot competence evaluation result, as shown in FIG. 5, which includes: Figure 3 Step 301: obtaining a weight coefficient set of each evaluation index corresponding to the flight phase of the flight target.

[0063] In step 301, the weight coefficient set refers to the index importance configuration parameter preset for different flight phases, which includes the weight distribution proportion of the three types of indexes of operation mode, attention state and workload. The set is determined by expert evaluation and historical data analysis together, and reflects the relative importance of each evaluation index in a specific flight phase.

[0064] In the embodiments of the present application, the system first identifies the current flight phase type, then queries the weight coefficient combination corresponding to the phase from the preset weight configuration database, and the combination includes three decimal values corresponding to the three types of evaluation indexes, and finally loads the weight coefficient set into the memory for subsequent calculation.

[0065] Step 302: performing first weighted calculation on the classification result according to the weight coefficient set to obtain a technical ability score.

[0066] In step 302, the first weighted calculation refers to the process of weighting the classification result using the operation mode weight in the weight coefficient set, which multiplies the probability value of the operation mode classification result by the corresponding weight to obtain a quantitative score reflecting the technical ability level. The technical ability score refers to the quantitative value obtained by weighting the operation mode classification result, which reflects the operation standardization and technical proficiency of the pilot in a specific flight phase. The higher the score, the greater the operation risk or the more obvious the technical defects.

[0067] ​In the embodiment of the present application, the operation mode weight value is extracted from the weight coefficient set, the high-risk operation probability in the classification result is multiplied by the weight value, and a technical ability score is generated. The higher the score value is, the greater the operation risk is.

[0068] Step 303: According to the weight coefficient set, a second weighted calculation is performed on the state prediction result to obtain an attention state score.

[0069] In step 303, the second weighted calculation refers to a process of performing weighted processing on the state prediction result by using the attention state weight in the weight coefficient set. The attention state prediction value is multiplied by the corresponding weight to obtain a quantitative score reflecting the attention state. The attention state score refers to a quantitative value obtained by weighting the attention state prediction result, representing the rationality and stability of the pilot's attention distribution. The higher the score is, the more serious the attention abnormality is or the worse the attention management ability is.

[0070] In the embodiment of the present application, the attention state weight value is extracted from the weight coefficient set, the attention deterioration trend value in the state prediction result is multiplied by the weight value, and an attention state score is generated. The higher the score value is, the worse the attention state is.

[0071] Step 304: According to the weight coefficient set, a third weighted calculation is performed on the load prediction result to obtain a workload score.

[0072] In step 304, the third weighted calculation refers to a process of performing weighted processing on the load prediction result by using the workload weight in the weight coefficient set. The workload prediction value is multiplied by the corresponding weight to obtain a quantitative score reflecting the load level. The workload score refers to a quantitative value obtained by weighting the workload prediction result, representing the intensity and change trend of the pilot's cognitive load. The higher the score is, the greater the psychological pressure is or the weaker the load regulation ability is.

[0073] In the embodiment of the present application, the workload weight value is extracted from the weight coefficient set, the load rising trend value in the load prediction result is multiplied by the weight value, and a workload score is generated. The higher the score value is, the greater the load pressure is.

[0074] Step 305: The technical ability score, the attention state score, and the workload score are input into a preset comprehensive evaluation function to generate a basic ability layer evaluation value, a dynamic performance layer evaluation value, and a risk warning layer evaluation value.

[0075] In step 305, the comprehensive evaluation function refers to a mathematical function for converting the three types of weighted scores into three levels of evaluation values, which maps the technical ability score, attention state score and workload score to the evaluation value interval of the basic ability layer, dynamic performance layer and risk warning layer respectively through a linear combination formula. The basic ability layer evaluation value refers to the quantitative result of the basic psychological characteristics obtained by the comprehensive evaluation function, reflecting the inherent cognitive ability and emotional stability level of the pilot. The dynamic performance layer evaluation value refers to the quantitative result of the real-time operation efficiency obtained by the comprehensive evaluation function, embodying the actual operation performance and quality of the pilot in specific flight tasks. The risk warning layer evaluation value refers to the quantitative result of the potential risk obtained by the comprehensive evaluation function, indicating the risk degree of operation failure or state abnormality of the pilot in subsequent tasks.

[0076] In the embodiments of the present application, the technical ability score is input into the basic ability layer conversion function, the attention state score is input into the dynamic performance layer conversion function, and the workload score is input into the risk warning layer conversion function, and the corresponding three-level evaluation values are generated through three parallel linear calculation formulas.

[0077] Step 306: According to the preset grade mapping rule, the basic ability layer evaluation value, the dynamic performance layer evaluation value and the risk warning layer evaluation value are converted into corresponding psychological evaluation indexes.

[0078] In step 306, the grade mapping rule refers to the corresponding relationship table for converting numerical evaluation values into graded psychological evaluation indexes, which defines the mapping relationship between different numerical intervals and grade identifiers, and is used to discretize continuous numerical values into graded indexes with semantic interpretation. The psychological evaluation index refers to the graded evaluation result obtained by converting the numerical evaluation value through the grade mapping rule, which includes the semantic description of three dimensions of basic ability grade, dynamic performance grade and risk warning grade, providing intuitive reference for training improvement.

[0079] In the embodiments of the present application, the basic ability layer evaluation value, the dynamic performance layer evaluation value and the risk warning layer evaluation value are compared with the preset grade threshold value respectively, and the corresponding grade identifier is determined according to the interval to generate three-level psychological evaluation indexes.

[0080] Step 307: Based on each psychological evaluation index, a pilot competence evaluation result is generated.

[0081] In the embodiments of the present application, the three grade identifiers are combined according to the standard format, and the corresponding grade description text is added to generate a pilot competence evaluation result document containing complete evaluation conclusions.

[0082] The following is a specific example: In the approach and landing training scene of the A airline pilot training center, based on the operation mode classification result 0.76, the state prediction result 0.62 and the load prediction result 0.85 obtained in the foregoing embodiment, first, the weight coefficient set corresponding to the approach stage is obtained as [0.5, 0.3, 0.2], wherein the three values respectively represent the weights of the operation mode, the attention state and the workload; according to the weight coefficient set, first weighting calculation is performed, using the operation mode weight 0.5 multiplied by the classification result 0.76 to obtain the technical ability score 0.38, the calculation formula is technical ability score = operation mode weight x classification result; second weighting calculation is performed, using the attention state weight 0.3 multiplied by the state prediction result 0.62 to obtain the attention state score 0.186; third weighting calculation is performed, using the workload weight 0.2 multiplied by the load prediction result 0.85 to obtain the workload score 0.17; input the three scores into the comprehensive evaluation function, wherein the basic ability layer evaluation value = 50 + 50 x technical ability score = 50 + 50 x 0.38 = 69, the dynamic performance layer evaluation value = 30 + 30 x attention state score = 30 + 30 x 0.186 = 35.58, and the risk warning layer evaluation value = 20 + 20 x workload score = 20 + 20 x 0.17 = 23.4; according to the threshold range 60-80 for good, 30-60 for improvement and 20-40 for low risk set in the grade mapping rule, the basic ability layer evaluation value 69 is mapped to the good grade, the dynamic performance layer evaluation value 35.58 is mapped to the improvement grade, and the risk warning layer evaluation value 23.4 is mapped to the low risk grade; finally, based on the three psychological evaluation indexes, a formal pilot competence evaluation result report containing grade description and specific improvement suggestions is generated.

[0083] In the embodiments of the present application, through the dynamic weight distribution and multi-level conversion mechanism, accurate mapping from the original prediction data to the structured evaluation result is realized, ensuring that the evaluation result not only retains numerical accuracy but also has grade interpretation, and providing a clear and explicit improvement direction for pilot ability improvement.

[0084] In order to solve the problem of insufficient matching degree between model training data and real scene, in some embodiments, step 105: constructing a competence evaluation model, comprising: Step 401: obtaining an initial training sample set, each initial sample in the training sample set comprising: historical fused multi-modal data, historical attention fixation mode, historical workload index, historical core competence information and historical pilot competence evaluation result.

[0085] In step 401, the historical fusion multi-modal data refers to a multi-source data set that has been time-synchronized in past training. The historical attention fixation pattern refers to a recognized attention abnormality state record. The historical workload index refers to a calculated load quantification value. The historical core competency information refers to a past version of the International Civil Aviation Organization standard. The historical pilot competency assessment result refers to a past generated evaluation conclusion containing three-level indicators.

[0086] In the embodiment of the present application, historical training records are extracted from the training database, each record containing all multi-modal data collected during a complete training period, attention patterns and workload indices obtained through analysis, competency standard documents used at the time, and finally generated assessment result reports, which together constitute an initial training sample set.

[0087] Step 402: Using the analytic hierarchy process, determine the index weight priority of each modality data in the initial sample in different flight stages of the flight target to generate a target sample.

[0088] In step 402, the index weight priority refers to the importance ranking of different data types in a specific flight stage determined through expert evaluation and mathematical calculation, which reflects the contribution difference of each modality data to the evaluation result, and is used for weighted processing of the original sample. The target sample refers to the weighted sample generated by applying weight priority to each modality data in the initial training sample through the analytic hierarchy process. It assigns appropriate weight coefficients to data of different importance, strengthens the influence of key features in the training process, and thus improves the sensitivity and learning efficiency of the model to important features.

[0089] In the embodiment of the present application, the analytic hierarchy process is used to construct a structure model containing target layer, criterion layer and scheme layer. A judgment matrix is established through expert scoring, the consistency ratio is calculated and verified, then the weight vector corresponding to each flight stage is extracted, and the different modality data in the initial sample is weighted to generate a target sample that strengthens important features.

[0090] Step 403: Based on the random forest classifier and the long short-term memory neural network, an initial assessment model is constructed.

[0091] In step 403, the initial assessment model refers to a machine learning model framework that has not been trained, which contains a tree ensemble algorithm for pattern classification and a recurrent neural network structure for time series prediction, and has the ability to process multi-dimensional input and output three-level evaluation results.

[0092] In the embodiments of the present application, the number of trees and depth parameters of the random forest classifier are first configured, then the hidden layer structure and node number of the long short-term memory neural network are set, and finally the two model components are integrated to construct an initial model architecture capable of simultaneously processing classification and prediction tasks. The specific implementation process is as follows: first, the dynamic weight priorities of each evaluation index are determined for different flight stages using the analytic hierarchy process, and on this basis, a hybrid model architecture is constructed by integrating the random forest classifier and the long short-term memory neural network, wherein the random forest classifier is configured with multiple decision trees for identifying high-risk maneuvering modes such as landing bounce trends and other operation characteristics, and the long short-term memory neural network is designed with a gating mechanism for processing time series data such as electrodermal signals to predict emotion stability decline trajectories. Finally, the outputs of the two types of models are integrated through a weighted fusion layer to form a multifunctional evaluation model capable of simultaneously processing classification and prediction tasks.

[0093] Step 404: Based on the target sample, the initial evaluation model is trained until the preset training stopping condition is met, and a competency evaluation model is obtained.

[0094] In step 404, the preset training stopping condition refers to the termination criteria set during the model training process, including maximum iteration number and loss function threshold, etc. indicators, which are used to control the training process and stop at the appropriate time to avoid overfitting.

[0095] In the embodiments of the present application, the target sample is input into the initial evaluation model for forward propagation calculation of the predicted output, the difference between the predicted result and the true label is compared to calculate the loss value, the model parameters are updated through the back propagation algorithm, and when the training round reaches the set value and the loss value is lower than the threshold, the training is stopped to obtain the competency evaluation model ready for use.

[0096] The following is a specific example: In the model building process of the A airline pilot training center, first, an initial training sample set containing 1000 groups of historical training samples is obtained, each group of samples contains a 28-dimensional x 5000 time point matrix of historical fusion multi-modal data in the approach phase, a historical attention fixation mode record containing a duration of 3 seconds and an intensity of 0.88, a historical workload index of 82.3, historical core competence information, that is, ICAO standard documents, and historical pilot competence evaluation results containing basic ability layer 78 points, dynamic performance layer 65 points and risk early warning layer 82 points; the index weight priority of each modal data in the approach phase is determined by using the analytic hierarchy process, and the judgment matrix is calculated by expert scoring to obtain the weight of control data 0.4, the weight of physiological data 0.3, the weight of attention data 0.2, and the weight of load data 0.1. The data in the initial sample is weighted to generate a target sample according to these weights, wherein the control data is multiplied by 0.4, the physiological data is multiplied by 0.3, the attention data is multiplied by 0.2, and the load data is multiplied by 0.1. Based on the random forest classifier, 100 decision trees are configured to identify high-risk control patterns, and based on the long short-term memory neural network, 128 hidden units are configured to predict emotional stability trends, and an initial evaluation model is built. The initial evaluation model is trained using the target sample, and the maximum number of iterations is set to 200 rounds. When the loss function value reaches 0.05, the training is stopped, wherein L represents the loss function value, N represents the number of samples, represents the true label value of the th sample, represents the predicted value of the th sample, and finally a competence evaluation model with a verification set accuracy of 0.92 is obtained. The model can accurately output three-level evaluation indicators consistent with the historical evaluation results.

[0097] In the embodiments of the present application, by means of historical data weighting processing and hybrid model training, an evaluation model that can accurately reflect the characteristics of different flight phases is built, which not only retains historical experience but also integrates the importance difference of multi-dimensional features, thereby improving the accuracy and reliability of the model in actual application.

[0098] In order to solve the rationality problem of multi-modal data weight distribution, in some embodiments, step 402: the analytic hierarchy process is used to determine the index weight priority of each modal data in the initial sample in different flight phases of the flight target to generate a target sample, comprising: Step 501: build a hierarchical structure model, the target layer of the hierarchical structure model is the flight phase competence evaluation result, the criterion layer contains the fusion multi-modal data, the attention fixation mode, the workload index and the core competence information, and the scheme layer is different flight phases.

[0099] In step 501, the hierarchical structure model refers to a tree structure that decomposes a complex decision-making problem into a target layer, a criterion layer, and a scheme layer, wherein the target layer represents the final goal flight phase competency evaluation result to be achieved, the criterion layer includes multi-dimensional factor fusion multi-modal data, attention fixation mode, workload index, and core competency information that affect the target, and the scheme layer includes all possible flight phase options.

[0100] In the embodiment of the present application, first, it is clear that the evaluation target accurately reflects the flight phase competency level, then four criterion layer elements, i.e., multi-modal data comprehensiveness, attention mode characteristics, load level index, and capability standard compliance, are determined, and finally all flight phase types that need to be supported are listed to form a complete hierarchical structure model.

[0101] Step 502: constructing a first judgment matrix of each element of the criterion layer with respect to the target layer, and a second judgment matrix of each element of the scheme layer with respect to each element of the criterion layer.

[0102] In step 502, the first judgment matrix refers to an importance comparison matrix of each element of the criterion layer with respect to the target layer. The second judgment matrix refers to an importance comparison matrix of each flight phase of the scheme layer with respect to each element of the criterion layer, both of which are constructed in a pairwise comparison manner.

[0103] In the embodiment of the present application, a plurality of flight experts are invited to use the 1-9 scale method to perform pairwise importance comparison on the four elements of the criterion layer to generate the first judgment matrix, and at the same time, the importance of each flight phase with respect to each element of the criterion layer is compared to generate a plurality of second judgment matrices.

[0104] Step 503: calculating the maximum eigenvalue and the corresponding eigenvector of each judgment matrix, and calculating a consistency ratio based on the maximum eigenvalue.

[0105] In step 503, the maximum eigenvalue refers to the principal eigenvalue of the judgment matrix. The eigenvector is a vector corresponding to the maximum eigenvalue. The consistency ratio is a ratio obtained by comparing the consistency index of the judgment matrix with the random consistency index, which is used to check the consistency degree of the judgment logic.

[0106] In the embodiment of the present application, the eigenvalue and the eigenvector of each judgment matrix are calculated, the maximum eigenvalue is selected and the corresponding consistency index is calculated, and then compared with the random consistency index of the same order matrix to obtain the consistency ratio value.

[0107] Step 504: when the consistency ratio is less than a preset ratio threshold, normalizing the eigenvector to obtain the effective weight vector of each layer.

[0108] In step 504, the preset ratio threshold is an acceptable upper limit of consistency level set in advance. The effective weight vector is the normalized feature vector after passing the consistency test, representing the reasonable weight distribution of each element in the corresponding level.

[0109] In the embodiment of the present application, when the calculated consistency ratio is less than 0.1, the feature vector is normalized to make the sum of each component 1, and an effective weight vector reflecting the relative importance of each element is obtained.

[0110] Step 505: Extracting a scheme layer weight vector and a criterion layer weight vector from the effective weight vector.

[0111] In step 505, the scheme layer weight vector refers to the weight set of each flight phase in the scheme layer relative to the total target, and the criterion layer weight vector refers to the weight set of each element in the criterion layer relative to the total target, both of which are standardized weight values extracted from the effective weight vector.

[0112] In the embodiment of the present application, the scheme layer weight vector representing the weight of the flight phase and the criterion layer weight vector representing the weight of the criterion element are extracted from the weight vector passing the test, providing a basis for subsequent combination calculation.

[0113] Step 506: According to the current flight phase of the flight target, extracting a corresponding phase weight coefficient from the scheme layer weight vector.

[0114] In step 506, the phase weight coefficient refers to the specific weight value corresponding to the current processed flight phase in the scheme layer weight vector, reflecting the relative importance of the flight phase among all phases.

[0115] In the embodiment of the present application, according to the type of the flight phase currently needing to be processed, the corresponding weight value is searched from the scheme layer weight vector as the importance coefficient of the phase.

[0116] Step 507: Combining the phase weight coefficient with the criterion layer weight vector to generate the index weight priority of each modality data in a specific flight phase.

[0117] In step 507, the combination calculation refers to the process of multiplying or weighting the phase weight coefficient with the criterion layer weight vector, and the generated result represents the optimal weight priority of each modal data in a specific flight phase. The specific flight phase refers to all flight phase types (such as take-off, climb, cruise, approach, landing, etc.) predefined in the analytic hierarchy process scheme layer, which is a static set containing all possible phases; while the "current flight phase" refers to a specific flight phase in which the flight target is currently located, which is an instance in the "specific flight phase" set; the two are the relationship between abstract definition and specific instance, and the scheme layer weight vector contains the weight data of all "specific flight phases", and the corresponding specific weight value needs to be extracted from this vector according to the "current flight phase" when the combination calculation is performed.

[0118] In the embodiment of the present application, the phase weight coefficient is multiplied with each component in the criterion layer weight vector in turn to obtain the combined weight value of each criterion element for the current flight phase, forming a complete weight priority sequence.

[0119] Step 508: According to the index weight priority, the numerical values of the corresponding modal data in the initial sample are weighted to generate a target sample.

[0120] In step 508, the weighting process refers to the process of coefficient adjustment of the numerical values of the corresponding modal data in the initial sample according to the weight priority, so as to generate an optimized target sample by emphasizing important features and weakening secondary features.

[0121] In the embodiment of the present application, the different types of data in the initial sample are multiplied by the corresponding weight coefficients according to the calculated weight priority, to generate a target sample set that emphasizes important features.

[0122] The following is a specific example: In the model construction process of the pilot training center of A airline, the analytic hierarchy process is used to determine the index weight priority of each modal data in the approach phase. First, a hierarchical structure model is constructed, in which the target layer is the flight phase competence evaluation result, the criterion layer contains four elements of fusion multi-modal data, attention fixation mode, workload index and core competence information, and the scheme layer contains five flight phases including the approach phase; five flight experts are invited to construct the first judgment matrix M 1 [[1, 3, 5, 7], [1 / 3, 1, 4, 6], [1 / 5, 1 / 4, 1, 3], [1 / 7, 1 / 6, 1 / 3, 1]] represents the importance comparison of each element in the criterion layer to the target layer, and the second judgment matrix M 2[[1, 2, 4, 5], [1 / 2, 1, 3, 4], [1 / 4, 1 / 3, 1, 2], [1 / 5, 1 / 4, 1 / 2, 1]] represents the importance comparison of each element of the approach phase alignment criterion layer; the maximum eigenvalue of the first judgment matrix is calculated = 4.12 and the corresponding eigenvector V1 = [0.58, 0.28, 0.10, 0.04], wherein the maximum eigenvalue calculation formula is , and the eigenvector is obtained by solving ; the consistency index is calculated , wherein CI represents the consistency index, n represents the order of the judgment matrix, M represents the judgment matrix, and the numerical value is substituted , the random consistency index RI = 0.89 is obtained by looking up the table, the consistency ratio , the numerical value is substituted , and 0.045 is less than the preset threshold 0.1; the eigenvector V 1 is normalized to obtain the effective weight vector of the criterion layer = [0.58, 0.28, 0.10, 0.04]; the scheme layer weight vector W s = [0.46, 0.28, 0.16, 0.10] is also calculated from the second judgment matrix; according to the current flight phase being the approach phase, the corresponding phase weight coefficient 0.46 is extracted from W s ; the phase weight coefficient 0.46 is combined with the criterion layer weight vector to calculate the index weight priority of each modal data in the approach phase ; the values of the corresponding modal data in the initial sample are weighted according to the weight priority, wherein the fusion multi-modal data is multiplied by 0.2668, the attention fixation mode data is multiplied by 0.1288, the workload index is multiplied by 0.046, and the core competence information is multiplied by 0.0184, and finally the target sample emphasizing important features is generated for subsequent model training.

[0123] In the embodiments of the present application, the weight priority of each modal data in different flight phases is scientifically determined by the analytic hierarchy process, the rationalization and standardization of data weighting are realized, the target sample with clear feature importance distinction is provided for model training, and the accuracy and reliability of the subsequent evaluation model are improved.

[0124] In order to solve the problem of insufficient attention state recognition accuracy, in some embodiments, step 103: the fusion multi-modal data is analyzed based on a hidden Markov model to identify the attention fixation mode, comprising: Step 601: extracting a visual attention point sequence and an operation response sequence from the fusion multi-modal data.

[0125] In step 601, the visual focus point sequence refers to the coordinate record of the pilot's gaze position changing over time during the driving process, containing screen coordinates and duration information of the gaze point. The operation response sequence refers to the timestamp record of the pilot operating the control device, containing parameters such as operation type and response delay time.

[0126] In the embodiment of the present application, the gaze point coordinate sequence collected by the eye tracker and the operation lever action time sequence are extracted from the fused multi-modal data, and the two sequences are time-aligned to ensure that each time point contains corresponding visual focus point and operation response data.

[0127] Step 602: input the visual focus point sequence and the operation response sequence as observation sequences into the hidden Markov model, calculate the probability of being in each hidden state at each time under the observation sequence through the forward-backward algorithm in the hidden Markov model, and the hidden Markov model contains three hidden states of dispersion state, concentration state and fixation state.

[0128] In step 602, the forward-backward algorithm is a probability calculation method in the hidden Markov model, which is used to calculate the probability of being in different hidden states at each time under the given observation sequence. The hidden state refers to the cognitive state that cannot be directly observed, including the dispersion state representing inattention, the concentration state representing normal concentration, and the fixation state representing excessive concentration. The dispersion state as the basic reference state in the attention state transition model, its core function is to provide a probability comparison benchmark for identifying the concentration state and the fixation state; by continuously calculating the background probability of the system being in the dispersion state, the model can accurately judge whether the current observation sequence deviates from the benchmark state, thereby triggering the judgment of the concentration or fixation state, so the dispersion state is a necessary component for constructing the state transition probability matrix and realizing effective pattern recognition.

[0129] In the embodiment of the present application, the visual focus point sequence and the operation response sequence are input as observations, the forward algorithm is used to calculate the probability accumulation from the front to the back, the backward algorithm is used to calculate the probability accumulation from the back to the front, and the probability distribution of being in three hidden states at each time point is calculated by combining the results of the two.

[0130] Step 603: when the probability of the concentration state or the fixation state exceeds a preset probability threshold, it is determined that the attention fixation mode occurs.

[0131] In step 603, the preset probability threshold is a critical value determined according to historical data statistical analysis, which is used to judge whether the attention state is abnormal.

[0132] In this embodiment of the application, the state probability value at each time point is monitored in real time. When the probability of the concentrated state or the fixed state exceeds the set threshold for multiple consecutive time steps, it is determined that an attention fixed mode has occurred, and the start time, duration and intensity index of the mode are recorded.

[0133] Here is a specific example: In the approach and landing training scenario at the A Airlines pilot training center, the fused multimodal data matrix generated based on the aforementioned embodiment contains 5000 time points, each with 28 dimensions. First, the visual attention point sequence (i.e., the sequence of duration of gaze at the instrument panel) and the operation response sequence (i.e., the operation response delay sequence) are extracted from this matrix. These two sequences are then used as observation sequences and input into a pre-trained Hidden Markov Model (HMM). This model contains three hidden states: dispersed state, concentrated state, and fixed state. The transition probability matrix A = [[0.7,0.2,0.1],[0.3,0.6,0.1],[0.2,0.2,0.6]] represents the probability of transitions between states, and the emission probability matrix B uses a Gaussian distribution to describe the relationship between observations and states. The state probability at each time point is calculated using a forward-backward algorithm, where the forward probability calculation formula is... ,in Indicates time Hidden The forward probability, Indicates time -1 is hidden. The forward probability, Indicates from hidden state Move to hidden state The probability, Indicates hidden state Observed value The probability, Indicates time The observed values. The formula for calculating the backward probability is: ,in Indicates time Hidden The backward probability, Indicates time +1 is hidden. The forward probabilities were calculated. At the 15th second, the probability of a focused state was 0.92, the probability of a fixed state was 0.05, and the probability of a distracted state was 0.03. When the probability of a focused state exceeded the preset threshold of 0.85 for 4 consecutive seconds from the 15th to the 18th second, an attention fixation mode was determined to have occurred. The start time was recorded as the 15th second, the duration as 4 seconds, and the average probability as 0.93. This recognition result, together with the subsequent workload index of 82.3, constituted the evaluation input data.

[0134] In the embodiments of the present application, the dynamic analysis of multi-modal data is performed by using a hidden Markov model, thereby realizing accurate recognition of attention states and abnormal pattern detection, providing an effective technical means for pilot state monitoring, and improving the intelligent level of flight safety management.

[0135] To solve the problem of insufficient pertinence of training reports, in some embodiments, step 106: generating an individualized training report based on the pilot competency evaluation result, includes: Step 701: extracting a weak link indicator from the pilot competency evaluation result.

[0136] In step 701, the weak link indicator refers to the ability dimension score lower than the preset threshold in the evaluation result, reflecting the ability short board that needs to be improved by the pilot, including the score items that do not meet the standard in each level evaluation value and the gap with the standard value.

[0137] In the embodiments of the present application, the system automatically scans all score items in the three-level evaluation result, marks the indicators lower than the corresponding threshold as weak links, and records the difference between the specific score and the threshold.

[0138] Step 702: matching the weak link indicator with the flight scene in the preset historical flight database to determine the target scene that needs to be strengthened.

[0139] In step 702, the historical flight database refers to a database storing past training records and scene characteristics. The target scene refers to the training environment setting most matched with the current weak link, and the flight situation that needs to be trained is determined by similarity calculation.

[0140] In the embodiments of the present application, the weak link indicator is matched with the scene characteristics in the database for matching calculation, and the historical training scene with the highest similarity is selected as the target scene, ensuring that the training content is highly related to the ability short board.

[0141] Step 703: reconstructing the attention allocation waveform and cognitive load curve under the target scene according to the time series data of the attention fixation pattern and the time series data of the workload index.

[0142] In step 703, the time sequence data of the attention fixation mode refers to the sequence data of the pilot's attention state change recorded in time sequence based on the analysis of the hidden Markov model, which contains the continuous record of the attention state type (dispersion / concentration / fixation) and its duration. The time sequence data of the workload index refers to the sequence of cognitive load quantitative values recorded in time sequence, which is a continuous numerical sequence generated by fusing the subjective scale data and the physiological entropy value data. The attention allocation waveform refers to the curve of the change of the attention state with time, and the cognitive load curve refers to the trend graph of the change of the workload index with time, which together reflect the change process of the cognitive state of the pilot in a specific scene.

[0143] In the embodiment of the present application, the historical time sequence data of the attention fixation mode and the workload index is extracted, intercepted and reorganized according to the time period of the target scene, to generate a visual curve reflecting the change of the cognitive state in the scene.

[0144] Step 704: identifying the attention abnormal period meeting the preset first abnormal identification condition from the reconstructed attention allocation waveform, and identifying the cognitive load over-limit period meeting the preset second abnormal identification condition from the reconstructed cognitive load curve.

[0145] In step 704, the first abnormal identification condition refers to the judgment standard of the attention state abnormality, specifically, when the fixation state continuously appears in the attention allocation waveform for more than a preset time threshold, or the frequency of the concentration state is lower than a preset frequency threshold, it is identified as an attention abnormal period. The second abnormal identification condition refers to the judgment standard of the cognitive load over-limit, specifically, when the value of the cognitive load curve continuously exceeds a preset load threshold, it is identified as a cognitive load over-limit period. The attention abnormal period refers to the time period of the continuous abnormality of the attention state, and the cognitive load over-limit period refers to the time period of the continuous over-limit of the load level.

[0146] In the embodiment of the present application, the reconstructed waveform and curve are scanned and detected to identify the time period meeting the attention continuous fixation condition and the time period of the load value continuously exceeding the threshold, and the start and end time of these abnormal periods are recorded.

[0147] Step 705: labeling the operation action corresponding to the attention abnormal period and the cognitive load over-limit period as an operation item to be improved.

[0148] In step 705, the operation item to be improved refers to the specific operation action corresponding to the abnormal period, which is determined by analyzing the operation record to improve the operation details, and provides clear target content for training.

[0149] In the embodiment of the present application, the operation record is queried according to the abnormal period timestamp, and all operation actions in the time period are extracted and labeled as operation items that need to be focused on and improved in training.

[0150] Step 706: generating a personalized training report according to the target scene, the operation item to be improved, and the expected training target.

[0151] In step 706, the expected training target refers to a quantifiable ability improvement index determined by comparing the weak link index with the historical flight database, including the target achievement value and the target improvement degree, which is derived from the difference analysis of the industry standard value and the personal historical baseline value.

[0152] In the embodiments of the present application, the training target value is set according to the gap between the weak link index and the standard value, and a complete training plan including training content, training duration and standard is generated in combination with the target scene and the operation item to be improved.

[0153] The following is a specific example: In the approach and landing training scene of the pilot training center of A airline, the pilot competence evaluation result obtained based on the foregoing embodiments includes 78 points in the basic ability layer, 65 points in the dynamic performance layer, and 82 points in the risk warning layer. First, extract the dynamic performance layer of 65 points and the risk warning layer of 82 points from the result as weak link indexes, wherein the dynamic performance layer is 5 points lower than the threshold value of 70 points, and the risk warning layer is 7 points higher than the threshold value of 75 points; match these indexes with the historical flight database, and calculate the similarity degree by the similarity calculation formula wherein S represents the scene matching similarity, , the weight coefficients are all taken as 0.5, the numerical similarity, the feature similarity, and the matching degree with the bad weather approach scene is calculated to be 0.85, which is determined as the target scene; according to the state probability of 0.85-0.92 in the 15-18 second concentrated state data and the value of 75-88 in the 10-20 second working load index time series data recorded, the attention allocation waveform under the target scene is reconstructed to present a continuous high value and an upward trend of the cognitive load curve; the period of 15.5-17.5 seconds during which the concentration state probability continuously exceeds 0.9 is identified as the attention abnormal period from the reconstructed attention allocation waveform, and the period of 16-18 seconds during which the load value continuously exceeds 85 is identified as the cognitive load over-limit period from the cognitive load curve; the throttle control delay of 1.5 seconds and the insufficient heading correction of 2 degrees corresponding to the two periods are marked as the operation items to be improved by querying the operation record; finally, according to the target scene of bad weather approach, the operation items to be improved of throttle control and heading correction, and the expected training target of improving the dynamic performance layer to more than 70 points, a personalized training report including 5 hours of instrument identification training and 3 hours of load control training is generated, wherein the instrument identification training focuses on improving the gaze range coverage problem, and the load control training focuses on practicing the operation stability under stress situation.

[0154] In the embodiment of the present application, the training efficiency and the ability improvement effect of the pilot are effectively improved by accurately identifying the ability short board and matching the corresponding training scene, combining the positioning and analysis of specific operation problems, and generating a highly targeted and personalized training plan.

[0155] Figure 4 The structural diagram of a pilot competence dynamic evaluation system based on multi-modal data provided in the embodiment of the present application is described in the specific implementation part: The acquisition module 41 is configured to acquire flight control data, physiological signal data, psychological test data, subjective scale data, and core competence information defined by the International Civil Aviation Organization when the pilot operates the flight target.

[0156] The alignment module 42 is configured to perform alignment processing on the flight control data, the physiological signal data, and the psychological test data by using a time synchronization algorithm to obtain fused multi-modal data.

[0157] The analysis module 43 is configured to analyze the fused multi-modal data based on a hidden Markov model to identify an attention fixation mode.

[0158] The fusion module 44 is configured to fuse the subjective scale data and the physiological entropy value data in the physiological signal data to construct a workload index.

[0159] The input module 45 is configured to input the fused multi-modal data, the attention fixation mode, the workload index, and the core competence information into a pre-trained competence evaluation model, and output a pilot competence evaluation result by the competence evaluation model, wherein the pilot competence evaluation result includes three-level psychological evaluation indexes of a basic ability layer, a dynamic performance layer, and a risk warning layer.

[0160] The generation module 46 is configured to generate a personalized training report based on the pilot competence evaluation result.

[0161] The pilot competence dynamic evaluation system based on multi-modal data in the embodiment of the present application is used to implement the aforementioned pilot competence dynamic evaluation method based on multi-modal data, and therefore the specific implementation part of the pilot competence dynamic evaluation system based on multi-modal data can refer to the embodiment part of the pilot competence dynamic evaluation method based on multi-modal data in the foregoing, and the specific implementation part can refer to the description of the corresponding embodiment part, which will not be described here.

[0162] The present application also provides an electronic device comprising a memory for storing a computer program and a processor for executing the computer program to implement the steps of the pilot competence dynamic evaluation method based on multi-modal data described above.

[0163] The application further provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the pilot competence dynamic evaluation method based on multi-modal data.

[0164] In an example embodiment, the computer readable storage medium can include, but is not limited to, a U disk, a read-only memory, a random access memory, a mobile hard disk, a magnetic disk or an optical disk, and various media capable of storing a computer program.

[0165] The embodiment of the application further provides a computer program product, wherein the computer program product includes a computer program, and the computer program is executed by a processor to implement the steps in the pilot competence dynamic evaluation method based on multi-modal data.

[0166] The skilled person can further realize that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be realized by electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of the examples have been described in the above description in general terms. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the application.

[0167] The above describes in detail the pilot competence dynamic evaluation method, system, device and storage medium provided by the application. The principle and implementation of the application are described by using specific examples. The above description of the examples is only used to help understand the method of the application and its core idea. It should be pointed out that for ordinary skilled person in the art, without departing from the principle of the application, the application can be improved and modified in several ways, and these improvements and modifications also fall within the protection scope of the application.

Claims

1. A pilot competency dynamic assessment method based on multi-modal data, characterized in that, The method comprises the following steps: Collecting flight control data, physiological signal data, psychological test data, subjective scale data, and core competency information defined by the International Civil Aviation Organization when a pilot operates a flight target; Aligning the flight control data, physiological signal data, and psychological test data using a time synchronization algorithm to obtain fused multi-modal data; Based on a hidden Markov model, the fused multi-modal data is analyzed to identify an attention fixation mode; Fusing the subjective scale data and the physiological entropy data in the physiological signal data to construct a workload index; The fused multi-modal data, the attention fixation mode, the workload index, and the core competency information are input into a pre-trained competency evaluation model, and a pilot competency evaluation result is output by the competency evaluation model, which includes three-level psychological evaluation indicators of a basic ability layer, a dynamic performance layer, and a risk warning layer; Based on the pilot competency evaluation result, a personalized training report is generated. 2.The method for dynamically evaluating pilot competency based on multi-modal data according to claim 1, characterized in that, The input of the fused multi-modal data, the attention fixation mode, the workload index, and the core competency information into the pre-trained competency evaluation model to output the pilot competency evaluation result by the competency evaluation model comprises: The core competency information is input into the pre-trained competency evaluation model as an evaluation benchmark by the mapping establishment module of the competency evaluation model, and the evaluation benchmark is respectively mapped with the fused multi-modal data to establish a first index mapping relationship, with the attention fixation mode to establish a second index mapping relationship, and with the workload index to establish a third index mapping relationship; The pattern classifier in the competency evaluation model performs pattern recognition and classification based on the first index mapping relationship to obtain an operation mode classification result; The time series predictor in the competency evaluation model performs state prediction based on the second index mapping relationship to obtain a state prediction result, and performs load prediction based on the third index mapping relationship to obtain a load prediction result; The classification result, state prediction result, and load prediction result are weighted and fused to generate a pilot competency evaluation result.

3. The method for pilot proficiency dynamic assessment based on multi-modal data according to claim 2, characterized in that, The weighting and fusion of the classification result, state prediction result, and load prediction result to generate the pilot competency evaluation result comprises: Obtaining a set of weight coefficients of each evaluation index corresponding to the flight phase of the flight target; According to the set of weight coefficients, the classification result is first weighted and calculated to obtain a technical ability score; According to the set of weight coefficients, the state prediction result is second weighted and calculated to obtain an attention state score; According to the set of weight coefficients, the load prediction result is third weighted and calculated to obtain a workload score; inputting the technical ability score, the attention state score and the workload score into a preset comprehensive evaluation function to generate a basic ability layer evaluation value, a dynamic performance layer evaluation value and a risk early warning layer evaluation value; according to a preset level mapping rule, converting the basic ability layer evaluation value, the dynamic performance layer evaluation value and the risk early warning layer evaluation value into corresponding psychological evaluation indexes; generating a pilot competence evaluation result based on each psychological evaluation index. 4.The method for dynamically evaluating pilot competency based on multi-modal data according to claim 1, wherein, The competence evaluation model is constructed, including: obtaining an initial training sample set, each initial sample in the training sample set including: historical fusion multi-modal data, historical attention fixation mode, historical workload index, historical core competence information and historical pilot competence evaluation result; determining the index weight priority of each modal data in the initial sample in different flight stages of the flight target by using the analytic hierarchy process to generate a target sample; constructing an initial evaluation model based on a random forest classifier and a long short-term memory neural network; training the initial evaluation model based on the target sample until a preset training stop condition is met to obtain the competence evaluation model.

5. The method for pilot proficiency dynamic assessment based on multi-modal data according to claim 4, characterized in that, The method of determining the index weight priority of each modal data in the initial sample in different flight stages of the flight target by using the analytic hierarchy process to generate a target sample includes: constructing a hierarchical structure model, the target layer of the hierarchical structure model being a flight stage competence evaluation result, the criterion layer including the fusion multi-modal data, the attention fixation mode, the workload index and the core competence information, and the scheme layer being different flight stages; constructing a first judgment matrix of each element of the criterion layer to the target layer and a second judgment matrix of each element of the scheme layer to each element of the criterion layer; calculating the maximum eigenvalue of each judgment matrix and the corresponding eigenvector, and calculating a consistency ratio based on the maximum eigenvalue; when the consistency ratio is less than a preset ratio threshold, normalizing the eigenvector to obtain an effective weight vector of each level; extracting a scheme layer weight vector and a criterion layer weight vector from the effective weight vector; extracting a corresponding stage weight coefficient from the scheme layer weight vector according to the current flight stage of the flight target; combining and calculating the stage weight coefficient and the criterion layer weight vector to generate the index weight priority of each modal data in a specific flight stage; weighting the values of the corresponding modal data in the initial sample according to the index weight priority to generate a target sample.

6. The method for pilot proficiency dynamic assessment based on multi-modal data according to claim 1, wherein, The method of analyzing the fusion multi-modal data based on the hidden Markov model to identify the attention fixation mode includes: extracting a visual attention point sequence and an operation response sequence from the fusion multi-modal data; inputting the visual attention point sequence and the operation response sequence as observation sequences into a hidden Markov model, calculating the probability of being in each hidden state at each time under the observation sequences by a forward-backward algorithm in the hidden Markov model, the hidden Markov model including three hidden states of dispersion state, concentration state and fixation state; When the probability of the concentration state or the fixation state exceeds a preset probability threshold, it is determined that the attention fixation mode occurs.

7. The method for pilot proficiency dynamic assessment based on multi-modal data according to claim 1, characterized in that, The personalized training report is generated based on the pilot competency evaluation result, including: Weak link indicators are extracted from the pilot competency evaluation result; The weak link indicators are matched with flight scenes in a preset historical flight database to determine a target scene that needs to be reinforced for training; According to the timing data of the attention fixation mode and the timing data of the workload index, the attention allocation waveform and the cognitive load curve under the target scene are reconstructed; From the reconstructed attention allocation waveform, an attention abnormal period that meets a preset first abnormal identification condition is identified, and from the reconstructed cognitive load curve, a cognitive load over-limit period that meets a preset second abnormal identification condition is identified; The operation actions corresponding to the attention abnormal period and the cognitive load over-limit period are marked as to-be-improved operation items; According to the target scene, the to-be-improved operation items, and the expected training target, a personalized training report is generated.

8. A pilot competency dynamic assessment system based on multi-modal data, characterized in that, Comprising: The acquisition module is used to collect flight control data, physiological signal data, psychological test data, subjective scale data, and core competency information defined by the International Civil Aviation Organization when the pilot operates the flight target; The alignment module is used to perform alignment processing on the flight control data, the physiological signal data, and the psychological test data by using a time synchronization algorithm to obtain fused multi-modal data; The analysis module is used to analyze the fused multi-modal data based on a hidden Markov model to identify an attention fixation mode; The fusion module is used to fuse the subjective scale data and the physiological entropy value data in the physiological signal data to construct a workload index; The input module is used to input the fused multi-modal data, the attention fixation mode, the workload index, and the core competency information into a pre-trained competency evaluation model, and output a pilot competency evaluation result by the competency evaluation model, wherein the pilot competency evaluation result includes three-level psychological evaluation indicators of a basic ability layer, a dynamic performance layer, and a risk warning layer; The generation module is used to generate a personalized training report based on the pilot competency evaluation result.

9. An electronic device, comprising: Comprising: A memory for storing a computer program; A processor for executing the computer program to implement the steps of the pilot competency dynamic evaluation method based on multi-modal data according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the pilot competency dynamic evaluation method based on multi-modal data according to any one of claims 1 to 7.

Images