Civil aviation crew cognitive state difference evaluation method and system based on LSTM-MHSA algorithm and hierarchical coupling framework

Abstract

The application belongs to the field of flight safety, and discloses a civil aviation crew cognitive state difference evaluation method and system based on an LSTM-MHSA algorithm and a hierarchical coupling framework, which comprises the following steps: S1: collecting the brain electrical and psychological cognitive data of pilots before and after the execution of a flight task, and reconstructing the scalarized brain electrical features and psychological cognitive features into a unified individual cognitive ability feature multivariate time sequence sample set after pretreatment; S2: constructing a deep learning model LSTM-MHSA to extract the individual cognitive ability time evolution features of pilots and the cognitive interaction fusion features of the captain and the first officer; S3: using the LSTM-MHSA model to evaluate the overall ability score of pilots in different psychological cognitive dimensions and the psychological cognitive coordination mode of the captain and the first officer during the flight task, so as to realize the difference evaluation of the civil aviation crew cognitive state. The application aims to comprehensively evaluate the psychological cognitive ability of pilots from the individual and crew levels.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of flight safety, and particularly relates to a civil aviation crew cognitive state difference evaluation method and system based on an LSTM-MHSA algorithm and a hierarchical coupling framework. BACKGROUND

[0002] In the field of aviation, the operation accuracy and psychological state of pilots are directly related to flight safety. In recent years, with the development of neuroscience and related technologies, electroencephalogram (EEG) data has shown great potential in the evaluation of pilot psychological state and psychological cognitive ability. EEG data can reflect the activity state of the brain in real time and objectively, and through specific EEG indicators, the fatigue degree, attention concentration, emotional state and psychological cognitive load of pilots can be evaluated. Due to the high sensitivity and objectivity of EEG data, it is regarded as the "gold standard" for evaluating the psychological state and psychological cognitive ability of pilots. However, relying solely on EEG data to monitor the state of pilots has several limitations. Although EEG data can capture real-time brain activity information, it is easily disturbed by environmental noise and is difficult to directly and accurately map the specific performance of pilots when performing specific flight tasks, especially considering the inconvenience and safety hazards caused by wearing EEG equipment during flight. At the same time, psychological scale data as a key source of information can comprehensively quantify the psychological state and psychological cognitive ability of pilots. These scale data are usually collected in non-flight environments and provide scientific and objective benchmarks for the evaluation of pilot mental health. Specifically, the main shortcomings of EEG data are its susceptibility to interference, weak direct correlation and high professional threshold for interpretation. While psychological scale data can provide valuable psychological evaluation in non-flight states, they lack real-time and direct reflection of specific flight situations.

[0003] The fusion analysis of the two aims to solve the following problems: first, through the complement of multi-source data, the comprehensiveness and accuracy of the pilot state evaluation are improved; second, the stability of the psychological scale data is used to assist in interpreting the dynamic changes of the electroencephalogram data, and the dependence on professional interpretation is reduced; third, the correlation analysis of the psychological state before and after the flight and the real-time brain activity is realized, and more scientific support is provided for the comprehensive evaluation and training of the pilot. Therefore, the present application proposes an innovative electroencephalogram measurement scheme, that is, the electroencephalogram measurement is performed before and after the pilot flight. This method not only maximally reduces the intervention on the pilot during the flight task and improves the comfort of measurement, but also can capture the psychological state changes of the pilot before and after the flight, thereby providing more comprehensive and accurate data support for flight safety evaluation. Through the fusion of the electroencephalogram data and the psychological scale data, this method can fully utilize the advantages of various data and realize the evaluation and identification of the psychological and psychological cognitive ability state of the pilot. However, at present, there is still a lack of an effective technical means to realize this method, and there are many challenges in feature extraction, data fusion and error identification: the traditional electroencephalogram processing method has problems such as redundant feature extraction, high computational complexity or poor decomposition effect; the long short-term memory network has good ability to process sequence data, but adding a simple feature weighting strategy may not be able to fully capture the complex relationship in the input data, resulting in a decrease in recognition accuracy; and the traditional network parameter optimization methods such as grid search and random search may not be able to find the global optimal solution in a short time, or the optimization effect is limited. SUMMARY

[0004] To solve the problems in the prior art, the present application provides a civil aviation crew cognitive state difference evaluation method and system based on an LSTM (Long Short-Term Memory)-MHSA (Multi-Head Self-Attention) algorithm and a hierarchical coupling framework, which constructs a “individual-crew” double-layer evaluation model through deep fusion of individual psychological cognitive ability and crew psychological cognitive interaction, forms a closed-loop solution scheme running through pre-event prediction, in-event monitoring and post-event evaluation, and realizes the difference evaluation of the civil aviation crew cognitive state.

[0005] To achieve the above object, the present application provides the following scheme:

[0006] A civil aviation crew cognitive state difference evaluation method based on an LSTM-MHSA algorithm and a hierarchical coupling framework, the method comprising:

[0007] S1: collecting the electroencephalogram and psychological cognitive data of the pilot before and after the flight task execution, respectively, and after preprocessing, reconstructing the scalarized electroencephalogram features and psychological cognitive features to form a unified individual cognitive ability feature multivariate time sequence sample set;

[0008] S2: According to the unified individual cognitive ability characteristic multi-time sequence sample set, a deep learning model LSTM-MHSA combining LSTM and multi-head self-attention mechanism is constructed to extract the time evolution characteristics of the individual cognitive ability of the pilot and the cognitive interaction fusion characteristics of the captain and the co-pilot;

[0009] S3: According to the time evolution characteristics of the individual cognitive ability of the pilot and the cognitive interaction fusion characteristics of the captain and the co-pilot, the LSTM-MHSA model is used to evaluate the overall ability score of the pilot in different psychological cognitive dimensions and the psychological cognitive coordination mode of the captain and the co-pilot during the flight task, so as to realize the difference evaluation of the cognitive state of the civil aviation crew.

[0010] Preferably, in S1, the EEG and psychological cognitive data of the pilot are collected before and after the flight task execution, and after preprocessing, the scalarized EEG features and psychological cognitive features are reconstructed to form a unified individual cognitive ability characteristic multi-time sequence sample set, which includes:

[0011] S1.1: Before and after the flight task, the eight-lead EEG instrument is used to collect the brain cortex electrical signals of the pilot according to the international EEG standard, the key electrode positions are selected, and the conductive paste is used to ensure stable contact, the reference electrode is positioned at the earlobe, and the IND lead can be adjusted;

[0012] S1.2: For the EEG data, 1-60Hz third-order Butterworth band-pass filtering and 50Hz notch filtering are used for denoising, the data is centralized and whitened, ICA technology is used to decompose the signal based on negative entropy, and eye movement artifacts and channel interference are removed, finally, the wavelet threshold denoising method is used to process the signal separated by ICA;

[0013] S1.3: For the test result data of the psychological cognitive module, the data exceeding 3 times the standard deviation range is removed, the missing value is processed by mean interpolation method, the original data is standardized, the min-max standardization method is used to linearly transform the data to the [0, 1] interval, the conversion formula is based on the maximum and minimum values of the data, and finally the normalization processing is performed;

[0014] S1.4: The energy entropy, information entropy, sample entropy and power spectral density are extracted as EEG features by using the variational mode decomposition, the energy entropy of different intrinsic mode functions is calculated, the information entropy is used to evaluate the value of information in the EEG signal, the sample entropy is used to measure the complexity of the time series, and the power spectrum of the signal is estimated by the periodogram method;

[0015] S1.5: The scores of the perception speed, memory span, N-back task accuracy, attention-stroop effect, spatial psychological cognitive error, operation ability and psychological motor ability in the scale test are selected as the core indicators for evaluating the six psychological cognitive dimensions of memory, attention, intelligence, spatial perception ability, reaction speed and professional ability.

[0016] S1.6: The brain electrical and psychological cognitive features obtained from multiple pre-flight and post-flight sampling time points are scalarized to form a unified individual cognitive ability feature multi-time sequence sample.

[0017] Preferably, in S1.4, the application of variational mode decomposition extracts energy entropy, information entropy, sample entropy, and power spectral density as brain electrical features, including:

[0018] Using VMD algorithm, the original brain electrical signal f(t) is decomposed into K

[0019]

[0020]

[0021] Where ω k is the center frequency of the kth IMF, δ(t) is the Dirac function, u k represents the kth intrinsic mode function (IMF), u k (t) represents the instantaneous amplitude of the kth IMF at time t, which is a function of time t, ω k (t) represents the instantaneous frequency of the kth IMF, which is a function of time t, and describes the local frequency characteristics of the IMF at time t, j represents the imaginary unit, i.e. In the complex domain, it is used to describe the analytical representation of the signal, where u k (t) is the IMF time series to be solved, ω k (t) is the corresponding instantaneous frequency, and j is used in the exponential term to construct the analytical signal.

[0022] Based on each IMF, the energy entropy, information entropy, sample entropy, and power spectral density are calculated.

[0023] Preferably, in S1.6, the brain electrical and psychological cognitive features obtained from multiple pre-flight and post-flight sampling time points are scalarized to form a unified individual cognitive ability feature multi-time sequence sample, including:

[0024] Let the cognitive ability observation value of pilot i at the tth sampling time be Where D=10 is the cognitive ability index dimension fused with brain electrical and psychological test, where represents the brain electrical features of pilot i at the tth sampling time, including energy entropy, information entropy, sample entropy, and power spectral density, which are converted to scalar values in the [0,1] interval through standardization processing of the four key features.

[0025] The psychological cognitive test results of the pilot i at the t-th sampling time represent the test scores of six dimensions including memory, attention, intelligence, spatial perception, reaction force and professional ability, and are normalized to a scalar value in the interval [0, 1];

[0026] The scalarized electroencephalogram features The normalized psychological cognitive test scores Splicing, obtaining the complete cognitive ability observation value of the pilot i at the t-th sampling time:

[0027] The complete cognitive ability sequence of the pilot i As an input of the LSTM-MHSA model, where T is the sampling time step.

[0028] Preferably, in S2, according to the unified individual cognitive ability feature multi-time sequence sample set, a deep learning model LSTM-MHSA combining LSTM and multi-head self-attention mechanism is constructed to extract the pilot's personal cognitive ability time sequence evolution feature and the co-pilot cognitive interaction fusion feature, including:

[0029] S2.1: Design a deep learning model LSTM-MHSA combining LSTM and MHSA, which takes multi-layer bidirectional LSTM as the backbone and introduces MHSA to model the internal relationship of the sequence;

[0030] S2.2: The complete psychological cognitive ability sequence of the pilot i is used as the input of the model, and the LSTM-MHSA model is used to hierarchically encode the psychological cognitive ability time sequence data of a single pilot to obtain cognitive feature representations at different scales and different abstraction levels;

[0031] S2.3: Use the MHSA mechanism to model the psychological cognitive interaction relationship between the co-pilot and the co-pilot;

[0032] S2.4: Train the LSTM-MHSA model in an end-to-end manner to realize adaptive extraction from the original multi-element psychological cognitive ability data to the individual-crew level coupled features, use the cross-entropy loss function to measure the deviation of the features extracted by the model from the actual psychological cognitive state of the pilot, use the Adam adaptive optimization algorithm to iteratively update the model parameters, minimize the loss function, and use methods such as Dropout and L2 regularization to prevent model overfitting.

[0033] Preferably, in S2.1, a deep learning model LSTM-MHSA combining LSTM and MHSA is designed, which takes multi-layer bidirectional LSTM as the backbone and introduces MHSA to model the internal relationship of the sequence, including:

[0034] The overall structure of the LSTM-MHSA model is represented as:

[0035] H (0) = Embedding(X)

[0036] H (l) = BiLSTM (l) (H (l-1) ), l = 1, 2, L, L

[0037] U = MHSA(H (L) )

[0038] o = Softmax(FC(U))

[0039] where the Embedding layer maintains a weight matrix, where each row corresponds to a vector representation of a discrete feature. During the training process, the vectors are continuously updated and optimized along with the model parameters, ultimately learning the semantic information of the input features. X = {x1, x2, L, x T} is the original time series data of the pilot's psychological cognitive ability, is the input embedding representation, L is the number of LSTM layers, is the hidden state sequence of the l-th LSTM layer, is the output sequence of MHSA, is the predicted probability distribution of the pilot's psychological cognitive state;

[0040] The backbone of the multi-layer bidirectional LSTM includes:

[0041]

[0042]

[0043]

[0044] where, and represent the forward and backward LSTM units respectively, is the hidden state at time step t, and are the hidden states of is the complete hidden state of the bidirectional LSTM at time t. The bidirectional LSTM can utilize information from multiple tests by processing the sequence in both forward and reverse directions.

[0045] The MHSA is introduced to model the internal relationships between sequences, including:

[0046]

[0047] U = [head1; head2; L; headN ]W O

[0048] wherein, and is a learnable linear transformation matrix, is the output of the nth attention head, N is the number of attention heads, d k = 2d h / N is the hidden layer dimension of each head, Attention(·) The function calculates the normalized dot-product attention:

[0049]

[0050] The multi-head self-attention mechanism learns the mutual relationship between sequence elements in different subspaces by parallel computing multiple attention functions, enhancing the feature representation capability of the model. Finally, the outputs of all heads are spliced and linearly transformed to obtain a high-level feature representation U that integrates different scale dependency relationships.

[0051] Preferably, in S2.2, the complete psychological cognitive ability sequence of the pilot i is As the input of the model, the LSTM-MHSA model is used to hierarchically encode the psychological cognitive ability time series data of a single pilot, obtaining cognitive feature representations at different scales and different abstraction levels, including:

[0052] The psychological cognitive ability sequence is encoded using a bidirectional LSTM network. The forward LSTM updates the hidden state step by step

[0053]

[0054] wherein, is the parameter of the forward LSTM;

[0055] The reverse LSTM updates the hidden state in reverse order

[0056]

[0057] wherein, is the parameter of the reverse LSTM;

[0058] The forward and reverse hidden states are spliced to obtain the bidirectional LSTM encoding at the t-th step

[0059] The bidirectional LSTM encoding sequence is fed into a stacked multi-layer LSTM to realize hierarchical abstraction of psychological cognitive features. The hidden state update formula of the l-th layer LSTM is:

[0060]

[0061] in, represents the output of bidirectional LSTM, θ (l) is the parameter of the lth layer LSTM;

[0062] After stacking L layers of LSTM, the multi-scale psychological cognitive features of pilot i at time t are obtained

[0063] The final psychological cognitive ability characteristics of pilot i at time t are obtained by adaptively aggregating the output of multi-layer LSTM through the gating mechanism.

[0064]

[0065]

[0066] Among them, W g and b g is the parameter of the gating layer, σ(·) is the sigmoid activation function, and e represents the Hadamard product;

[0067] Through bidirectional LSTM, multi-layer LSTM and gated aggregation, multi-scale temporal features that depict the evolutionary pattern of individual pilots' psychological cognitive abilities are extracted.

[0068] Preferably, in S2.3, using the MHSA mechanism to model the psychological cognitive interaction relationship between the pilot and co-pilot within the crew includes:

[0069] Assume that the multivariate psychological cognitive ability characteristics of the pilot and co-pilot in the tth time window are and Then the crew's psychological cognitive ability at time t can be expressed as Where d is the dimension of individual psychological cognitive characteristics, and a complete flight mission is divided into T time windows, that is, the psychological cognitive ability sequence of the crew {s1, s2, L, s T};

[0070] MHSA uses the crew's psychological cognitive ability sequence S = [s1, s2, L, s T ] · Mapped into query matrix Q, key matrix K and value matrix V through linear transformation:

[0071] Q=SW Q ,K=SW K ,V=SW V

[0072] in, is the learnable linear transformation matrix, dh is the hidden layer dimension of the self-attention mechanism;

[0073] Calculate the attention weight matrix A between different time steps:

[0074]

[0075] wherein A ij represents the attention weight of the i th time window to the j th time window, and the greater the weight, the higher the psychological cognitive interaction intensity of the crew members in the two time windows;

[0076] Apply the attention weight matrix to the value matrix V to obtain the self-attention output matrix H:

[0077]

[0078] MHSA adopts a multi-head parallel computing strategy, and if there are N attention heads, then the attention output matrix of the n th head is:

[0079] H (n) =A (n) V (n) ,n=1,2,L,N

[0080] wherein A (n) and V (n) are the attention weight matrix and the value matrix learned independently by the n th head;

[0081] Concatenate the outputs of all heads and pass them through a linear transformation to obtain the final output matrix U of MHSA:

[0082] U=[H (1) ;H (2) ;L;H (N) ]W O

[0083] wherein, is a linear transformation matrix, and each row vector u t of U represents the psychological cognitive feature of the crew fused with the interaction information of the pilot and the co-pilot;

[0084] Integrating individual psychological cognitive ability feature extraction and crew psychological cognitive interaction feature extraction, a hierarchical psychological cognitive feature sequence {u1,u2,L,u T} is obtained, which takes into account individual psychological cognition and crew interaction, and describes the psychological cognitive state changes of the pilot and the co-pilot in the flight task.

[0085] Preferably, in S3, according to the time evolution characteristics of the individual cognitive ability of the pilot and the cognitive interaction fusion characteristics of the captain and the co-pilot, an LSTM-MHSA model is used to evaluate the overall ability score of the pilot in different psychological cognitive dimensions and the psychological cognitive coordination mode of the captain and the co-pilot during the flight task, and to realize the difference evaluation of the cognitive state of the civil aviation crew, including:

[0086] S3.1: Load the trained LSTM-MHSA model, configure the running environment of the model, and convert the time sequence data of the individual pilot's psychological cognitive ability into the input format required by the model, wherein the 3D tensor is sample number x multiple time steps x feature dimension, for the evaluation of the crew level, the psychological cognitive ability data of the captain and the co-pilot are aligned by time and spliced into an overall input;

[0087] S3.2: Obtain the output of the LSTM-MHSA model, including the individual level psychological cognitive ability feature sequence and the crew level psychological cognitive interaction weight matrix; average pooling is performed on the individual level feature sequence to obtain the overall ability score of the pilot in different psychological cognitive dimensions, and the psychological cognitive interaction weight matrix of the crew level is analyzed to identify the psychological cognitive coordination mode of the captain and the co-pilot during the flight task;

[0088] S3.3: Regularly fine-tune and update the model using newly collected pilot data;

[0089] Wherein, S3.1 specifically includes: first, load the pre-trained LSTM-MHSA model parameters θ * , and configure the software and hardware environment required for model running; then, convert the individual pilot's cognitive ability time sequence data into the tensor format required by the model; specifically, for the evaluation of a single pilot, a three-dimensional tensor is constructed, where N=1 is the sample number, T is the time step, and D is the cognitive feature dimension; for the evaluation of the crew level, the cognitive ability sequences of the captain and the co-pilot are aligned in the time dimension and spliced in the sample dimension to form a crew input tensor with a shape of ;

[0090] S3.2 specifically includes: by analyzing the output of the LSTM-MHSA forward inference process, the cognitive competence evaluation results of the two levels are obtained; in the individual level, the features of each dimension in the LSTM hidden state sequence H (i) are averaged and pooled to obtain the comprehensive score vector of the pilot in different cognitive ability dimensions

[0091]

[0092] wherein, represents the j-th eigenvalue of the LSTM hidden state sequence of the i-th pilot at time step t, and is the hidden state matrix H output by the LSTM model (i) An element in ;d h The dimension of the LSTM hidden state, that is, the number of hidden layer neurons, determines the final score vector s (i) Dimensions; (i) Comprehensively characterize the pilot's cognitive performance throughout the entire mission process, reflecting the level of their attention allocation, information processing, decision-making and judgment abilities; at the same time, cluster analysis is performed on the MHSA output matrix U to identify the cognitive interaction patterns of the pilot and co-pilot at key flight nodes.

[0093] The present invention also provides a civil aviation crew cognitive state difference assessment system based on the LSTM-MHSA algorithm and the hierarchical coupling framework, the system is used to implement any one of the methods, the system includes: a reconstruction module, an extraction module and an assessment module;

[0094] The reconstruction module is used to collect the pilot's EEG and psychological cognitive data before and after the flight mission, and reconstruct the scalarized EEG features and psychological cognitive features into a unified individual cognitive ability feature multivariate time series sample set after preprocessing;

[0095] The extraction module is used to construct a deep learning model LSTM-MHSA that integrates LSTM and multi-head self-attention mechanism based on a unified multivariate time series sample set of individual cognitive ability characteristics, and extract the temporal evolution characteristics of the pilot's individual cognitive ability and the fusion characteristics of the pilot and co-pilot's cognitive interaction;

[0096] The evaluation module is used to evaluate the pilot's overall ability score in different psychological cognitive dimensions and the psychological cognitive coordination mode of the captain and co-pilot during the flight mission based on the temporal evolution characteristics of the pilot's individual cognitive ability and the cognitive interaction fusion characteristics of the captain and co-pilot, using the LSTM-MHSA model, to achieve differential evaluation of the cognitive status of civil aviation crews.

[0097] Compared with the prior art, the present invention has the following beneficial effects:

[0098] (1) The traditional EEG acquisition process faces challenges due to its lack of flexibility and comfort. Data acquisition during flight may not only constitute a potential obstacle to the pilot's normal flight operations, but also continuously wearing EEG acquisition equipment will aggravate the pilot's discomfort and increase the complexity and overall cost of operation. The present invention proposes an innovative experimental acquisition scheme, that is, EEG data acquisition is only performed at two key time points before and after the flight, which improves the convenience and comfort of data acquisition, avoids potential interference with the pilot's normal flight operations, and reduces the complexity and time cost of operation.

[0099] (2) The traditional pilot competence evaluation system mainly relies on subjective judgment and past experience, resulting in evaluation results often accompanied by greater uncertainty. Compared with the traditional evaluation method relying on subjective judgment and experience, the evaluation method based on physiological and psychological data proposed by the present application, which fuses pre-flight and post-flight electroencephalogram data and psychological cognitive ability test data, has significant scientific and reliable advantages. Through objective data analysis, this method can more accurately identify possible problems of pilots in competence, thereby effectively reducing the influence of subjective judgment on evaluation results.

[0100] (3) The traditional pilot missing competence recognition model often uses a single method or algorithm, which is difficult to fully utilize the potential information of data. The present application innovatively proposes a civil aviation crew cognitive state difference evaluation system. The model increases the multi-head self-attention mechanism on the basis of the LSTM framework, so that the model can better capture the relationship between different parts of the input sequence, significantly improving the processing ability, calculation efficiency, expression ability and generalization ability of the model for complex sequence data. This method can focus on the changes in the psychological cognitive state of pilots and timely warn the high-risk period of psychological cognitive ability decline. At the same time, it can locate the key psychological cognitive nodes and interaction modes that affect the psychological cognitive competence of pilots, making the evaluation results more transparent and easier for relevant personnel to take improvement actions. BRIEF DESCRIPTION OF DRAWINGS

[0101] In order to more clearly illustrate the technical solutions of the present application, the following briefly introduces the drawings needed to be used in the embodiments. Obviously, the drawings described in the following embodiments are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0102] Figure 1 The schematic diagram of the main steps S1-S3 described in the embodiments of the present application;

[0103] Figure 2 The schematic diagram of the eight-electrode position of the pilot electroencephalogram in the embodiments of the present application (top-down view);

[0104] Figure 3 The schematic diagram of the LSTM-MHSA algorithm and the hierarchical coupling framework in the embodiments of the present application. DETAILED DESCRIPTION

[0105] The technical solutions in the embodiments of the present application will be described in detail below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0106] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0107] Embodiment one

[0108] As shown in Figure 1 , Figure 3 , the present application provides a civil aviation crew cognitive state difference evaluation method based on LSTM-MHSA algorithm and hierarchical coupling framework, comprising the following steps:

[0109] S1: data acquisition and preprocessing. The pilot's electroencephalogram data before and after the flight task are collected and the psychological cognitive evaluation is carried out, the energy entropy, information entropy, sample entropy and power spectral density are extracted from the electroencephalogram signal, the memory, attention, intelligence, spatial perception, reaction and professional ability are extracted from the psychological cognitive test as the psychological cognitive ability characteristics, and the scalarized electroencephalogram characteristics are reconstructed to form a unified individual cognitive ability characteristic multivariate time sequence sample set. The specific steps are as follows:

[0110] S1.1 is to accurately capture the psychological cognitive state change of the pilot before and after the task execution, and the present application specially designs a simple and efficient electroencephalogram acquisition scheme for the pilot. The experimenters select the key electrode positions closely related to the pilot's psychological cognitive ability, and use conductive paste to ensure stable signal contact, thereby minimizing the interference to the pilot. At the same time, in order to ensure the purity and reliability of the data, the pilot is required to maintain a resting state during the acquisition process, and to exclude the influence of environmental noise and other factors as much as possible. Unlike conventional electroencephalogram experiments, the present application fully considers the actual needs of this special group of pilots, optimizes the number and position of electrodes, and obtains key psychological cognitive information at a relatively small cost. This electroencephalogram data acquisition scheme not only improves the experimental efficiency, but also lays a foundation for subsequent construction of civil aviation crew cognitive state difference evaluation model.

[0111] An eight-lead electroencephalograph is used to perform non-invasive electrical signal acquisition operation on the pilot's cerebral cortex. This process strictly follows the international electroencephalogram (EEG) electrode placement standard, selects Fp1, Fp2, C3, C4, T3, T4, O1, O2 and other key electrode positions, and cooperates with conductive paste to ensure that the Ag / AgCl electrode and the scalp form a stable and low-impedance contact. The reference electrodes are respectively positioned at A1 (left ear lobe) and A2 (right ear lobe) to enhance the accuracy and stability of signal acquisition. The position of the IND lead can be flexibly adjusted according to experimental requirements in actual operation to adapt to individual differences of different pilots.

[0112] According to Figure 2The EEG eight-lead position diagram (top-down view) shown is for the pilot to wear the EEG instrument to ensure that each electrode is accurately placed in the predetermined position. During the collection process, the pilot needs to maintain a resting state to minimize the potential interference of external environmental noise and physiological activity on the EEG signal, thereby ensuring that the collected EEG data has high purity and reliability.

[0113] The EEG data preprocessing process S1.2 includes: using 1-60Hz third-order Butterworth band-pass filtering and 50Hz notch filtering for denoising; the data is centralized, whitened (through covariance matrix and eigenvalue decomposition); ICA technology is used to decompose the signal based on negative entropy to remove eye movement artifacts and channel interference; finally, the wavelet threshold denoising method is used to process the signal after ICA separation.

[0114] The weight vector w is constantly updated according to the iteration formula of FastICA until it converges. Through independent component analysis, several components are decomposed, the correlation coefficient of the decomposed components and the data obtained from the scalp record is calculated, and the maximum value of the components is removed, so that the eye movement artifacts and the noise components generated by the surrounding channel EEG are removed, leaving clean EEG signals. Then, the wavelet threshold denoising method is used to further denoise the EEG signal separated by FastICA. The original signal is wavelet decomposed and wavelet threshold processed, then the wavelet coefficients of different scales are extracted, and finally the signal is reconstructed by wavelet inverse transform. The sym8 wavelet basis is used for three-layer wavelet reduction decomposition, and then the threshold value is processed to suppress noise. According to the selection of wavelet basis and decomposition level, and considering the denoising effect, the signal is segmented for three-layer wavelet decomposition, and then the wavelet threshold method of soft threshold is used to further denoise the EEG signal.

[0115] The S1.3 psychological cognitive module test result data preprocessing process includes processing of abnormal values, i.e. excluding data outside the range of 3 times the standard deviation to ensure data accuracy; missing value processing using mean interpolation method; standardization of the original data, using min-max standardization method to linearly transform the data to the [0, 1] interval, and the conversion formula is based on the maximum and minimum values of the data; and finally normalization processing.

[0116] The processing of abnormal values adopts the method of excluding data outside the range of ±3 times the standard deviation; the processing of missing values adopts the method of mean interpolation; the standardization processing adopts the method of min-max standardization to linearly transform the original data to the interval between 0 and 1.

[0117] The normalization processing converts negative indicators into positive indicators and makes the results fall within the interval [0, 1].

[0118] S1.4 The VMD algorithm is used to adaptively decompose the EEG signal, extract multiple intrinsic mode functions, and calculate a series of nonlinear dynamic features based on the intrinsic mode functions (IMF), including energy entropy, information entropy, sample entropy, and power spectral density.

[0119] First, the VMD algorithm decomposes the original EEG signal f(t) into K IMFs by solving the following variational problem: k

[0120]

[0121] where ωk is the center frequency of the kth IMF, δ(t) is the Dirac function, uk(t) represents the kth intrinsic mode function (IMF), uk(t) represents the instantaneous amplitude of the kth IMF at time t, ωk(t) represents the instantaneous frequency of the kth IMF, which is a function of time t, and j represents the imaginary unit, i.e. k k k k k k In the complex domain, the analytical representation of the signal is used to describe the signal, where uk(t) is the IMF time series to be solved, ωk(t) is the corresponding instantaneous frequency, and j is used in the exponential term to construct the analytical signal. Solving the above problem can obtain a set of IMFs {u1(t), u2(t),..., uk(t)}. K

[0122] Next, the following features are calculated based on each IMF:

[0123] (1) Energy entropy: describes the uniformity of the energy distribution of the IMF. The greater the energy entropy, the more uniform the energy distribution of the IMF.

[0124] (2) Information entropy: evaluates the richness of information in the IMF. The greater the information entropy, the more information the IMF contains.

[0125] (3) Sample entropy: measures the complexity and unpredictability of the IMF. The greater the sample entropy, the stronger the unpredictability of the IMF.

[0126] (4) Power spectral density: describes the energy distribution of the IMF at different frequencies. The power spectral density reflects the oscillation intensity and energy concentration of the IMF in each frequency band.

[0127] ​​​​​​​​​S1.5 Select the scores of the perceptual speed, memory span, N-back task accuracy, attention-stroop effect, spatial cognitive error, calculation ability, and psychomotor ability in the scale test as the core indicators for evaluating the six psychological cognitive dimensions of memory, attention, intelligence, spatial perception ability, reaction speed, and professional ability.

[0128] The pilot will perform a psychological cognitive module test before and after the flight to evaluate the changes and differences in the pilot's pre-flight and post-flight psychological competence. The test tasks in the pilot psychological cognitive module test in the present application consider the different psychological cognitive abilities of pilots from the following dimensions: memory, attention, intelligence, spatial perception, reaction, and professional ability. The test content includes the evaluation of key abilities such as perceptual speed, N-back memory, short-term memory span, attention, psychomotor, spatial cognitive, and calculation ability. The features extracted in the present application can be widely adapted to various pilot psychological scales. The following is a detailed description of the feature extraction in this step:

[0129] (1) Memory feature extraction

[0130] N-back memory test score: Through the N-back task (N takes values of 1, 2, 3, etc., representing the step length of memory backtracking), the pilot's ability in memory maintenance and updating is evaluated. The specific features include the proportion of correct answers, reaction time, and error types (including omission, error replacement, etc.).

[0131] Short-term memory span: Using digital span, word span, etc. test, the amount of information that the pilot can accurately remember and repeat in a short period of time is measured, and the features include maximum memory sequence length, error rate, etc.

[0132] (2) Attention feature extraction

[0133] Attention concentration ability: Through sustained performance test (including visual tracking task), the pilot's ability to maintain attention concentration in long-term tasks is evaluated, and the features include average reaction time, attention dispersion times, error rate, etc.

[0134] Attention allocation ability: Using dual-task or multi-task test, the pilot's efficiency in handling multiple information sources simultaneously is examined, and the features include task switching speed, task completion quality, etc.

[0135] (3) Intelligence feature extraction

[0136] Logical reasoning ability: Through solving logical problems, graphical reasoning, etc. test, the pilot's abstract thinking and problem solving ability are evaluated, and the features include correct answer rate, problem solving speed, etc.

[0137] Knowledge application ability: Problem-solving tasks combined with professional knowledge to assess the ability of pilots to apply theoretical knowledge to practical situations, characterized by the accuracy and efficiency of problem-solving.

[0138] (4) Spatial perception feature extraction

[0139] Spatial orientation ability: Using three-dimensional space rotation, spatial direction judgment tests to evaluate pilots' understanding and operation ability of spatial structure, characterized by the proportion of correct judgment, reaction time, etc.

[0140] Visual-spatial integration ability: Through visual-spatial memory tasks such as map memory and figure reconstruction, the ability of pilots to convert visual information into spatial psychological cognition is measured, characterized by memory accuracy and reconstruction efficiency.

[0141] (5) Reaction force feature extraction

[0142] Simple reaction time: Through simple stimulus-response tasks, the instantaneous reaction speed of pilots to sudden stimuli is measured, characterized by average reaction time and fastest reaction time.

[0143] Choice reaction time: Choose the correct one from multiple possible reactions to assess the rapid decision-making ability of pilots in complex situations, characterized by correct selection reaction time and error rate.

[0144] (6) Professional ability feature extraction

[0145] Flight-related knowledge mastery: Through professional knowledge tests such as flight theory and flight rules, the professional competence of pilots is evaluated, characterized by the comprehensiveness and accuracy of knowledge mastery.

[0146] Scenario simulation performance: Using flight simulators or virtual reality technology to simulate emergency situations in flight, the decision-making and operation skills of pilots under pressure are evaluated, characterized by the effectiveness of emergency handling and time management.

[0147] The scores of pilot age, perceptual speed reaction time, short-term memory span, N-back accuracy, attention-stroop effect (interference exclusion ability), spatial psychological cognition (distance and speed perception error), calculation ability, and psychological motor ability are selected as feature indicators to analyze the changes in the psychological state of pilots. The proposed method has universality and compatibility, and can be widely applied to various scales aimed at comprehensive evaluation of pilots' psychological ability from six core dimensions: memory, attention, intelligence level, spatial perception acuity, reaction speed, and professional skills.

[0148] The input of the S1.6 model is composed of the brain electrical and psychological cognitive features obtained at multiple pre-flight and post-flight sampling time points, forming a unified individual cognitive ability feature multi-time sequence sample.

[0149] Let the cognitive ability observation value of the pilot i at the t-th sampling time point be where D = 10 is the cognitive ability index dimension fused with brain electrical and psychological test. Among them represents the brain electrical features of pilot i at the t-th sampling time point, including energy entropy, information entropy, sample entropy and power spectral density four key features, which are converted into scalar values in the [0, 1] interval by standardizing the four features. Specifically, for the energy entropy feature First, calculate the minimum value E min and the maximum value E max of the energy entropy in the entire data set, and then apply the maximum and minimum value normalization formula to convert it to a scalar value in the [0, 1] interval Similarly, for the information entropy feature the sample entropy feature and the total power feature maximum and minimum value normalization is performed to obtain the scalar value The scalarized brain electrical feature vector can be expressed as

[0150] represents the psychological cognitive test results of pilot i at the t-th sampling time point, including memory, attention, intelligence, spatial perception, reaction force and professional ability six dimensions of test scores, and normalized to a scalar value in the [0, 1] interval.

[0151] The scalarized brain electrical feature is spliced with the normalized psychological cognitive test score , where contains 6 dimensions of psychological cognitive test scores (from to ), and all are normalized to the 0-1 interval, so its expression is The complete cognitive ability observation value of pilot i at the t-th sampling time point is obtained:

[0152] The complete cognitive ability sequence of pilot i is taken as the input of the LSTM-MHSA model, where T is the sampling time step.

[0153] S2: Train the algorithm model to realize feature extraction. A deep learning model LSTM-MHSA is constructed by combining LSTM and multi-head self-attention mechanism, LSTM is used to extract the time evolution features of individual pilot psychological cognitive ability, MHSA is used to capture the psychological cognitive interaction feature matrix of the crew, and the deviation degree of the model output from the real psychological cognitive state is measured. The specific steps are:

[0154] S2.1 Construct the LSTM-MHSA model structure, design a deep learning model that combines LSTM (Long Short-Term Memory Network) and MHSA (Multi-Head Self-Attention Mechanism). The model uses multi-layer bidirectional LSTM as the backbone, which can effectively extract long and short-term dependence features in time series data. On the basis of LSTM, MHSA is introduced to model the internal relationship of the sequence, and the model's ability to capture key features is enhanced.

[0155] The core of the LSTM-MHSA model is to combine LSTM and MHSA, which aims to fully exploit the time series dependence and key features in pilot psychological cognitive ability data. The overall structure of the model can be represented as:

[0156]

[0157] where Embedding layer maintains a weight matrix, each row of which corresponds to a vector representation of a discrete feature. During training, these vectors will be updated and optimized along with the model parameters, ultimately learning the semantic information of the input features. X = {x1, x2, L, x T} is the original time series data of pilot psychological cognitive ability, is the input embedding representation, L is the number of LSTM layers, is the hidden state sequence of the l-th LSTM layer, is the output sequence of MHSA, is the predicted probability distribution of pilot psychological cognitive state.

[0158] In the LSTM part, multi-layer bidirectional LSTM is used to model the time evolution law of pilot psychological cognitive ability. The calculation process of the l-th BiLSTM is as follows:

[0159]

[0160] where, and represent the forward and backward LSTM units respectively, is the hidden state of them at time step t, is the complete hidden state of bidirectional LSTM at time t. Bidirectional LSTM can utilize information from multiple tests by processing sequences in both forward and reverse directions.

[0161] In the MHSA part, the multi-head self-attention mechanism is used to model the dependencies between different time steps of the pilot, capturing the internal association patterns of the psychological cognitive ability sequence. Specifically, for the output sequence of the l-th layer LSTM The calculation process of MHSA is as follows:

[0162]

[0163] where, and are learnable linear transformation matrices, is the output of the n-th attention head, N is the number of attention heads, d k = 2d h / N is the hidden layer dimension of each head. The Attention(·) function calculates the normalized dot product attention:

[0164]

[0165] The multi-head self-attention mechanism can learn the mutual relationship between sequence elements in different subspaces by parallel computing multiple attention functions, enhancing the feature representation ability of the model. Finally, the outputs of all heads are concatenated and linearly transformed to obtain a high-level feature representation U that integrates different scale dependency relationships.

[0166] In summary, the LSTM-MHSA model combines the ability of LSTM to capture long-term and short-term temporal dependencies and the advantage of MHSA to model complex internal associations of sequences, forming an end-to-end pilot psychological cognitive ability evaluation framework. Through multi-layer bidirectional LSTM, temporal features are extracted, and multi-head self-attention mechanism is used to mine key patterns within the sequence.

[0167] S2.2 The complete psychological cognitive ability sequence of pilot i As the input of the model, the LSTM-MHSA model is used to hierarchically encode the psychological cognitive ability time series data of a single pilot, obtaining cognitive feature representations at different scales and different abstraction levels.

[0168] First, the psychological cognitive ability sequence is encoded using a bidirectional LSTM network. The forward LSTM updates the hidden state step by step

[0169]

[0170] where are the parameters of the forward LSTM. The backward LSTM updates the hidden state in reverse order

[0171]

[0172] where are the parameters of the reverse LSTM. The concatenated hidden states of the forward and reverse LSTMs can obtain the bidirectional LSTM encoding at the t-th step

[0173] Next, the encoding sequence of the bidirectional LSTM is fed into a stacked multi-layer LSTM to realize the hierarchical abstraction of the psychological cognitive features. The hidden state update formula of the l-th layer LSTM is

[0174]

[0175] where denotes the output of the bidirectional LSTM, and (l) are the parameters of the l-th layer LSTM. After stacking L layers of LSTMs, the multi-scale psychological cognitive features of the pilot i at the t-th time step can be obtained

[0176] Finally, the outputs of the multi-layer LSTMs are adaptively aggregated by a gating mechanism to obtain the final psychological cognitive ability features of the pilot i at the t-th time step

[0177]

[0178] where W g and b g are the parameters of the gating layer, and σ(·) is the sigmoid activation function. The gating layer can adaptively adjust the importance of the psychological cognitive features at different abstraction levels at different time steps, capturing the dynamic changes of the pilot's psychological cognitive state.

[0179] In summary, through bidirectional LSTM, multi-layer LSTM, and gating aggregation, multi-scale time sequence features that characterize the evolution pattern of the individual psychological cognitive ability of the pilot can be extracted to provide rich feature representations for subsequent hierarchical coupling analysis.

[0180] S2.3 Crew Psychological Cognitive Interaction Feature Extraction. The crew psychological cognitive interaction feature extraction aims to model the psychological cognitive interaction relationship between the captain and the co-pilot during the flight using the MHSA mechanism, and to mine the interaction patterns of both parties at different psychological cognitive dimensions and different time scales. Let the multi-dimensional psychological cognitive ability features of the captain and the co-pilot in the t-th time window be and then the psychological cognitive ability of the crew at the t-th time step can be represented as Where d is the dimension of individual psychological cognitive characteristics. Dividing a complete flight mission into T time windows, we can get the psychological cognitive ability sequence of the crew {s1, s2, L, s T}.

[0181] MHSA firstly transforms the crew's psychological cognitive ability sequence S = [s1, s2, L, s T ] · Mapped into query matrix Q, key matrix K and value matrix V through linear transformation:

[0182] Q=SW Q ,K=SW K ,V=SW V (10)

[0183] in is the learnable linear transformation matrix, d h is the hidden layer dimension of the self-attention mechanism. Next, calculate the attention weight matrix A between different time steps:

[0184]

[0185] Among them A ij Represents the attention weight of the i-th time window to the j-th time window. The larger the weight, the higher the intensity of psychological cognitive interaction between the crew members in the two time windows. Applying the attention weight matrix to the value matrix V, we get the self-attention output matrix H:

[0186]

[0187] To depict the psychological cognitive interaction patterns of the pilot and co-pilot from different perspectives, MHSA adopts a multi-head parallel computing strategy. Suppose there are N attention heads, then the attention output matrix of the nth head is:

[0188] H (n) =A (n) V (n) ,n=1,2,L,N (13)

[0189] Among them A (n) and V (n) are the attention weight matrix and value matrix learned independently by the nth head respectively. Finally, the outputs of all heads are concatenated and linearly transformed to obtain the final output matrix U of MHSA:

[0190] U=[H (1) ;H (2) ;L;H (N) ]W O (14)

[0191] in is a linear transformation matrix. Each row vector u t represents the psychological cognitive characteristics of the crew after fusing the interaction information of the captain and the copilot.

[0192] The two steps of individual psychological cognitive ability feature extraction and crew psychological cognitive interaction feature extraction can obtain the hierarchical psychological cognitive feature sequence {u1, u2, L, u T} that takes into account individual psychological cognition and crew interaction, used to describe the psychological cognitive state changes of the captain and the copilot in the flight task.

[0193] S2.4 Model training and optimization. In the model training and optimization phase, the LSTM-MHSA model is trained in an end-to-end manner, with the goal of minimizing the difference between the psychological cognitive features extracted by the model and the real psychological cognitive state of the pilot. First, the collected pilot original multi-element psychological cognitive ability dataset D is divided into training set D train , validation set D val and test set D test , with proportions of 70%, 10% and 20% respectively.

[0194] For each sample (X (i) , y (i) ) ∈ D train , where X is the original psychological cognitive ability sequence of the i-th pilot, y (i) ∈ {0, 1} C is the corresponding psychological cognitive state label (such as normal, fatigue, inattention, etc.), the LSTM-MHSA model first extracts the individual psychological cognitive feature sequence and then fuses the crew interaction features to obtain the final hierarchical coupled psychological cognitive feature sequence

[0195] After each training epoch, the performance of the model is evaluated on the validation set D val , and the learning rate is adjusted and it is determined whether to end the training according to the validation loss and early stopping strategy. The method of guiding the training process through the validation set can effectively prevent the model from overfitting the training data.

[0196] Finally, the trained LSTM-MHSA model is used to evaluate the performance on the test set D test .

[0197] S3: Construct the cognitive state difference evaluation system for civil aviation crew. Based on the LSTM-MHSA model, the individual-level feature sequence is averaged and pooled to obtain the overall ability score of the pilot in different psychological cognitive dimensions, and the psychological cognitive interaction weight matrix of the crew level is analyzed to identify the psychological cognitive coordination mode of the captain and the co-pilot during the flight task.

[0198] The specific steps are:

[0199] S3.1 Model invocation and data input, load the trained LSTM-MHSA model, and configure the running environment of the model. The individual pilot's psychological cognitive ability time series data (obtained in S1) is converted into the input format required by the model, where the 3D tensor is the sample number x multiple time steps x feature dimension. For crew-level evaluation, the psychological cognitive ability data of the captain and the co-pilot are aligned by time and spliced into a whole input. Specifically as follows:

[0200] First, load the pre-trained LSTM-MHSA model parameters θ * , and configure the software and hardware environment required for model running. Next, the individual pilot's cognitive ability time series data is converted into the tensor format required by the model. Specifically, for the evaluation of a single pilot, a three-dimensional tensor is constructed, where N=1 is the sample number, T is the time step, and D is the cognitive feature dimension. For crew-level evaluation, the cognitive ability sequences of the captain and the co-pilot are aligned in the time dimension and spliced in the sample dimension to form a crew input tensor with a shape of .

[0201] S3.2 Model output analysis, obtain the output of the LSTM-MHSA model, including the individual-level psychological cognitive ability feature sequence and the crew-level psychological cognitive interaction weight matrix; average pool the individual-level feature sequence to obtain the overall ability score of the pilot in different psychological cognitive dimensions. Analyze the crew-level psychological cognitive interaction weight matrix to identify the psychological cognitive coordination mode of the captain and the co-pilot during the flight task. Specifically as follows:

[0202] By analyzing the output of the LSTM-MHSA forward inference process, we can obtain the cognitive competence evaluation results at two levels. At the individual level, the features in each dimension of the LSTM hidden state sequence H (i) are averaged and pooled to obtain the comprehensive score vector of the pilot in different cognitive ability dimensions

[0203]

[0204] denotes the j-th dimensional feature value of the LSTM hidden state sequence of the i-th pilot at time step t, is an element of the hidden state matrix H output by the LSTM model (i) h denotes the dimension of the LSTM hidden state, i.e., the number of hidden layer neurons, which determines the dimension of the final score vector s (i) (i) comprehensively characterizes the cognitive performance of the pilot throughout the entire mission process, reflecting the level of his attention allocation, information processing, decision-making ability, etc. At the same time, the MHSA output matrix U is subjected to cluster analysis to identify the cognitive interaction patterns of the co-pilot at key flight nodes, such as information sharing, task coordination, abnormal handling, etc. This provides an important reference for evaluating the performance of the flight crew.

[0205] S3.3 Hyperparameter Adjustment and Model Optimization. To ensure that the LSTM-MHSA model can adapt to the changing flight environment and crew combination, the model is regularly fine-tuned and updated using newly collected pilot data.

[0206] The optimal model structure configuration is sought in a reasonable hyperparameter search space, such as the LSTM hidden layer dimension d h ∈{32,64,128}, the number of MHSA heads n head ∈{1,2,4}, the Dropout rate p drop ∈[0.1,0.5]. The model is regularly fine-tuned using new pilot data to continuously improve its generalization performance.

[0207] In the hierarchical coupling civil aviation crew cognitive state difference evaluation method, by comparing and analyzing the psychological cognitive ability of a single pilot before and after performing a flight task, the dynamic change law of his psychological cognitive state can be accurately described. The LSTM-MHSA model uses a deep learning algorithm to adaptively extract key psychological cognitive features of the pilot before and after the flight. In the pre-flight stage, the model focuses on the baseline psychological cognitive level of the pilot in the resting state, including the comprehensive performance of attention, memory, reasoning and judgment, etc., to establish a reference benchmark for individual psychological cognitive ability. In the post-flight stage, the model focuses on the changes in the pilot's psychological cognitive state after experiencing a complete flight task, and describes the dynamic characteristics of his attention allocation, information processing, decision-making response, and other psychological cognitive functions under continuous work load. By aligning the psychological cognitive feature representations of the pre-flight and post-flight stages in the end-to-end semantic space, this method can capture the fluctuations in the pilot's psychological cognitive ability during the task execution process and timely detect signs of psychological cognitive state degradation that may affect flight safety. Based on the quantitative indicators obtained from the comparative analysis, the system can intelligently generate a personalized psychological cognitive ability report for the pilot and provide targeted improvement suggestions, such as attention training, stress management, etc., thereby continuously improving the pilot's flight proficiency.​​

[0208] One of the core innovations of the hierarchical coupling evaluation paradigm is the modeling of the psychological cognitive interaction between the pilot and the co-pilot, forming an integrated group psychological cognitive analysis perspective. The LSTM-MHSA model uses a multi-head attention mechanism to explore the psychological cognitive interaction patterns of the pilot and the co-pilot in different scales throughout the flight task. For each flight phase, the model estimates the differences in attention allocation of the pilot and the co-pilot at key psychological cognitive event nodes (including complex weather, emergency disposal, etc.), as well as the information transmission characteristics in the decision chain, to depict the fusion and complementarity of the psychological cognitive interaction between the roles. By synthesizing multiple flight phases throughout the process, the model extracts the key interaction patterns of the pilot and the co-pilot in terms of psychological cognitive load sharing, abnormal diagnosis, and collaborative decision-making, and quantitatively evaluates the emergence level of the overall intelligence of the crew. Through the modeling of the psychological cognitive interaction between the pilot and the co-pilot, the limitations of traditional evaluation methods are overcome, and a new idea of group psychological cognitive quantitative evaluation centered on the "crew" is formed. This provides an important handle for in-depth understanding of the human factors of crew resource management. Based on the comparison and analysis of the pilot and the co-pilot, the system can diagnose possible psychological cognitive complementary blind spots within the crew, optimize task division, and provide quantitative reference for abnormal judgment and collaborative decision-making in crisis situations, thereby improving the accuracy of pilot proficiency judgment.

[0209] Embodiment Two

[0210] The application also provides a civil aviation crew cognitive state difference evaluation system based on the LSTM-MHSA algorithm and the hierarchical coupling framework, which is used to implement any of the methods, and the system comprises a reconstruction module, an extraction module, and an evaluation module.

[0211] The reconstruction module is used to collect the brain electrical activity and psychological cognitive data of the pilot before and after the flight task execution, and to reconstruct the scalarized brain electrical activity features and psychological cognitive features into a unified individual cognitive ability feature multivariate time series sample set after preprocessing.

[0212] The extraction module is used to construct a deep learning model LSTM-MHSA that integrates LSTM and multi-head self-attention mechanism according to the unified individual cognitive ability feature multivariate time series sample set, to extract the pilot's personal cognitive ability time evolution features and the cognitive interaction fusion features of the pilot and the co-pilot.

[0213] The evaluation module is used to use the LSTM-MHSA model to evaluate the overall ability scores of the pilot in different psychological cognitive dimensions and the psychological cognitive collaboration patterns of the pilot and the co-pilot during the flight task according to the pilot's personal cognitive ability time evolution features and the cognitive interaction fusion features of the pilot and the co-pilot, to realize the difference evaluation of the civil aviation crew cognitive state.

[0214] The above described embodiments are only to illustrate the preferred modes of the present application, and are not intended to limit the scope of the present application. Any modification and improvement made by those skilled in the art to the technical solutions of the present application without departing from the design spirit of the present application shall fall within the protection scope of the present application as defined by the claims.

Claims

1. A method for evaluating the difference in cognitive status of civil aviation crew based on the LSTM-MHSA algorithm and hierarchical coupling framework, characterized by: The method comprises: S1: The pilot's EEG and psychological cognitive data are collected before and after the flight mission. After preprocessing, the scalarized EEG features and psychological cognitive features are reconstructed to form a unified multivariate time series sample set of individual cognitive ability features; S2: Based on a unified multivariate time series sample set of individual cognitive ability characteristics, a deep learning model LSTM-MHSA is constructed that integrates LSTM and multi-head self-attention mechanism to extract the temporal evolution characteristics of pilots' individual cognitive abilities and the fusion characteristics of the pilot and co-pilot's cognitive interaction; S3: Based on the temporal evolution characteristics of pilots' individual cognitive abilities and the fusion characteristics of the captain and co-pilot's cognitive interaction, the LSTM-MHSA model is used to evaluate the pilots' overall ability scores in different psychological cognitive dimensions and the psychological cognitive coordination patterns of the captain and co-pilot during the flight mission, thereby achieving differential assessment of the cognitive status of civil aviation crews. In S1, the pilot's EEG and psychological cognitive data are collected before and after the flight mission, and after preprocessing, the scalarized EEG features and psychological cognitive features are reconstructed to form a unified individual cognitive ability feature multivariate time series sample set including: S1.1: Before and after the flight mission, an eight-lead electroencephalogram (EEG) device was used to collect the pilot's cerebral cortical electrical signals according to international EEG standards. Key electrode positions were selected and conductive paste was used to ensure stable contact. The reference electrode was positioned on the earlobe, and the IND lead was adjustable. S1.2: For EEG data, 1-60 Hz third-order Butterworth bandpass filtering and 50 Hz notch filtering were used for denoising. The data were centered and whitened, and the signals were decomposed using ICA technology based on negative entropy to remove eye artifacts and channel interference. Finally, the wavelet threshold denoising method was used to process the ICA-separated signals. S1.3: For the psychological cognitive module test results, data outside the range of 3 times the standard deviation were eliminated. Missing values ​​were handled using the mean interpolation method. The original data were standardized using the min-max standardization method to linearly transform the data to the interval [0, 1]. The transformation formula was based on the maximum and minimum values ​​of the data, and finally normalized. S1.4: Apply variational mode decomposition to extract energy entropy, information entropy, sample entropy, and power spectral density as EEG features. Calculate the energy entropy of different intrinsic mode functions. Use information entropy to assess the value of information in EEG signals. Use sample entropy to measure the complexity of time series. Estimate the power spectrum of signals using the periodogram method. S1.5: Select the scores of perceptual speed, memory span, N-back task accuracy, attention-stroop effect, spatial cognitive error, computational ability, and psychomotor ability from the scale test as core indicators for assessing the six cognitive dimensions of memory, attention, intelligence, spatial perception, reaction speed, and professional ability; S1.6: The EEG and psychological cognitive features obtained from multiple sampling moments before and after the flight are scalarized to form a unified multivariate time series sample of individual cognitive ability characteristics; In S2, based on a unified multivariate time series sample set of individual cognitive ability features, a deep learning model LSTM-MHSA that integrates LSTM and multi-head self-attention mechanism is constructed to extract the temporal evolution features of the pilot's individual cognitive ability and the fusion features of the pilot and co-pilot's cognitive interaction, including: S2.1: Design a deep learning model LSTM-MHSA that integrates LSTM and MHSA. The model uses a multi-layer bidirectional LSTM as the backbone and introduces MHSA to model the internal relationships of sequences. S2.2: Complete psychological cognitive ability sequence of pilot i As the input of the model, the LSTM-MHSA model is used to hierarchically encode the time series data of the psychological cognitive ability of a single pilot to obtain cognitive feature representations at different scales and levels of abstraction; S2.3: Use the MHSA mechanism to model the psychological cognitive interaction between the pilot and co-pilot within the crew; S2.4: Train the LSTM-MHSA model in an end-to-end manner to achieve adaptive extraction of individual-crew coupling features from raw multivariate psychological cognitive ability data. Use the cross-entropy loss function to measure the degree of deviation between the features extracted by the model and the pilot's actual psychological cognitive state. Use the Adam adaptive optimization algorithm to iteratively update the model parameters and minimize the loss function. Use methods such as Dropout and L2 regularization to prevent model overfitting.

2. The method according to claim 1, characterized in that In S1.4, the application of variational mode decomposition to extract energy entropy, information entropy, sample entropy and power spectrum density as EEG features includes: The VMD algorithm is used to solve the variational problem and decompose the original EEG signal f(t) into K Among them, ω k is the center frequency of the kth IMF, δ(t) is the Dirac function, u k represents the kth intrinsic mode function (IMF), u k (t) represents the instantaneous amplitude of the kth IMF at time t, which is a function of time t, ω k (t) represents the instantaneous frequency of the kth IMF, which is a function of time t. It describes the local frequency characteristics of the IMF at time t. j represents the imaginary unit, that is, The analytical representation used to describe the signal in the complex domain, where u k (t) is the IMF time series to be solved, ω k (t) is the corresponding instantaneous frequency, j in the exponential term Used to construct analytical signals; Energy entropy, information entropy, sample entropy and power spectral density are calculated based on each IMF.

3. The method according to claim 1, characterized in that In S1.6, the EEG and psychological cognitive features obtained from multiple sampling moments before and after the flight are scalarized to form a unified multivariate time series sample of individual cognitive ability features, including: Assume that the observed cognitive ability of pilot i at the tth sampling time is Among them, D=10 is the cognitive ability index dimension that integrates EEG and psychological tests. represents the EEG characteristics of pilot i at the t-th sampling moment, including four key features: energy entropy, information entropy, sample entropy, and power spectrum density. The four key features are converted into scalar values ​​in the interval [0, 1] by normalization; represents the psychological cognitive test results of pilot i at the t-th sampling moment, including test scores in six dimensions: memory, attention, intelligence, spatial perception, reaction, and professional ability, and normalized to a scalar value in the interval {0,1}; The scalared EEG features and normalized cognitive test scores} Splicing, we get the complete cognitive ability observation value of pilot}i}} at the tth sampling moment: The complete cognitive ability sequence of pilot i As the input of the LSTM-MHSA model, where}T} is the number of sampling time steps.

4. The method according to claim 1, wherein In S2.1, a deep learning model LSTM-MHSA is designed that integrates LSTM and MHSA. The model uses a multi-layer bidirectional LSTM as the backbone and introduces MHSA to model the internal relationships of the sequence, including: The overall structure of the LSTM-MHSA model is expressed as: H (0) =Embedding(X) H (l) =BiLSTM (l) (H (l-1) ),l=1,2,L,L U=MHSA(H (L) ) o=Softmax(FC(U)) Among them, the Embedding layer maintains a weight matrix, in which each row corresponds to the vector representation of a discrete feature. During the training process, the vector will be continuously updated and optimized along with the model parameters, and finally the semantic information of the input feature will be learned. T } is the pilot's original psychological cognitive ability time series data,} is the input embedding representation, L is the number of LSTM layers, is the hidden state sequence of the l-th layer LSTM, is the output sequence of MHSA, is the predicted probability distribution of the pilot’s psychological cognitive state; With multi-layer bidirectional LSTM as the backbone, it includes: in, and} Represent the forward and backward LSTM units respectively, for and} The hidden state at time step t is, is the complete hidden state of the bidirectional LSTM at time t. The bidirectional LSTM can comprehensively utilize the information of multiple tests by processing the sequence in both the forward and reverse directions; The introduction of MHSA to model the relationships within the sequence includes: U=[head1;head2;L;head N ]W O in, and is a learnable linear transformation matrix, is the output of the nth attention head, N is the number of attention heads,}d k =2d h / N is the hidden layer dimension of each head, and the Attention(·) function calculates the normalized dot product attention: The multi-head self-attention mechanism learns the relationship between sequence elements in different subspaces by computing multiple attention functions in parallel, thus enhancing the feature representation ability of the model. Finally, the outputs of all heads are concatenated and linearly transformed to obtain a high-level feature representation that integrates dependencies at different scales. U .

5. The method according to claim 4, characterized in that In S2.2, the complete psychological cognitive ability sequence of pilot i is As the input of the model, the LSTM-MHSA model is used to hierarchically encode the time series data of the psychological cognitive ability of a single pilot to obtain cognitive feature representations at different scales and levels of abstraction, including: A bidirectional LSTM network is used to encode the sequence of psychological cognitive abilities, and the forward LSTM gradually updates the hidden state. in, are the parameters of the forward LSTM; Reverse LSTM updates hidden states in reverse order in, are the parameters of reverse LSTM; Concatenate the forward and reverse hidden states to obtain the bidirectional LSTM encoding of the tth step The encoding sequence of the bidirectional LSTM The data is fed into a stack of multi-layer LSTMs to achieve hierarchical abstraction of psychological cognitive features. The hidden state update formula of the lth layer LSTM is: in, represents the output of bidirectional LSTM,}θ (l) is the parameter of the lth layer LSTM; After stacking L layers of LSTM, the multi-scale psychological cognitive features of pilot i at time t are obtained The final psychological cognitive ability characteristics of pilot i at time t are obtained by adaptively aggregating the output of multi-layer LSTM through the gating mechanism. Among them, W g and}b g }} is the parameter of the gating layer, σ(·) is the sigmoid activation function,} e represents the Hadamard product; Through bidirectional LSTM, multi-layer LSTM and gated aggregation, multi-scale temporal features that depict the evolutionary pattern of individual pilots' psychological cognitive abilities are extracted.

6. The method according to claim 5, characterized in that In S2.3, the MHSA mechanism is used to model the psychological cognitive interaction between the pilot and co-pilot within the crew, including: Assume that the multivariate psychological cognitive ability characteristics of the pilot and co-pilot in the tth time window are respectively} and z t (c) , then the crew's psychological cognitive ability at time t can be expressed as Where d is the dimension of individual psychological cognitive characteristics, and a complete flight mission is divided into T time windows, that is, the psychological cognitive ability sequence of the crew {s1, s2, L, s T }; MHSA will crew psychological cognitive ability sequence}S=[s1,s2,L,s T ] · } is mapped into query matrix Q, key matrix}K} and value matrix V through linear transformation: Q=SW Q ,K=SW K ,V=SW V in, is the learnable linear transformation matrix, d h }} is the hidden layer dimension of the self-attention mechanism; Calculate the attention weight matrix A between different time steps: Among them,}A ij }} represents the attention weight of the i-th time window to the j-th time window. The larger the weight, the higher the intensity of psychological cognitive interaction between the crew members in the two time windows. Applying the attention weight matrix to the value matrix V yields the self-attention output matrix H: MHSA adopts a multi-head parallel computing strategy. There are N attention heads, and the attention output matrix of the nth head is: H (n) =A (n) V (n) ,n=1,2,L,N Among them,}A (n) } and V (n) are the attention weight matrix and value matrix learned independently by the n-th head respectively; The outputs of all heads are concatenated and linearly transformed to obtain the final output matrix U of MHSA: U=[H (1) ;H (2) ;L;H (N) ]W O in, is the linear transformation matrix, each row vector of U}u t }} represents the crew's psychological cognitive characteristics that integrate the pilot and co-pilot interaction information; The hierarchical psychological cognitive feature sequence {u1,u2,L,u T }, describing the changes in the psychological cognitive state of the pilot and co-pilot during the flight mission.

7. The method according to claim 1, characterized in that In S3, based on the temporal evolution characteristics of the pilot's individual cognitive ability and the cognitive interaction and fusion characteristics of the captain and co-pilot, the LSTM-MHSA model is used to evaluate the pilot's overall ability scores in different psychological cognitive dimensions and the psychological cognitive coordination mode of the captain and co-pilot during the flight mission, thereby achieving differential assessment of the cognitive status of civil aviation crews, including: S3.1: Load the trained LSTM-MHSA model, configure the model's runtime environment, and convert individual pilots' cognitive performance time series data into the model's required input format, where a 3D tensor is calculated as the number of samples × multiple time steps × feature dimensions. For crew-level assessment, align the pilot and co-pilot cognitive performance data by time and concatenate them into a single input. S3.2: Obtain the output of the LSTM-MHSA model, including the individual-level psychological cognitive ability feature sequence and the crew-level psychological cognitive interaction weight matrix. Perform average pooling on the individual-level feature sequence to obtain the pilot's overall ability score across different psychological cognitive dimensions. Analyze the crew-level psychological cognitive interaction weight matrix to identify the psychological cognitive coordination pattern between the pilot and co-pilot during the flight mission. S3.3: Regularly fine-tune and update the model using newly collected pilot data; Among them, the S3.1 specifically includes: first, loading the pre-trained LSTM-MHSA model parameters θ * , and configure the software and hardware environment required for the model to run; then, the cognitive ability time series data of individual pilots} Convert to the tensor format required by the model; specifically, for the evaluation of a single pilot, construct a three-dimensional tensor Where N=1 is the number of samples, T is the time step, and D is the cognitive feature dimension. For the crew-level evaluation, the cognitive ability sequences of the pilot and co-pilot are aligned in the time dimension and spliced ​​in the sample dimension to form a shape of The unit input tensor of ; Said S3.2 specifically includes: by parsing the output of the LSTM-MHSA forward reasoning process, obtaining cognitive competence assessment results at two levels; at the individual level, (i) The features of each dimension are averaged and pooled to obtain the comprehensive score vector of the pilot in different cognitive ability dimensions. in, represents the j-th eigenvalue of the LSTM hidden state sequence of the i-th pilot at time step t, and is the hidden state matrix H output by the LSTM model (i) An element in ;d h The dimension of the LSTM hidden state, that is, the number of hidden layer neurons, determines the final score vector s (i) Dimensions; (i) Comprehensively characterize the pilot's cognitive performance throughout the entire mission process, reflecting the level of their attention allocation, information processing, decision-making and judgment abilities; at the same time, cluster analysis is performed on the MHSA output matrix U to identify the cognitive interaction patterns of the pilot and co-pilot at key flight nodes.

8. A civil aviation crew cognitive state difference assessment system based on the LSTM-MHSA algorithm and hierarchical coupling framework, the system being used to implement the method according to any one of claims 1 to 7, characterized in that: The system includes: a reconstruction module, an extraction module and an evaluation module; The reconstruction module is used to collect the pilot's EEG and psychological cognitive data before and after the flight mission, and reconstruct the scalarized EEG features and psychological cognitive features into a unified individual cognitive ability feature multivariate time series sample set after preprocessing; The extraction module is used to construct a deep learning model LSTM-MHSA that integrates LSTM and multi-head self-attention mechanism based on a unified multivariate time series sample set of individual cognitive ability characteristics, and extract the temporal evolution characteristics of the pilot's individual cognitive ability and the fusion characteristics of the pilot and co-pilot's cognitive interaction; The evaluation module is used to evaluate the pilot's overall ability score in different psychological cognitive dimensions and the psychological cognitive coordination mode of the captain and co-pilot during the flight mission based on the temporal evolution characteristics of the pilot's individual cognitive ability and the cognitive interaction fusion characteristics of the captain and co-pilot, using the LSTM-MHSA model, to achieve differential evaluation of the cognitive status of civil aviation crews.

Images