Civil aviation unit cognitive state difference evaluation method and system based on LSTM-MHSA algorithm and hierarchical coupling framework

Abstract

The invention belongs to the field of flight safety, and discloses a civil aviation unit cognitive state difference evaluation method and system based on an LSTM-MHSA algorithm and a hierarchical coupling framework, and the method comprises the steps: S1, respectively collecting the electroencephalogram and psychological cognitive data of a pilot before and after the execution of a flight task, and after preprocessing, the scaled electroencephalogram features and psychological cognitive features are reconstructed to form a unified individual cognitive ability feature multivariate time sequence sample set. S2, constructing a deep learning model LSTM-MHSA, and extracting pilot personal cognitive ability time sequence evolution characteristics and pilot and copilot cognitive interaction fusion characteristics; s3, using an LSTM-MHSA model to evaluate the overall ability scores of the pilot in different psychological cognition dimensions and the psychological cognition cooperation mode of the pilot and the copilot in the flight mission period, and achieving the difference evaluation of the cognitive state of the civil aviation unit. The method aims at comprehensively evaluating the psychological cognitive ability of the pilot from the aspects of individuals and units.

Description

Technical Field

[0001] The present invention belongs to the field of flight safety, and particularly relates to a method and system for evaluating the cognitive state differences of civil aviation crews based on the LSTM-MHSA algorithm and a hierarchical coupling framework. Background Art

[0002] In the aviation field, the operation accuracy and psychological state of pilots are directly related to flight safety. In recent years, with the development of neuroscience and related technologies, electroencephalogram (EEG) data has shown great potential in the evaluation of pilots' psychological states and psychological cognitive abilities. EEG data can reflect the activity state of the brain in real time and objectively. Through specific EEG indicators, key indicators such as pilots' fatigue level, attention concentration, emotional state, and psychological cognitive load can be evaluated. Due to the high sensitivity and objectivity of EEG data, it is regarded as the "gold standard" for evaluating pilots' psychological states and psychological cognitive abilities. However, there are several limitations in relying solely on EEG data for pilots' state monitoring. Although EEG data can capture information on brain activity in real time, it is extremely vulnerable to environmental noise interference and is difficult to directly and accurately map the specific performance of pilots when performing specific flight tasks, especially considering the inconvenience and safety hazards that may be brought about by wearing EEG devices during flight. At the same time, psychological scale data, as a key source of information, can comprehensively quantify pilots' psychological states and psychological cognitive abilities. These scale data are usually collected in a non-flight environment, providing a scientific and objective benchmark for the mental health assessment of pilots. Specifically, the main deficiencies of EEG data lie in its susceptibility to interference, weak direct relevance, and high professional threshold for interpretation; while psychological scale data, although it can provide valuable psychological assessments in a non-flight state, lacks real-time performance and direct reflection of specific flight situations.

[0003] The integrated analysis of the two aims to solve the following problems: First, by complementing multi-source data, improve the comprehensiveness and accuracy of pilot state assessment; second, utilize the stability of psychological scale data to assist in interpreting the dynamic changes of EEG data and reduce the dependence on professional interpretation; third, achieve the correlation analysis of the psychological state before and after flight and real-time brain activities, and provide more scientific support for the comprehensive assessment and training of pilots. Therefore, the present invention proposes an innovative EEG measurement scheme, that is, performing EEG measurement on pilots before and after flight. This method not only minimizes the interference to pilots during flight tasks and improves the comfort of measurement, but also can capture the psychological state changes of pilots before and after flight, and provide more comprehensive and accurate data support for flight safety assessment. By integrating EEG data and psychological scale data, this method can make full use of the advantages of various data and achieve the assessment and recognition of pilots' psychological and psychological cognitive ability states. However, there is currently a lack of an effective technical means to implement this method, especially there are many challenges in feature extraction, data fusion, and error recognition: traditional EEG processing methods have problems such as redundant feature extraction, high computational complexity, or poor decomposition effects; long short-term memory networks have good ability to process sequential data, but adding a simple feature weighting strategy may not be able to fully capture the complex relationships in the input data, resulting in a decrease in recognition accuracy; traditional network parameter optimization methods such as grid search and random search may not be able to find the global optimal solution in a short time, or the optimization effect is limited. Summary of the Invention

[0004] To solve the problems existing in the prior art, the present invention provides a method and system for evaluating the cognitive state differences of civil aviation flight crews based on the LSTM (Long Short-Term Memory)-MHSA (Multi-Head Self-Attention) algorithm and a hierarchical coupling framework. By deeply integrating the hierarchical evaluation of individual psychological cognitive abilities and crew psychological cognitive interactions, a "single individual - crew" double-layer evaluation model is constructed to form a closed-loop solution that runs through pre-event prediction, in-event monitoring, and post-event evaluation, and realizes the differential evaluation of the cognitive state of civil aviation flight crews.

[0005] To achieve the above object, the present invention provides the following solutions:

[0006] A method for evaluating the cognitive state differences of civil aviation flight crews based on the LSTM-MHSA algorithm and a hierarchical coupling framework, the method comprising:

[0007] S1: Collect the EEG and psychological cognitive data of pilots before and after the execution of flight tasks, and after preprocessing, reconstruct the scalarized EEG features and psychological cognitive features to form a unified multi-temporal sample set of individual cognitive ability features;

[0008] S2: According to the unified multi-temporal sample set of individual cognitive ability characteristics, construct a deep learning model LSTM-MHSA that integrates LSTM and multi-head self-attention mechanism, and extract the temporal evolution characteristics of pilots' personal cognitive ability and the fusion characteristics of cognitive interaction between the captain and the first officer;

[0009] S3: According to the temporal evolution characteristics of pilots' personal cognitive ability and the fusion characteristics of cognitive interaction between the captain and the first officer, use the LSTM-MHSA model to evaluate the overall ability scores of pilots in different psychological cognitive dimensions and the psychological cognitive cooperation mode between the captain and the first officer during flight tasks, so as to realize the differential evaluation of the cognitive states of civil aviation flight crews.

[0010] Preferably, in S1, the electroencephalogram and psychological cognitive data of pilots are collected before and after the execution of flight tasks, and after preprocessing, the scalarized electroencephalogram features and psychological cognitive features are reconstructed to form a unified multi-temporal sample set of individual cognitive ability characteristics, including:

[0011] S1.1: Before and after the flight task, use an eight-channel electroencephalograph to collect the electrocortical signals of pilots according to the international EEG standard, select the key electrode positions and use conductive paste to ensure stable contact, the reference electrode is located at the earlobe, and the IND lead can be adjusted;

[0012] S1.2: For electroencephalogram data, use a third-order Butterworth band-pass filter with a frequency range of 1 - 60 Hz and a 50 Hz notch filter for denoising. The data is centralized and whitened, and the ICA technique is used to decompose the signal based on negentropy to remove electrooculogram artifacts and channel interference. Finally, the wavelet threshold denoising method is used to process the signal separated by ICA;

[0013] S1.3: For the test result data of the psychological cognitive module, the data beyond 3 times the standard deviation range is removed. The missing values are processed by the mean imputation method, and the original data is standardized. The min-max normalization method is used to linearly transform the data into the interval [0, 1]. The conversion formula is based on the maximum and minimum values of the data, and finally normalization processing is carried out;

[0014] S1.4: Apply variational mode decomposition to extract energy entropy, information entropy, sample entropy and power spectral density as electroencephalogram features. Calculate the energy entropy of different intrinsic mode functions, use information entropy to evaluate the value of information in electroencephalogram signals, use sample entropy to measure the complexity of time series, and estimate the power spectrum of the signal by the periodogram method;

[0015] S1.5: Select the scores of perceptual speed, memory span, N-back task accuracy rate, attention-stroop effect, spatial psychological cognitive error, arithmetic ability and psychomotor ability in the scale test as the core indicators for evaluating the six psychological cognitive dimensions of memory, attention, intelligence, spatial perception ability, reaction speed and professional ability;

[0016] S1.6: The EEG and psychological cognitive features obtained at multiple sampling moments before and after flight are subjected to standardization processing to form a unified multivariate time-series sample of individual cognitive ability features.

[0017] Preferably, in the above S1.4, applying variational mode decomposition to extract energy entropy, information entropy, sample entropy, and power spectral density as EEG features includes:

[0018] Using the VMD algorithm to solve the variational problem, the original EEG signal f(t) is decomposed into K

[0019]

[0020] where ω k is the central frequency of the k-th IMF, δ(t) is the Dirac function, and u k represents the k-th intrinsic mode function (IMF), and u k (t) represents the instantaneous amplitude of the k-th IMF at time t, which is a function of time t. ω k (t) represents the instantaneous frequency of the k-th IMF, which is a function of time t and describes the local frequency characteristics of the IMF at time t. j represents the imaginary unit, that is in the complex domain is used to describe the analytic representation of the signal, where u k (t) is the IMF time series to be solved, ω k (t) is the corresponding instantaneous frequency, and j is used in the exponential term to construct the analytic signal;

[0021] Calculate the energy entropy, information entropy, sample entropy, and power spectral density based on each IMF.

[0022] Preferably, in the above S1.6, the EEG and psychological cognitive features obtained at multiple sampling moments before and after flight are subjected to standardization processing to form a unified multivariate time-series sample of individual cognitive ability features, including:

[0023] Let the cognitive ability observation value of pilot i at the t-th sampling moment be where D = 10 is the dimension of the cognitive ability index integrating EEG and psychological tests, and where represents the EEG features of pilot i at the t-th sampling moment, including four key features: energy entropy, information entropy, sample entropy, and power spectral density, which are converted into scalar values in the range of [0, 1] through standardization processing of the four key features;

[0024] Denote the psychological and cognitive test results of pilot \(i\) at the \(t\)-th sampling moment, including the test scores of six dimensions: memory, attention, intelligence, spatial perception, reaction ability, and professional ability, and normalize them to scalar values within the interval \([0, 1]\);

[0025] Concatenate the scalarized EEG features with the normalized psychological and cognitive test scores to obtain the complete cognitive ability observation value of pilot \(i\) at the \(t\)-th sampling moment:

[0026] Take the complete cognitive ability sequence of pilot \(i\) as the input of the LSTM-MHSA model, where \(T\) is the number of sampling time steps.

[0027] Preferably, in step S2, based on the unified individual cognitive ability characteristic multi-temporal sample set, construct a deep learning model LSTM-MHSA that combines LSTM and the multi-head self-attention mechanism, and extract the temporal evolution characteristics of the pilot's individual cognitive ability and the cognitive interaction fusion characteristics between the captain and the first officer, including:

[0028] S2.1: Design a deep learning model LSTM-MHSA that combines LSTM and MHSA. The model uses a multi-layer bidirectional LSTM as the backbone and introduces MHSA to model the internal relationships within the sequence;

[0029] S2.2: Take the complete psychological and cognitive ability sequence of pilot \(i\) as the input of the model, and use the LSTM-MHSA model to perform hierarchical encoding on the temporal data of the pilot's psychological and cognitive ability to obtain cognitive feature representations at different scales and different abstraction levels;

[0030] S2.3: Use the MHSA mechanism to model the psychological and cognitive interaction relationship between the captain and the first officer within the crew;

[0031] S2.4: Train the LSTM-MHSA model in an end-to-end manner to achieve the adaptive extraction of the individual-crew level coupling features from the original multi-psychological and cognitive ability data. Use the cross-entropy loss function to measure the deviation between the features extracted by the model and the actual psychological and cognitive state of the pilot. Adopt the Adam adaptive optimization algorithm to iteratively update the model parameters, minimize the loss function, and use methods such as Dropout and L2 regularization to prevent the model from overfitting.

[0032] Preferably, in step S2.1, design a deep learning model LSTM-MHSA that combines LSTM and MHSA. The model uses a multi-layer bidirectional LSTM as the backbone and introduces MHSA to model the internal relationships within the sequence, including:

[0033] The overall structure of the LSTM-MHSA model is expressed as:

[0034] H (0) = Embedding(X)

[0035] H (l) = BiLSTM (l) (H (l-1) ), l = 1, 2, …, L

[0036] U = MHSA(H (L) )

[0037] o = Softmax(FC(U))

[0038] Among them, the Embedding layer maintains a weight matrix, where each row corresponds to the vector representation of a discrete feature. During the training process, the vectors will be continuously updated and optimized along with the model parameters, and finally learn the semantic information of the input features. X = {x1, x2, …, x T} is the time-series data of the pilot's original psychological cognitive ability, is the input embedding representation, L is the number of LSTM layers, is the hidden state sequence of the l-th layer of LSTM, is the output sequence of MHSA, is the predicted probability distribution of the pilot's psychological cognitive state;

[0039] Taking the multi-layer bidirectional LSTM as the backbone includes:

[0040]

[0041] Among them, and respectively represent the forward and backward LSTM cells, is and 's hidden state at time step t, is the complete hidden state of the bidirectional LSTM at time t. The bidirectional LSTM can comprehensively utilize the information of multiple tests by processing the sequence in both forward and backward directions;

[0042] Introducing MHSA to model the internal relationships of the sequence includes:

[0043]

[0044] Among them, and are learnable linear transformation matrices, is the output of the n-th attention head, N is the number of attention heads, d k = 2dh / N is the hidden layer dimension for each head, and the Attention(·) function calculates the normalized dot-product attention:

[0045]

[0046] The multi-head self-attention mechanism learns the relationships between sequence elements in different subspaces by calculating multiple attention functions in parallel, enhancing the model's feature representation ability; finally, the outputs of all heads are concatenated and linearly transformed to obtain the high-level feature representation U that fuses different-scale dependency relationships.

[0047] Preferably, in S2.2, the complete psychological cognitive ability sequence of pilot i is used as the input of the model, and the LSTM-MHSA model is used to hierarchically encode the time-series data of the psychological cognitive ability of a single pilot to obtain cognitive feature representations at different scales and different abstraction levels, including:

[0048] The bidirectional LSTM network is used to encode the psychological cognitive ability sequence, and the forward LSTM gradually updates the hidden state

[0049]

[0050] where are the parameters of the forward LSTM;

[0051] The reverse LSTM updates the hidden state in reverse order

[0052]

[0053] where are the parameters of the reverse LSTM;

[0054] The forward and reverse hidden states are concatenated to obtain the bidirectional LSTM encoding at the t-th step

[0055] The encoding sequence of the bidirectional LSTM is fed into the stacked multi-layer LSTM to achieve hierarchical abstraction of the psychological cognitive features. The update formula for the hidden state of the l-th layer LSTM is:

[0056]

[0057] where represents the output of the bidirectional LSTM, and θ (l) are the parameters of the l-th layer LSTM;

[0058] After stacking L layers of LSTM, the multi-scale psychological cognitive features of pilot i at time t are obtained

[0059] Adaptive aggregation of the outputs of multiple layers of LSTM through a gating mechanism to obtain the final psychological cognitive ability characteristics of pilot \(i\) at time \(t\).

[0060]

[0061] Among them, \(W\) g and \(b\) g are the parameters of the gating layer, \(\sigma(\cdot)\) is the sigmoid activation function, and \(e\) represents the Hadamard product;

[0062] Extract multi-scale time-series features that depict the evolution pattern of pilots' individual psychological cognitive abilities through bidirectional LSTM, multi-layer LSTM, and gating aggregation

[0063] Preferably, in the step S2.3, using the MHSA mechanism to model the psychological cognitive interaction relationship between the pilot and the co-pilot in the crew includes:

[0064] Let the multi-dimensional psychological cognitive ability characteristics of the pilot and the co-pilot within the \(t\)-th time window be and Then the psychological cognitive ability of the crew at time \(t\) can be expressed as where \(d\) is the dimension of the individual psychological cognitive characteristics, and a complete flight mission is divided into \(T\) time windows, that is, the psychological cognitive ability sequence \(\{s_1, s_2, \cdots, s\) T \}\);

[0065] MHSA maps the crew's psychological cognitive ability sequence \(S = [s_1, s_2, \cdots, s\) T · to the query matrix \(Q\), key matrix \(K\), and value matrix \(V\) through linear transformation:

[0066] \(Q = SW\) Q , \(K = SW\) K , \(V = SW\) V

[0067] Among them, is a learnable linear transformation matrix, and \(d\) h is the hidden layer dimension of the self-attention mechanism;

[0068] Calculate the attention weight matrix \(A\) between different time steps:

[0069]

[0070] Among them, \(A\) ij ​Denote the attention weight of the $i$-th time window to the $j$-th time window. The larger the weight, the higher the psychological cognitive interaction intensity of the crew members within the two time windows;

[0071] Apply the attention weight matrix to the value matrix $V$ to obtain the self-attention output matrix $H$:

[0072]

[0073] MHSA adopts a multi-head parallel computing strategy. Suppose there are $N$ attention heads, then the attention output matrix of the $n$-th head is:

[0074] $H$ (n) $=$ (n) $A$ (n) $V$, $n = 1, 2, \cdots, N$

[0075] where $A$ (n) and $V$ (n) are the attention weight matrix and value matrix independently learned by the $n$-th head respectively;

[0076] Concatenate the outputs of all heads and obtain the final output matrix $U$ of MHSA through a linear transformation:

[0077] $U = [H$ (1) ; $H$ (2) ; $\cdots$; $H$ (N) $W$ O

[0078] where is the linear transformation matrix, and each row vector $u$ t of $U$ represents the crew psychological cognitive characteristics integrating the interaction information of the pilot and co-pilot;

[0079] Integrate the individual psychological cognitive ability feature extraction and crew psychological cognitive interaction feature extraction to obtain a hierarchical psychological cognitive feature sequence $\{u_1, u_2, \cdots, u$ T $\}$ that takes into account both individual psychological cognition and crew interaction, describing the change of the psychological cognitive state of the pilot and co-pilot during the flight mission.

[0080] Preferably, in step S3, according to the temporal evolution characteristics of the pilot's personal cognitive ability and the cognitive interaction fusion characteristics of the pilot and co-pilot, use the LSTM-MHSA model to evaluate the overall ability scores of the pilot in different psychological cognitive dimensions and the psychological cognitive coordination mode of the pilot and co-pilot during the flight mission, and the differential evaluation of the cognitive state of the civil aviation crew includes:

[0081] S3.1: Load the trained LSTM-MHSA model, configure the model's running environment, and convert the time-series data of individual pilots' psychological cognitive abilities into the input format required by the model. Among them, the 3D tensor is the number of samples × multiple time steps × feature dimensions. For the evaluation at the crew level, align the psychological cognitive ability data of the pilot and co-pilot in time and splice them into an overall input;

[0082] S3.2: Obtain the output of the LSTM-MHSA model, including the psychological cognitive ability feature sequence at the individual level and the psychological cognitive interaction weight matrix at the crew level; perform average pooling on the feature sequence at the individual level to obtain the overall ability scores of the pilot in different psychological cognitive dimensions, and analyze the psychological cognitive interaction weight matrix at the crew level to identify the psychological cognitive cooperation patterns between the pilot and co-pilot during the flight mission;

[0083] S3.3: Regularly fine-tune and update the model using newly collected pilot data;

[0084] Among them, the specific steps of S3.1 are as follows: First, load the pre-trained LSTM-MHSA model parameters θ * , and configure the software and hardware environment required for the model to run; then, convert the time-series data of individual pilots' cognitive abilities into the tensor format required by the model; specifically, for the evaluation of a single pilot, construct a three-dimensional tensor where N = 1 is the number of samples, T is the time step, and D is the cognitive feature dimension. For the evaluation at the crew level, align the cognitive ability sequences of the pilot and co-pilot in the time dimension and splice them in the sample dimension to form a crew input tensor with a shape of ;

[0085] The specific steps of S3.2 are as follows: By parsing the output of the LSTM-MHSA forward inference process, obtain the cognitive competence evaluation results at two levels; at the individual level, perform average pooling on the features of each dimension in the LSTM hidden state sequence H (i) to obtain the comprehensive scoring vector

[0086]

[0087] Among them, represents the j-th dimensional feature value of the LSTM hidden state sequence of the i-th pilot at time step t, which is an element in the hidden state matrix H (i) output by the LSTM model; d h represents the dimension of the LSTM hidden state, that is, the number of neurons in the hidden layer, which determines the dimension of the final scoring vector s (i) ; s(i) Fully characterize the pilot's cognitive performance throughout the entire mission process, reflecting the levels of their abilities such as attention allocation, information processing, decision-making, etc.; at the same time, perform clustering analysis on the output matrix U of MHSA to identify the cognitive interaction patterns between the pilot and co-pilot at critical flight nodes.

[0088] The present invention also provides a civil aviation crew cognitive state difference evaluation system based on the LSTM-MHSA algorithm and a hierarchical coupling framework. The system is used to implement any one of the above methods. The system includes: a reconstruction module, an extraction module, and an evaluation module;

[0089] The reconstruction module is used to collect the electroencephalogram (EEG) and psychological cognitive data of the pilot before and after the execution of the flight mission respectively, and after preprocessing, reconstruct the scalarized EEG features and psychological cognitive features to form a unified individual cognitive ability feature multi-temporal sample set;

[0090] The extraction module is used to construct a deep learning model LSTM-MHSA that combines LSTM and the multi-head self-attention mechanism based on the unified individual cognitive ability feature multi-temporal sample set, and extract the temporal evolution features of the pilot's personal cognitive ability and the cognitive interaction fusion features between the pilot and co-pilot;

[0091] The evaluation module is used to use the LSTM-MHSA model to evaluate the overall ability scores of the pilot in different psychological cognitive dimensions and the psychological cognitive cooperation mode between the pilot and co-pilot during the flight mission according to the temporal evolution features of the pilot's personal cognitive ability and the cognitive interaction fusion features between the pilot and co-pilot, and realize the difference evaluation of the cognitive state of the civil aviation crew.

[0092] Compared with the prior art, the beneficial effects of the present invention are:

[0093] (1) The traditional EEG acquisition process faces challenges due to insufficient flexibility and comfort. Conducting data acquisition during the flight may not only pose a potential obstacle to the pilot's normal flight operations, but also wearing the EEG acquisition device continuously will exacerbate the pilot's discomfort and increase the complexity and overall cost of the operation. The present invention proposes an innovative experimental acquisition scheme, that is, only conduct EEG data acquisition at two critical time points before and after the flight, improving the convenience and comfort of data acquisition, avoiding potential interference with the pilot's normal flight operations, and reducing the complexity and time cost of the operation.

[0094] (2) The traditional pilot competency assessment system mainly relies on subjective judgment and past experience, resulting in assessment results often being accompanied by a large degree of uncertainty. Compared with the traditional assessment methods that rely on subjective judgment and experience, the assessment method based on physiological and psychological data proposed in the present invention, which integrates pre-flight and post-flight EEG data and psychological cognitive ability test data, has significant scientific and reliable advantages. Through objective data analysis, this method can more accurately identify potential problems in pilots' competency, thereby effectively reducing the impact of subjective judgment on assessment results.

[0095] (3) The traditional identification models for pilots' lack of competency often adopt a single method or algorithm and are difficult to fully utilize the potential information of data. The present invention innovatively proposes a civil aviation crew cognitive state difference assessment system. This model adds a multi-head self-attention mechanism to the basic framework of LSTM, enabling the model to better capture the relationships between different parts of the input sequence, significantly enhancing the model's processing ability, computational efficiency, expression ability, and generalization ability for complex sequence data. This method can pay attention to the changes in pilots' psychological cognitive states, timely warning of high-risk periods of psychological cognitive ability decline. At the same time, it can locate the key psychological cognitive nodes and interaction patterns that affect pilots' psychological cognitive competency, making the assessment results more transparent and easier for relevant personnel to take improvement actions. BRIEF DESCRIPTION OF THE DRAWINGS

[0096] In order to more clearly illustrate the technical solutions of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0097] Figure 1 It is a schematic diagram of the main steps S1 - S3 described in the embodiments of the present invention;

[0098] Figure 2 It is a schematic diagram of the eight-lead positions of the pilot's electroencephalogram in the embodiments of the present invention (top view of the cranial vertex);

[0099] Figure 3 It is a schematic diagram of the LSTM - MHSA algorithm and the hierarchical coupling framework in the embodiments of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0100] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are only some, rather than all, embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts fall within the scope of protection of the present invention.

[0101] To make the above objects, features, and advantages of the present invention more obvious and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0102] Embodiment 1

[0103] As Figure 1 、 Figure 3 shown, the present invention provides a method for evaluating the cognitive state differences of civil aviation flight crews based on the LSTM-MHSA algorithm and a hierarchical coupling framework, including the following steps:

[0104] S1: Data collection and preprocessing. Before and after the flight mission, collect the electroencephalogram (EEG) data of the pilots and conduct psychological cognitive assessments. Process the EEG signals to extract energy entropy, information entropy, sample entropy, and power spectral density. Extract memory, attention, intelligence, spatial perception, reaction ability, and professional ability from the psychological cognitive tests as psychological cognitive ability characteristics, and reconstruct them with the scalarized EEG characteristics to form a unified individual cognitive ability characteristic multi-temporal sample set. The specific steps are as follows:

[0105] S1.1 To accurately capture the changes in the psychological cognitive state of pilots before and after performing tasks, the present invention specifically designs a simple and efficient EEG collection scheme for pilots. The experimenters select key electrode positions closely related to the psychological cognitive ability of the pilots and use conductive paste to ensure stable signal contact, minimizing interference to the pilots. At the same time, to ensure the purity and reliability of the data, the pilots are required to maintain a resting state during the collection process to exclude the influence of environmental noise and other factors as much as possible. Different from conventional EEG experiments, the present invention fully considers the actual needs of this special occupational group of pilots and optimizes the number and position of electrodes to obtain key psychological cognitive information at a relatively low cost. This EEG data collection scheme not only improves the experimental efficiency but also lays a foundation for the subsequent construction of a model for evaluating the cognitive state differences of civil aviation flight crews.

[0106] Use an eight-channel EEG instrument to perform non-invasive electrical signal collection on the cerebral cortex of the pilots. This process strictly follows the international electroencephalogram (EEG) electrode placement standard, selects key electrode positions such as Fp1, Fp2, C3, C4, T3, T4, O1, and O2, and uses conductive paste to ensure a stable and low-impedance contact between the Ag / AgCl electrode and the scalp. The reference electrodes are respectively located at A1 (left earlobe) and A2 (right earlobe) to enhance the accuracy and stability of signal collection. The position of the IND lead can be flexibly adjusted according to experimental needs during actual operation to adapt to the individual differences of different pilots.

[0107] According to Figure 2The schematic diagram of the eight - lead EEG positions (top - view of the skull) shown is for a pilot wearing an EEG instrument to ensure that each electrode is accurately placed at the predetermined position. During the acquisition process, the pilot needs to maintain a resting state to minimize the potential interference of external environmental noise and physiological activities on the EEG signals, thus ensuring that the acquired EEG data has a high degree of purity and reliability.

[0108] The pre - processing process of EEG data in S1.2 includes: using a third - order Butterworth band - pass filter from 1 - 60 Hz and a 50 Hz notch filter for denoising; the data is centered and whitened (through covariance matrix and eigen - decomposition); the ICA technique is used to decompose the signal based on negentropy to remove electro - oculogram artifacts and channel interference; finally, the wavelet threshold denoising method is used to process the signal separated by ICA.

[0109] According to the iterative formula of FastICA, the weight vector w is continuously updated until it converges. Through independent component analysis, several components are decomposed. The correlation coefficients between the decomposed components and the data recorded on the scalp are calculated, and then the maximum value of the components is removed, so that the electro - oculogram artifacts and the noise components generated by the EEG of the surrounding channels can be removed, leaving a clean EEG signal. Then, the wavelet threshold denoising method is used to further denoise the EEG signal separated by FastICA. The original signal is decomposed by wavelet and processed by wavelet threshold, then the wavelet coefficients of different scales are extracted, and finally the signal is reconstructed by inverse wavelet transform. The sym8 wavelet basis is used for three - layer wavelet down - decomposition and then threshold processing to suppress noise. According to the selection of wavelet basis and decomposition layers and considering the denoising effect comprehensively, the signal is segmented for three - layer wavelet decomposition and then the soft - threshold wavelet threshold method is used to further denoise the EEG signal.

[0110] The pre - processing process of the test result data of the psychological cognitive module in S1.3 includes the processing of outliers, that is, removing the data outside the range of 3 times the standard deviation to ensure data accuracy; the processing of missing values uses the mean imputation method; the standardization of the original data uses the min - max standardization method to linearly transform the data into the interval [0, 1], and the conversion formula is based on the maximum and minimum values of the data; finally, normalization processing is carried out.

[0111] For the processing of outliers, the data outside the range of ±3 times the standard deviation is removed; for the processing of missing values, the mean imputation method is used; for standardization processing, the min - max standardization method is used to linearly transform the original data so that it is mapped between [0, 1];

[0112] Normalization processing converts negative - direction indicators into positive - direction indicators and makes the results fall between [0, 1].

[0113] S1.4 uses the VMD algorithm to adaptively decompose the EEG signals, extract multiple intrinsic mode functions, and calculate a series of non-linear dynamic features based on the intrinsic mode functions (IMFs), including energy entropy, information entropy, sample entropy, and power spectral density.

[0114] First, the VMD algorithm decomposes the original EEG signal f(t) into K IMFs u k (t) by solving the following variational problem:

[0115]

[0116] where ω k is the central frequency of the k-th IMF, δ(t) is the Dirac function, u k represents the k-th intrinsic mode function (IMF), u k (t) represents the instantaneous amplitude of the k-th IMF at time t, which is a function of time t, ω k (t) represents the instantaneous frequency of the k-th IMF, which is a function of time t and describes the local frequency characteristics of the IMF at time t, j represents the imaginary unit, i.e., used in the complex domain to describe the analytic representation of the signal, where u k (t) is the IMF time series to be solved, ω k (t) is the corresponding instantaneous frequency, and j is used in the exponential term to construct the analytic signal. Solving the above problem yields a set of IMFs {u1(t), u2(t), …, u K (t)}.

[0117] Next, the following features are calculated based on each IMF:

[0118] (1) Energy entropy: Characterizes the uniformity of the energy distribution of the IMF. The larger the energy entropy, the more uniform the energy distribution of the IMF.

[0119] (2) Information entropy: Evaluates the richness of information in the IMF. The larger the information entropy, the more information the IMF contains.

[0120] (3) Sample entropy: Measures the complexity and unpredictability of the IMF. The larger the sample entropy, the stronger the unpredictability of the IMF.

[0121] (4) Power spectral density: Describes the energy distribution of the IMF at different frequencies. The power spectral density reflects the oscillation intensity and energy concentration degree of the IMF in each frequency band.

[0122] S1.5 Select the scores on the perception speed, memory span, accuracy rate of the N-back task, attention-Stroop effect, spatial mental cognitive error, arithmetic ability, and psychomotor ability in the scale test as the core indicators for evaluating the six major psychological cognitive dimensions of memory, attention, intelligence, spatial perception ability, reaction speed, and professional ability.

[0123] Pilots will undergo a psychological cognitive module test before and after flights to evaluate the changes, similarities, and differences in their psychological competence characteristics before and after flights. The question types in the pilot psychological cognitive module test used in the present invention consider pilots' different psychological cognitive abilities from the following dimensions: memory, attention, intelligence, spatial perception, reaction, and professional ability. The test content includes the evaluation of key abilities such as perception speed, N-back memory, short-term memory span, attention, psychomotor, spatial mental cognition, and arithmetic ability. The features extracted in the present invention can be widely adapted to various pilots' psychological scales. The following is a detailed description of the feature extraction in this step:

[0124] (1) Memory feature extraction

[0125] N-back memory test score: Through the N-back task (N takes values of 1, 2, 3, etc., representing the step length of memory recall), evaluate the pilot's ability in memory retention and update. Specific features include the proportion of correct answers, reaction time, and error types (including omissions, incorrect substitutions, etc.).

[0126] Short-term memory span: Use tests such as digit span and word span to measure the amount of information that pilots can accurately remember and repeat in a short time. Features include the maximum memory sequence length, error rate, etc.

[0127] (2) Attention feature extraction

[0128] Attention concentration ability: Evaluate the pilot's ability to maintain attention concentration in long-term tasks through continuous performance tests (including visual tracking tasks). Features include average reaction time, number of attention distractions, error rate, etc.

[0129] Attention allocation ability: Use dual-task or multi-task tests to examine the pilot's efficiency in processing multiple information sources simultaneously. Features include task switching speed, task completion quality, etc.

[0130] (3) Intelligence feature extraction

[0131] Logical reasoning ability: Evaluate the pilot's abstract thinking and problem-solving abilities through tests such as solving logical problems and graphic reasoning. Features include correct answer rate, problem-solving speed, etc.

[0132] Knowledge application ability: Problem-solving tasks that combine professional knowledge are used to examine the pilot's ability to apply theoretical knowledge to practical situations. The characteristics include the accuracy and efficiency of problem-solving.

[0133] (4) Feature extraction of spatial perception

[0134] Spatial orientation ability: Tests such as three-dimensional space rotation and spatial direction judgment are used to evaluate the pilot's understanding and operation ability of the spatial structure. The characteristics include the proportion of correct judgments, reaction time, etc.

[0135] Visual-spatial integration ability: Through visual-spatial memory tasks, such as map memory and graphic reconstruction, the pilot's ability to transform visual information into spatial mental cognition is measured. The characteristics include memory accuracy, reconstruction efficiency, etc.

[0136] (5) Feature extraction of reaction ability

[0137] Simple reaction time: Through simple stimulus-response tasks, the pilot's immediate reaction speed to sudden stimuli is measured. The characteristics include average reaction time, fastest reaction time, etc.

[0138] Choice reaction time: Select the correct one from multiple possible reactions to evaluate the pilot's rapid decision-making ability in complex situations. The characteristics include the reaction time of correct selection, error rate, etc.

[0139] (6) Feature extraction of professional ability

[0140] Mastery of flight-related knowledge: Through professional knowledge tests such as flight theory and flight rules, the pilot's professional quality is evaluated. The characteristics include the comprehensiveness and accuracy of knowledge mastery, etc.

[0141] Performance in scenario simulation: Using flight simulators or virtual reality technology to simulate emergency situations during flight to evaluate the pilot's decision-making and operation skills under pressure. The characteristics include the effectiveness of emergency handling, time management, etc.

[0142] Select the scores of the pilot's age, perceptual speed reaction time, short-term memory span, N-back accuracy rate, attention-stroop effect (ability to exclude interference), spatial mental cognition (distance and speed perception error), arithmetic ability, and psychomotor ability as feature indicators to analyze the changes in the pilot's mental state. The proposed method has universality and compatibility and can be widely applied to various scales that comprehensively evaluate the pilot's mental ability from six core dimensions: memory, attention, intelligence level, spatial perception acuity, reaction speed, and professional skills.

[0143] The input of the S1.6 model consists of the EEG and psychological cognitive features obtained at multiple sampling moments before and after flight, which are scalarized to form a unified multi-temporal sample of individual cognitive ability features.

[0144] Let the cognitive ability observation value of pilot i at the t-th sampling moment be where D = 10 is the dimension of the cognitive ability index that fuses EEG and psychological tests. Among them represents the EEG features of pilot i at the t-th sampling moment, including four key features: energy entropy, information entropy, sample entropy, and power spectral density. By standardizing these four features, they are converted into scalar values in the range of [0, 1]. Specifically, for the energy entropy feature First, calculate the minimum value E of the energy entropy in the entire dataset min and the maximum value E max , and then apply the min-max normalization formula to convert it into a scalar value in the range of [0, 1] Similarly, for the information entropy feature the sample entropy feature and the total power feature perform min-max normalization to obtain scalar values The scalarized EEG feature vector can be expressed as

[0145] represents the psychological cognitive test results of pilot i at the t-th sampling moment, including the test scores of six dimensions: memory, attention, intelligence, spatial perception, reaction ability, and professional ability, and are normalized to scalar values in the range of [0, 1].

[0146] Concatenate the scalarized EEG features with the normalized psychological cognitive test scores , where contains the psychological cognitive test scores of 6 dimensions (from to ), and all are normalized to the 0-1 interval. Therefore, its expression is Obtain the complete cognitive ability observation value of pilot i at the t-th sampling moment:

[0147] Take the complete cognitive ability sequence of pilot i as the input of the LSTM-MHSA model, where T is the number of sampling time steps.

[0148] S2: Implement feature extraction using the training algorithm model. Construct a deep learning model LSTM-MHSA that integrates LSTM and the multi-head self-attention mechanism. Use LSTM to extract the temporal evolution features of the pilot's individual psychological cognitive ability, and use MHSA to capture the crew's psychological cognitive interaction feature matrix, and measure the deviation between the model output and the true psychological cognitive state. The specific steps are as follows:

[0149] S2.1 Construct the LSTM-MHSA model structure and design a deep learning model that integrates LSTM (Long Short-Term Memory Network) and MHSA (Multi-Head Self-Attention Mechanism). The model uses a multi-layer bidirectional LSTM as the backbone, which can effectively extract the long-term and short-term dependence features in the temporal data. On the basis of LSTM, MHSA is introduced to model the internal relationships in the sequence and enhance the model's ability to capture key features.

[0150] The core of the LSTM-MHSA model is to integrate two deep learning architectures, LSTM and MHSA, aiming to fully explore the temporal dependence relationships and key features in the pilot's psychological cognitive ability data. The overall structure of the model can be expressed as:

[0151]

[0152] Among them, the Embedding layer maintains a weight matrix, where each row corresponds to the vector representation of a discrete feature. During the training process, these vectors will be continuously updated and optimized together with the model parameters, and finally learn the semantic information of the input features. X = {x1, x2, L, x T} is the pilot's original temporal data of psychological cognitive ability, is the input embedding representation, L is the number of LSTM layers, is the hidden state sequence of the l-th layer of LSTM, is the output sequence of MHSA, is the predicted probability distribution of the pilot's psychological cognitive state.

[0153] In the LSTM part, a multi-layer bidirectional LSTM is used to model the temporal evolution law of the pilot's psychological cognitive ability. The calculation process of the l-th layer of BiLSTM is as follows:

[0154]

[0155] Among them, and respectively represent the forward and backward LSTM units, is their hidden state at time step t, is the complete hidden state of the bidirectional LSTM at time t. The bidirectional LSTM can comprehensively utilize the information from multiple tests by processing the sequence in both forward and backward directions.

[0156] In the MHSA part, a multi-head self-attention mechanism is used to model the dependencies between pilots at different time steps and capture the internal correlation pattern of the psychological cognitive ability sequence. Specifically, for the output sequence of the l-th layer LSTM The calculation process of MHSA is:

[0157]

[0158] in, and is a learnable linear transformation matrix, is the output of the nth attention head, N is the number of attention heads, and d k =2d h / N is the hidden layer dimension of each head. The Attention(·) function calculates the normalized dot product attention:

[0159]

[0160] The multi-head self-attention mechanism can learn the relationship between sequence elements in different subspaces by calculating multiple attention functions in parallel, thus enhancing the feature representation ability of the model. Finally, the outputs of all heads are concatenated and linearly transformed to obtain a high-level feature representation U that integrates dependencies of different scales.

[0161] In summary, the LSTM-MHSA model combines the ability of LSTM to capture long-term and short-term temporal dependencies and the advantages of MHSA in modeling complex internal associations of sequences to form an end-to-end pilot psychological cognitive ability assessment framework. The temporal features are extracted through multi-layer bidirectional LSTM, and the key patterns within the sequence are mined using the multi-head self-attention mechanism.

[0162] S2.2 The complete psychological cognitive ability sequence of pilot i As the input of the model, the LSTM-MHSA model is used to hierarchically encode the time series data of the psychological cognitive ability of a single pilot to obtain cognitive feature representations at different scales and levels of abstraction.

[0163] First, a bidirectional LSTM network is used to encode the psychological cognitive ability sequence. The forward LSTM gradually updates the hidden state

[0164]

[0165] in is the parameter of the forward LSTM. The reverse LSTM updates the hidden state in reverse order

[0166]

[0167] in is the parameter of the reverse LSTM. By concatenating the forward and reverse hidden states, we can get the bidirectional LSTM encoding of the tth step.

[0168] Next, the encoding sequence of the bidirectional LSTM The data is fed into a stack of multi-layer LSTMs to achieve hierarchical abstraction of psychological cognitive features. The hidden state update formula of the l-th layer LSTM is:

[0169]

[0170] in represents the output of bidirectional LSTM, θ (l) is the parameter of the lth layer LSTM. After stacking L layers of LSTM, the multi-scale psychological cognitive characteristics of pilot i at time t can be obtained:

[0171] Finally, the output of multi-layer LSTM is adaptively aggregated through the gating mechanism to obtain the final psychological cognitive ability characteristics of pilot i at time t

[0172]

[0173] Where W g and b g is the parameter of the gating layer, σ(·) is the sigmoid activation function, and e represents the Hadamard product. The gating layer can adaptively adjust the importance of psychological cognitive features at different levels of abstraction at different times and capture the dynamic changes of the pilot’s psychological cognitive state.

[0174] In summary, through bidirectional LSTM, multi-layer LSTM and gated aggregation, we can extract multi-scale time series features that describe the evolution pattern of individual pilots’ psychological cognitive abilities. Provide rich feature representation for subsequent hierarchical coupling analysis.

[0175] S2.3 Extraction of crew psychological cognitive interaction features. The extraction of crew psychological cognitive interaction features aims to use the MHSA mechanism to model the psychological cognitive interaction relationship between the pilot and the co-pilot during the flight, and to explore the interaction patterns of the two parties in different psychological cognitive dimensions and different time scales. Suppose the multivariate psychological cognitive ability characteristics of the pilot and the co-pilot in the tth time window are and Then the psychological cognitive ability of the crew at time t can be expressed as where d is the dimension of the individual's psychological and cognitive characteristics. Divide a complete flight mission into T time windows, and the psychological and cognitive ability sequence of the flight crew {s1, s2, …, s T} can be obtained.

[0176] MHSA first maps the psychological and cognitive ability sequence S = [s1, s2, …, s T · into the query (Q) matrix, key (K) matrix, and value (V) matrix through linear transformation:

[0177] Q = SW Q , K = SW K , V = SW V (10)

[0178] where is a learnable linear transformation matrix, and d h is the hidden layer dimension of the self-attention mechanism. Next, calculate the attention weight matrix A between different time steps:

[0179]

[0180] where A ij represents the attention weight of the i-th time window to the j-th time window. The greater the weight, the higher the psychological and cognitive interaction intensity of the crew members within the two time windows. Apply the attention weight matrix to the value matrix V to obtain the self-attention output matrix H:

[0181]

[0182] To characterize the psychological and cognitive interaction patterns of the captain and first officer from different perspectives, MHSA adopts a multi-head parallel computing strategy. Suppose there are N attention heads, then the attention output matrix of the n-th head is:

[0183] H (n) = A (n) V (n) , n = 1, 2, …, N (13)

[0184] where A (n) and V (n) are the attention weight matrix and value matrix independently learned by the n-th head, respectively. Finally, concatenate the outputs of all heads and obtain the final output matrix U of MHSA through linear transformation:

[0185] U = [H (1) ; H (2) ; …; H (N) W O (14)

[0186] where is a linear transformation matrix. U Each row vector u t represents the crew's psychological and cognitive characteristics that integrate the interaction information between the captain and the first officer.

[0187] By combining the two steps of individual psychological and cognitive ability feature extraction and crew psychological and cognitive interaction feature extraction, a hierarchical psychological and cognitive feature sequence {u1, u2, …, u T} can be obtained, which is used to describe the changes in the psychological and cognitive states of the captain and the first officer during the flight mission.

[0188] S2.4 Model Training and Optimization: In the model training and optimization stage, the LSTM-MHSA model is trained in an end-to-end manner, with the goal of minimizing the difference between the psychological and cognitive features extracted by the model and the true psychological and cognitive states of the pilots. First, the collected original multivariate psychological and cognitive ability dataset D of pilots is divided into a training set D train , a validation set D val and a test set D test , with proportions of 70%, 10% and 20% respectively.

[0189] For each sample (X (i) , y (i) ) ∈ D train , where is the original psychological and cognitive ability sequence of the i-th pilot, and y (i) ∈ {0, 1} C is the corresponding psychological and cognitive state label (such as normal, fatigued, inattentive, etc.). The LSTM-MHSA model first extracts the individual psychological and cognitive feature sequence and then fuses the crew interaction features to obtain the final hierarchical coupled psychological and cognitive feature sequence

[0190] After each training epoch, the performance of the model is evaluated on the validation set D val , and the learning rate is adjusted and it is determined whether to end the training according to the validation loss and the early stopping strategy. The method of guiding the training process through the validation set can effectively prevent the model from overfitting the training data.

[0191] Finally, the trained LSTM-MHSA model is used to evaluate the performance on the test set D test .

[0192] S3: Construct a civil aviation crew cognitive state difference evaluation system. Based on the LSTM-MHSA model, average pooling is performed on the feature sequences at the individual level to obtain the overall ability scores of pilots in different psychological cognitive dimensions. Analyze the psychological cognitive interaction weight matrix at the crew level to identify the psychological cognitive collaboration patterns between the captain and the first officer during flight tasks.

[0193] The specific steps are as follows:

[0194] S3.1 Model call and data input: Load the trained LSTM-MHSA model and configure the running environment of the model. Convert the time-series data of individual pilots' psychological cognitive abilities (obtained in S1) into the input format required by the model, where the 3D tensor is the number of samples × multiple time steps × feature dimensions. For the evaluation at the crew level, align the psychological cognitive ability data of the captain and the first officer in time and concatenate them into a whole input. Specifically as follows:

[0195] First, load the pre-trained LSTM-MHSA model parameters θ * , and configure the software and hardware environment required for the model to run. Then, convert the time-series data of individual pilots' cognitive abilities into the tensor format required by the model. Specifically, for the evaluation of a single pilot, construct a three-dimensional tensor where N = 1 is the number of samples, T is the time step, and D is the cognitive feature dimension. For the evaluation at the crew level, align the cognitive ability sequences of the captain and the first officer in the time dimension and concatenate them in the sample dimension to form a crew input tensor with a shape of .

[0196] S3.2 Model output parsing: Obtain the output of the LSTM-MHSA model, including the psychological cognitive ability feature sequences at the individual level and the psychological cognitive interaction weight matrix at the crew level; perform average pooling on the feature sequences at the individual level to obtain the overall ability scores of pilots in different psychological cognitive dimensions. Analyze the psychological cognitive interaction weight matrix at the crew level to identify the psychological cognitive collaboration patterns between the captain and the first officer during flight tasks. Specifically as follows:

[0197] By parsing the output of the LSTM-MHSA forward inference process, the cognitive competency evaluation results at two levels can be obtained. At the individual level, perform average pooling on the features of each dimension in the LSTM hidden state sequence H (i) to obtain the comprehensive scoring vector

[0198]

[0199] represents the j - dimensional eigenvalue of the LSTM hidden state sequence of the i - th pilot at time step t, which is an element in the hidden state matrix H output by the LSTM model (i) in; d h represents the dimension of the LSTM hidden state, that is, the number of neurons in the hidden layer, which determines the dimension of the final scoring vector s (i) ; s (i) Comprehensively characterize the cognitive performance of pilots throughout the mission process, reflecting the level of their abilities such as attention allocation, information processing, decision - making and judgment. At the same time, perform clustering analysis on the MHSA output matrix U to identify the cognitive interaction patterns between the pilot and co - pilot at key flight nodes, such as information sharing, task collaboration, and abnormal handling. This provides an important reference for evaluating the performance of flight crews

[0200] S3.3 Hyperparameter Tuning and Model Optimization. To ensure that the LSTM - MHSA model can adapt to the changing flight environment and crew combinations, regularly use newly collected pilot data to fine - tune and update the model

[0201] Search for the optimal model structure configuration in a reasonable hyperparameter search space, such as the LSTM hidden layer dimension d h ∈ {32, 64, 128}, the number of MHSA heads n head ∈ {1, 2, 4}, the Dropout rate p drop ∈ [0.1, 0.5]. Regularly use new pilot data to fine - tune the model to continuously improve the generalization performance of the model

[0202] In the hierarchical coupling civil aviation crew cognitive state difference evaluation method, by comparing and analyzing the psychological cognitive abilities of individual pilots before and after performing flight tasks, the dynamic change law of their psychological cognitive state can be accurately characterized. The LSTM - MHSA model adaptively extracts the key psychological cognitive characteristics of pilots in the pre - flight and post - flight stages through deep learning algorithms. In the pre - flight stage, the model focuses on the baseline psychological cognitive level of pilots in the rest state, including the comprehensive performance in dimensions such as attention, memory, reasoning and judgment, and establishes a reference benchmark for individual psychological cognitive abilities. In the post - flight stage, the model focuses on the changes in the psychological cognitive state of pilots after experiencing a complete flight task, and depicts the dynamic characteristics of psychological cognitive functions such as attention allocation, information processing, and decision - making response under continuous workload. By aligning the psychological cognitive feature representations of the pre - flight and post - flight stages in the end - to - end semantic space, this method can capture the fluctuations in the psychological cognitive abilities of pilots during the mission execution process and timely detect signs of degradation in psychological cognitive states that may affect flight safety. Based on the quantitative indicators obtained from the comparative analysis, the system can intelligently generate personalized psychological cognitive ability reports for pilots and provide targeted improvement suggestions, such as attention training, stress management, etc., so as to continuously improve the flight performance competence of pilots

[0203] One of the core innovations of the hierarchical coupling evaluation paradigm is to form an integrated group psychological and cognitive analysis perspective through the psychological and cognitive interaction modeling of the pilot and co-pilot. The LSTM-MHSA model uses the multi-head attention mechanism to explore the psychological and cognitive interaction patterns of the pilot and co-pilot in the whole process of flight tasks at different scales. For each flight stage, the model estimates the attention allocation differences between the pilot and co-pilot at key psychological and cognitive event nodes (including complex weather, emergency handling, etc.), as well as the information transmission characteristics in the decision-making chain, and depicts the integration and complementarity degree of psychological and cognitive interaction between roles. Vertically integrating multiple flight stages throughout the whole process, the model refines the key interaction patterns of the pilot and co-pilot in terms of psychological and cognitive load sharing, abnormal diagnosis, collaborative decision-making, etc., and quantitatively evaluates the emergence level of the overall intelligence of the flight crew. Through the psychological and cognitive interaction modeling of the pilot and co-pilot, it breaks through the limitations of traditional evaluation methods and forms a new idea of quantitative evaluation of group psychological and cognition centered on the "flight crew". This provides an important starting point for deeply understanding the human factor mechanism of flight crew resource management. Based on the comparative analysis of the pilot and co-pilot, the system can diagnose possible psychological and cognitive complementary blind spots within the flight crew, optimize the task division, and provide quantitative references for abnormal judgment and collaborative decision-making in crisis situations, thereby improving the accuracy of judging the flight competence of pilots.

[0204] Example Two

[0205] The present invention also provides a civil aviation flight crew cognitive state difference evaluation system based on the LSTM-MHSA algorithm and the hierarchical coupling framework. The system is used to implement any one of the methods described above. The system includes: a reconstruction module, an extraction module, and an evaluation module;

[0206] The reconstruction module is used to collect the electroencephalogram and psychological and cognitive data of the pilot before and after the execution of the flight task respectively, and after preprocessing, reconstruct the scalarized electroencephalogram features and psychological and cognitive features to form a unified individual cognitive ability feature multi-temporal sample set;

[0207] The extraction module is used to construct a deep learning model LSTM-MHSA that integrates LSTM and the multi-head self-attention mechanism according to the unified individual cognitive ability feature multi-temporal sample set, and extract the temporal evolution features of the pilot's personal cognitive ability and the cognitive interaction fusion features of the pilot and co-pilot;

[0208] The evaluation module is used to evaluate the overall ability scores of the pilot in different psychological and cognitive dimensions and the psychological and cognitive collaboration patterns of the pilot and co-pilot during the flight task according to the temporal evolution features of the pilot's personal cognitive ability and the cognitive interaction fusion features of the pilot and co-pilot, and use the LSTM-MHSA model to realize the difference evaluation of the cognitive state of the civil aviation flight crew.

[0209] The embodiments described above are only descriptions of the preferred embodiments of the present invention and do not limit the scope of the present invention. Without departing from the design spirit of the present invention, various deformations and improvements made by those of ordinary skill in the art to the technical solutions of the present invention shall fall within the protection scope determined by the claims of the present invention.

Claims

1. A method for evaluating the cognitive state differences of civil aviation flight crews based on the LSTM-MHSA algorithm and a hierarchical coupling framework, characterized in that The method comprises: S1: The pilot's EEG and psychological cognitive data are collected before and after the flight mission, and the scalarized EEG features and psychological cognitive features are reconstructed after preprocessing to form a unified multivariate time series sample set of individual cognitive ability features; S2: Based on the unified multivariate time series sample set of individual cognitive ability characteristics, a deep learning model LSTM-MHSA that integrates LSTM and multi-head self-attention mechanism is constructed to extract the temporal evolution characteristics of pilots' individual cognitive ability and the fusion characteristics of the cognitive interaction between the pilot and the co-pilot; S3: Based on the temporal evolution characteristics of the pilot's personal cognitive ability and the cognitive interaction fusion characteristics of the captain and co-pilot, the LSTM-MHSA model is used to evaluate the pilot's overall ability score in different psychological cognitive dimensions and the psychological cognitive coordination mode of the captain and co-pilot during the flight mission, so as to achieve differential evaluation of the cognitive status of civil aviation crews.

2. The method according to claim 1, wherein In S1, the pilot's EEG and psychological cognitive data are collected before and after the flight mission, and the scalarized EEG features and psychological cognitive features are reconstructed after preprocessing to form a unified individual cognitive ability feature multivariate time series sample set including: S1.1: Before and after the flight mission, an eight-lead EEG instrument is used to collect the pilot's cerebral cortical electrical signals according to international EEG standards. The key electrode positions are selected and conductive paste is used to ensure stable contact. The reference electrode is positioned on the earlobe, and the IND lead can be adjusted; S1.2: For EEG data, 1-60Hz third-order Butterworth bandpass filter and 50Hz notch filter were used for denoising. The data were centered and whitened, and the ICA technique was used to decompose the signal based on negative entropy to remove electrooculographic artifacts and channel interference. Finally, the wavelet threshold denoising method was used to process the signal after ICA separation. S1.3: For the test results of the psychological cognitive module, the data that exceeds the range of 3 times the standard deviation are eliminated, the mean interpolation method is used to deal with missing values, and the original data are standardized. The min-max standardization method is used to linearly transform the data to the interval [0, 1]. The conversion formula is based on the maximum and minimum values of the data, and finally normalization is performed; S1.4: Apply variational mode decomposition to extract energy entropy, information entropy, sample entropy and power spectrum density as EEG features, calculate the energy entropy of different intrinsic mode functions, use information entropy to evaluate the value of information in EEG signals, use sample entropy to measure the complexity of time series, and estimate the power spectrum of the signal through the periodogram method; S1.5: Select the scores of perceptual speed, memory span, N-back task accuracy, attention-stroop effect, spatial psychological cognitive error, calculation ability and psychomotor ability in the scale test as the core indicators for evaluating the six psychological cognitive dimensions of memory, attention, intelligence, spatial perception ability, reaction speed and professional ability; S1.6: The EEG and psychological cognitive features obtained from multiple sampling moments before and after the flight are scalarized to form a unified multivariate time series sample of individual cognitive ability characteristics.

3. The method according to claim 2, wherein In S1.4, the energy entropy, information entropy, sample entropy and power spectrum density are extracted by applying variational mode decomposition as EEG features, including: Using the VMD algorithm, by solving the variational problem, the original EEG signal f(t) is decomposed into K where, ω k is the center frequency of the k-th IMF, δ(t) is the Dirac function, u k represents the k-th intrinsic mode function (IMF), u k (t) represents the instantaneous amplitude of the k-th IMF at time point t, which is a function of time t, ω k (t) represents the instantaneous frequency of the k-th IMF, which is a function of time t and describes the local frequency characteristics of the IMF at time t. j represents the imaginary unit, that is used to describe the analytic representation of a signal in the complex domain, where u k (t) is the IMF time series to be solved, ω k (t) is the corresponding instantaneous frequency, and j is used in the exponential term to construct the analytic signal; Calculate the energy entropy, information entropy, sample entropy, and power spectral density based on each IMF.

4. The method according to claim 2, wherein In S1.6, the EEG and psychological cognitive characteristics obtained at multiple sampling times before and after flight are scalarized to form a unified individual cognitive ability characteristic multivariate time series sample, including: Let the observed value of the cognitive ability of pilot \(i\) at the \(t\)-th sampling moment be where \(D = 10\) is the dimension of the cognitive ability index that fuses EEG and psychological tests, and represents the EEG features of pilot \(i\) at the \(t\)-th sampling moment, including four key features: energy entropy, information entropy, sample entropy, and power spectral density. By standardizing the four key features, they are converted into scalar values in the range of \([0,1]\); It represents the psychological cognitive test results of pilot i at the t-th sampling moment, including the test scores of six dimensions: memory, attention, intelligence, spatial perception, reaction ability, and professional ability, and is normalized to a scalar value within the range of [0, 1]; The scalarized EEG features are concatenated with the normalized scores of psychological cognitive tests to obtain the complete cognitive ability observation value of pilot i at the t-th sampling moment: The complete cognitive ability sequence of pilot i is used as the input of the LSTM-MHSA model, where T is the number of sampling time steps.

5. The method according to claim 4, wherein In S2, according to the unified individual cognitive ability characteristic multivariate time series sample set, a deep learning model LSTM-MHSA that combines LSTM and multi-head self-attention mechanism is constructed to extract the time series evolution characteristics of the pilot's personal cognitive ability and the cognitive interaction fusion characteristics of the captain and first officer, including: S2.1: Design a deep learning model LSTM-MHSA that combines LSTM and MHSA. The model uses a multi-layer bidirectional LSTM as the backbone and introduces MHSA to model the internal relationships within the sequence. S2.2: Take the complete psychological cognitive ability sequence of pilot i as the input of the model, and use the LSTM-MHSA model to hierarchically encode the time-series data of the psychological cognitive ability of a single pilot to obtain cognitive feature representations at different scales and different abstraction levels; S2.3: Use the MHSA mechanism to model the psychological cognitive interaction relationship between the captain and first officer within the flight crew. S2.4: Train the LSTM-MHSA model in an end-to-end manner to achieve the adaptive extraction of individual-crew level coupling features from the original multivariate psychological cognitive ability data. Use the cross-entropy loss function to measure the deviation between the features extracted by the model and the actual psychological cognitive state of the pilot. Adopt the Adam adaptive optimization algorithm to iteratively update the model parameters and minimize the loss function. Use methods such as Dropout and L2 regularization to prevent the model from overfitting.

6. The method according to claim 5, characterized in that, In S2.1, design a deep learning model LSTM-MHSA that combines LSTM and MHSA. The model uses a multi-layer bidirectional LSTM as the backbone and introduces MHSA to model the internal relationships within the sequence, including: The overall structure of the LSTM-MHSA model is expressed as: H (0) = Embedding(X) H (l) = BiLSTM (l) (H (l-1) ), l = 1, 2, …, L U = MHSA(H (L) ) o = Softmax(FC(U)) Among them, the Embedding layer maintains a weight matrix, where each row corresponds to the vector representation of a discrete feature. During the training process, the vectors will be continuously updated and optimized together with the model parameters, and finally learn the semantic information of the input features. X = {x1, x2, …, x T} is the time-series data of the pilot's original psychological cognitive ability, is the input embedding representation, L is the number of LSTM layers, is the hidden state sequence of the l-th layer of LSTM, is the output sequence of the MHSA, is the predicted probability distribution of the pilot's psychological cognitive state; Using a multi-layer bidirectional LSTM as the backbone includes: Among them, and represent the forward and backward LSTM units respectively, is and the hidden states at time step t, is the complete hidden state of the bidirectional LSTM at time t. By processing the sequence in both forward and backward directions, the bidirectional LSTM can comprehensively utilize the information from multiple tests; Introducing MHSA to model the internal relationships within the sequence includes: U = [head1; head2; L; head N W O Among them, and are learnable linear transformation matrices, is the output of the n-th attention head, N is the number of attention heads, and d k = 2d h / N is the hidden layer dimension of each head, Attention(·) The function calculates normalized dot-product attention: The multi-head self-attention mechanism learns the relationships between sequence elements in different subspaces by parallelly calculating multiple attention functions, enhancing the model's feature representation ability. Finally, the outputs of all heads are concatenated and linearly transformed to obtain a high-level feature representation U that fuses different-scale dependency relationships.

7. The method according to claim 6, characterized in that, In the step S2.2, the complete psychological cognitive ability sequence of pilot i is used as the input of the model. The LSTM-MHSA model is used to perform hierarchical encoding on the time-series data of the psychological cognitive ability of a single pilot, and obtain cognitive feature representations at different scales and different abstraction levels, including: Encode the sequence of psychological cognitive abilities using a bidirectional LSTM network, and the forward LSTM gradually updates the hidden state Among them, are the parameters of the forward LSTM; Reverse LSTM updates the hidden state in reverse order Among them, are the parameters of the reverse LSTM; Concatenate the forward and backward hidden states to obtain the bidirectional LSTM encoding at step t The encoded sequence of the bidirectional LSTM is fed into the stacked multi-layer LSTM to achieve hierarchical abstraction of psychological cognitive features. The update formula for the hidden state of the $l$-th layer LSTM is as follows: Among them, represents the output of the bidirectional LSTM, and θ (l) is the parameter of the l-th layer of LSTM; After stacking L layers of LSTM, the multi-scale psychological and cognitive features of pilot i at time t are obtained Adaptive aggregation of the outputs of multiple layers of LSTM through a gating mechanism to obtain the final psychological cognitive ability characteristics of pilot i at time t Among them, W g and b g are the parameters of the gating layer, σ(·) is the sigmoid activation function, and e represents the Hadamard product; Extract multi-scale time series features that depict the evolution pattern of pilots' individual psychological cognitive abilities through bidirectional LSTM, multi-layer LSTM, and gated aggregation 8. The method according to claim 7, wherein In S2.3, using the MHSA mechanism to model the psychological cognitive interaction relationship between the captain and first officer within the flight crew includes: Let the multivariate psychological cognitive ability characteristics of the pilot and co-pilot within the \(t\)-th time window be and respectively. Then the psychological cognitive ability of the flight crew at time \(t\) can be expressed as where \(d\) is the dimension of individual psychological cognitive characteristics. Dividing a complete flight mission into \(T\) time windows, we obtain the psychological cognitive ability sequence \(\{s_1, s_2, \cdots, s\) T \}\); The MHSA maps the crew's psychological cognitive ability sequence S = [s1, s2, …, s T · to the query matrix Q, the key matrix K, and the value matrix V through a linear transformation:​ Q = SW Q , K = SW K , V = SW V Among them, is a learnable linear transformation matrix, and d h is the hidden layer dimension of the self-attention mechanism; Calculate the attention weight matrix A between different time steps: Among them, A ij represents the attention weight of the i-th time window to the j-th time window. The greater the weight, the higher the psychological cognitive interaction intensity of the crew members within the two time windows; Apply the attention weight matrix to the value matrix V to obtain the self-attention output matrix H: MHSA adopts a multi-head parallel calculation strategy. Suppose there are N attention heads, then the attention output matrix of the nth head is: H (n) = A (n) V (n) , n = 1, 2, …, N Among them, A (n) and V (n) are respectively the attention weight matrix and the value matrix independently learned by the nth head; Concatenate the outputs of all heads and obtain the final output matrix U of MHSA through linear transformation: U = [H (1) ; H (2) ; L; H (N) W O Among them, is a linear transformation matrix, and each row vector u of U t represents the crew's psychological cognitive characteristics that fuse the interaction information of the pilot and co-pilot; By comprehensively extracting the individual psychological cognitive ability characteristics and the crew psychological cognitive interaction characteristics, a hierarchical psychological cognitive characteristic sequence {u1, u2, …, u T} that takes into account both individual psychological cognition and crew interaction is obtained, which describes the changes in the psychological cognitive states of the pilot and co-pilot during the flight mission.

9. The method according to claim 1, characterized in that, In S3, according to the time series evolution characteristics of the pilot's personal cognitive ability and the cognitive interaction fusion characteristics of the captain and first officer, use the LSTM-MHSA model to evaluate the overall ability scores of the pilot in different psychological cognitive dimensions and the psychological cognitive collaboration mode between the captain and first officer during the flight mission, and realize the differential evaluation of the cognitive states of civil aviation flight crews, including: S3.1: Load the trained LSTM-MHSA model, configure the running environment of the model, and convert the time-series data of individual pilots' psychological and cognitive abilities into the input format required by the model. Among them, the 3D tensor is the number of samples × multiple time steps × feature dimension. For the evaluation at the crew level, align the psychological and cognitive ability data of the pilot and co-pilot in time and splice them into a whole for input; S3.2: Obtain the output of the LSTM-MHSA model, including the psychological and cognitive ability feature sequence at the individual level and the psychological and cognitive interaction weight matrix at the crew level; perform average pooling on the feature sequence at the individual level to obtain the overall ability scores of the pilot in different psychological and cognitive dimensions, and analyze the psychological and cognitive interaction weight matrix at the crew level to identify the psychological and cognitive cooperation patterns between the pilot and co-pilot during the flight mission; S3.3: Regularly fine-tune and update the model using newly collected pilot data; Among them, the specific content of S3.1 includes: First, load the pre-trained LSTM-MHSA model parameters θ * , and configure the software and hardware environment required for the model to run; then, convert the time-series data of the cognitive ability of individual pilots into the tensor format required by the model; specifically, for the evaluation of a single pilot, construct a three-dimensional tensor where N = 1 is the number of samples, T is the time step, and D is the dimension of cognitive features. For the evaluation at the crew level, align the cognitive ability sequences of the pilot and co-pilot in the time dimension and splice them in the sample dimension to form a tensor for crew input with a shape of ; The specific content of S3.2 includes: by parsing the output of the forward inference process of LSTM-MHSA, obtaining the cognitive competence evaluation results at two levels; at the individual level, performing average pooling on the features of each dimension in the LSTM hidden state sequence H (i) to obtain the comprehensive scoring vector of the pilot in different cognitive ability dimensions Among them, represents the j - th dimensional eigenvalue of the LSTM hidden state sequence of the i - th pilot at time step t, which is an element in the hidden state matrix H output by the LSTM model; (i) d h represents the dimension of the LSTM hidden state, that is, the number of neurons in the hidden layer, which determines the dimension of the final scoring vector s (i) s (i) comprehensively characterizes the cognitive performance of the pilot during the entire mission process, reflecting the level of their abilities such as attention allocation, information processing, decision - making judgment, etc.; meanwhile, perform clustering analysis on the MHSA output matrix U to identify the cognitive interaction patterns between the captain and the first officer at critical flight nodes.

10. A civil aviation crew cognitive state difference evaluation system based on the LSTM-MHSA algorithm and a hierarchical coupling framework, the system being used to implement the method according to any one of claims 1-9, characterized in that, The system includes: a reconstruction module, an extraction module, and an evaluation module; The reconstruction module is used to collect the electroencephalogram and psychological and cognitive data of the pilot before and after the flight mission respectively, and after preprocessing, reconstruct the scalarized electroencephalogram features and psychological and cognitive features to form a unified multi-time series sample set of individual cognitive ability features; The extraction module is used to construct a deep learning model LSTM-MHSA that integrates LSTM and multi-head self-attention mechanism according to the unified multi-time series sample set of individual cognitive ability features, and extract the time-series evolution features of the pilot's personal cognitive ability and the cognitive interaction fusion features of the pilot and co-pilot; The evaluation module is used to use the LSTM-MHSA model according to the time-series evolution features of the pilot's personal cognitive ability and the cognitive interaction fusion features of the pilot and co-pilot, evaluate the overall ability scores of the pilot in different psychological and cognitive dimensions and the psychological and cognitive cooperation patterns between the pilot and co-pilot during the flight mission, and realize the differential evaluation of the cognitive states of civil aviation crews.

Images