Pilot Operation Status Monitoring Method Based on Multi-Layer Fusion Recognition Model

Abstract

This invention provides a pilot operational status monitoring method based on a multi-layer fusion recognition model, comprising: S1, collecting peripheral physiological feature data, wrist acceleration time-series data, and wrist acceleration feature set data based on different sensors; S2, constructing an SVM recognition model based on the RBF kernel function; S3, establishing the SVM recognition model according to optimal parameters; S4, constructing an operational load recognition model based on the fusion of CNN-LSTM and intermediate layers; S5, training the CNN-LSTM operational load recognition model; S6, constructing a dynamic weighted fusion mode for the decision layer; and S7, using the trained recognition model for status recognition. This invention employs KPCA dimensionality reduction technology, and by inputting the multi-layer fusion model, it can fully perceive the pilot's status information in different dimensions, exhibiting good performance in operational status monitoring. It can be integrated into next-generation aircraft cockpits to better improve human-machine collaboration.

Description

Technical Field

[0001] This invention relates to the field of aviation document information extraction, specifically to a method for monitoring pilot operational status based on a multi-layer fusion recognition model. Background Technology

[0002] With the development of aviation technology, the flight environment and missions of aviation equipment are becoming increasingly complex. Pilots' primary tasks have shifted to monitoring situational information and making decisions on key tactical missions. To improve the level of collaboration between humans and equipment, human-machine interaction systems need corresponding improvements. One key to improving the human-machine interaction system of next-generation military aircraft is enhancing its intelligent perception of pilot operational status. Personnel status monitoring technology has wide applications in the field of intelligent aviation, primarily to improve pilot work efficiency and safety, prevent human error, and achieve optimal human-machine collaboration.

[0003] Traditionally, pilot condition assessments have primarily relied on expert interviews and questionnaires, such as the NASA-TLX Task Load Scale and the Fatigue Symptom Questionnaire. However, these indirect analyses have several drawbacks: questionnaire assessments are subjective, and their results are significantly influenced by individual subjective factors; questionnaires require interruptions to the subjects' work, which is impossible in actual flight missions; and subjective evaluations can only be conducted at discrete points in time, failing to provide continuous, mission-related monitoring information on changes in pilot condition.

[0004] To address these issues, researchers have proposed more objective and time-continuous methods to identify a person's state through single physiological signals, such as electrocardiograms, ocular measurements, electroencephalograms, respiration measurements, and electrodermal activity measurements. However, the identification of single physiological signals suffers from poor stability, reduced data information, low reliability, and low discriminative ability. Furthermore, some studies have explored combinations of different signals, but consensus remains lacking in many areas, such as which indicators should be used as input and what classification models should be employed. With the further development of ergonomic sensor technology, chip technology, and algorithms such as machine learning and deep learning, physiological computing has emerged as a viable approach. By collecting multimodal data and constructing classification models using machine learning or deep learning techniques to characterize the mapping relationship between physiological data and human state, a real-time, highly accurate, and robust human state monitoring capability can be established.

[0005] Besides using machine learning techniques to build classification models and input fusion feature sets, intelligent optimization of key model parameters is also a way to improve model performance. These optimization methods have been frequently used in previous studies. However, existing methods cannot fully guarantee the accuracy of monitoring results; therefore, there is an urgent need to research a high-performance method for monitoring pilot operational status. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention proposes a pilot operational status monitoring method based on a multi-layer fusion recognition model. This method utilizes KPCA technology to reduce the dimensionality of peripheral physiological feature sets collected and calculated by portable wearable devices, and iteratively searches for the optimal parameters of the SVM recognition model. Then, based on these optimal parameters, an SVM mental workload recognition model is established. A workload recognition model is built using a neural network structure based on the fusion of CNN-LSTM and intermediate layers. Furthermore, a multi-model dynamic weighted decision fusion strategy model is constructed. The fusion model is then used for status recognition, enabling intelligent monitoring of the pilot's operational status and ensuring excellent operational status monitoring performance.

[0007] Specifically, the present invention provides a method for monitoring pilot operational status based on a multi-layer fusion recognition model, which includes the following steps:

[0008] S1. Collect peripheral physiological characteristic set data, wrist acceleration time series data and wrist acceleration characteristic set data of pilots under different working conditions, and use KPCA technology to perform dimensionality reduction processing on the peripheral physiological characteristic set data.

[0009] S2. Construct an SVM mental workload recognition model based on the RBF kernel function, which includes the following sub-steps:

[0010] S21. Use the dimensionality-reduced sample data obtained in step S1 as input data, and set the input data as the training dataset:

[0011]

[0012] Where x i For a single sample, y i The state type set for the sample; For the sample space, R n This indicates that the sample space belongs to n-dimensional Euclidean space. Represents a state type vector;

[0013] S22. Construct an SVM mental workload recognition model based on the RBF kernel function:

[0014]

[0015]

[0016] 0≤α i ≤C, i=1,2,…,N

[0017] Where K(x) i ,x j ) represents the kernel function, α is the solution to the optimization problem, and C>0 is called the penalty parameter. The larger the value of C, the greater the penalty for misclassification, and the smaller the value of C, the smaller the penalty for misclassification.

[0018] Among them, the RBF kernel function K(x) i ,x j Further expressed as:

[0019]

[0020] In the formula, σ is the parameter of the Gaussian kernel function;

[0021] S23. Use the Northern Eagle Optimization Algorithm to iterate the parameters of the SVM recognition model to obtain the optimal parameters of the SVM recognition model.

[0022] S3. Based on the obtained optimal parameters, set up the SVM mental workload recognition model, and train the SVM using the training dataset to obtain the trained SVM recognition model.

[0023] S4. Construct an operational load recognition model based on the fusion of CNN-LSTM and intermediate layers. The operational load recognition model consists of two fused neural networks. The wrist acceleration time series data is input into the first neural network, and the wrist acceleration feature set data is input into the second neural network. The connection structure of the first neural network is: S1, C1, R1, M1, C2, R2, M2, FL1, L1, L2, FL2; the connection structure of the second neural network is: S2, C3, R3, M3, C4, R4, M3, FL3. Then, the two network structures are spliced ​​and fused. The fused connection structure is: CN1, FC1, FC2, SF1, P1. Wherein, the letters represent the module type, and the numbers represent the module number; S represents the input layer, C represents the convolutional layer, R represents the ReLU layer, M represents the max pooling layer, FL represents the Flatten layer, L represents the LSTM layer, CN represents the concatenated layer, FC represents the fully connected layer, SF represents the Softmax layer, and P represents the classification output layer.

[0024] S5. Use the collected data to train an operation load recognition model based on the fusion of CNN-LSTM and intermediate layers to obtain the trained CNN-LSTM operation load recognition model.

[0025] S6. Construct a dynamic weighted fusion model for the decision layer. Use the Sigmoid-Fitting method to obtain the output vector of the SVM mental load recognition model. Each sub-vector represents the probability of predicting that the peripheral physiological data belongs to each label. Extract the vector output by the Softmax layer of the CNN-LSTM operational load recognition model. Each sub-vector represents the probability of predicting that the wrist acceleration data belongs to each label. Use a linear combination method to construct a dynamic weighted decision fusion strategy model. The dynamic weighted decision fusion strategy model includes the weight of pilot operation characteristics, the weight of the classification performance coefficient of the sub-classifier, and the weight of physiological signal quality. Calculate the weights using the weight calculation formula. Based on the weight calculation results, perform weighted calculation on the vector output by the classifier to obtain the final probability calculation result vector.

[0026] The weight calculation formula is as follows:

[0027] P1 = A1 / (A1+A2);

[0028] P2 = A2 / (A1+A2);

[0029] Q1 = S1 / (S1+1);

[0030] Q2 = 1 / (S1+1);

[0031] W1 = (Z1 + P1 + Q1) / 3;

[0032] W2 = (Z2 + P2 + Q2) / 3;

[0033] In this equation, the SVM mental workload recognition model is defined as Model 1, and the CNN-LSTM operational workload recognition model is defined as Model 2; where W1 represents the weight of Model 1, and W2 represents the weight of Model 2; Z1 and Z2 represent the proportion of mental workload and operational workload in pilot operations, respectively; P1 and P2 represent the performance coefficients of Model 1 and Model 2, respectively; A1 represents the test set accuracy of Model 1, and A2 represents the test set accuracy of Model 2; Q1 and Q2 represent the physiological signal quality weights of Model 1 and Model 2, respectively; and S1 is the signal quality evaluation result of the PPG sensor.

[0034] S7. Perform state recognition using the trained recognition model, specifically including the following sub-steps:

[0035] S71. Real-time acquisition of peripheral physiological feature data and wrist acceleration data. For the peripheral physiological feature data, construct a feature set and perform dimensionality reduction using the method in step S1. Input the dimensionality-reduced data into the SVM mental load recognition model trained in step S3 to obtain the output result vector.

[0036] S72. For wrist acceleration data, input the wrist acceleration time series data and wrist acceleration feature set data into the CNN-LSTM operational load recognition model trained in step S5 to obtain the output result vector.

[0037] S73. The output vector is processed using the decision-level dynamic weighted fusion mode in step S6, and the category with the highest probability is selected as the final job status classification result.

[0038] Preferably, step S23 specifically includes the following sub-steps:

[0039] S231. Select the optimal kernel function K(x,z) and the optimal parameter C to obtain the optimal solution:

[0040] Choose α * A positive component yields:

[0041]

[0042] S232. Constructing the decision function:

[0043]

[0044] In the formula, x is the new sample input, and f(x) is the state classification result obtained by applying the SVM recognition model;

[0045] S233. Use the Northern Eagle optimization algorithm to optimize parameters;

[0046] S234. Normalize the training feature data and labels, as well as the validation data and labels.

[0047] S235. Initialize the population:

[0048] The population size is randomly generated, and each population contains two decision variables: the Gaussian kernel function parameter and the penalty factor.

[0049] S236. Optimize the parameters:

[0050] The fitness of each individual in the initial population is calculated using the objective function:

[0051]

[0052] In the formula, Q is the number of validation samples, and y q and These are the true values ​​and predicted values ​​of the validation samples, respectively;

[0053] After determining the initial optimal solution, multiple iterations are performed. In the first iteration, the current optimal solution is initialized to the best individual in the population. For each subsequent iteration, if the current optimal solution is defeated by any individual, the optimal solution is updated accordingly. After multiple iterations, the optimal parameters are obtained.

[0054] Preferably, in step S3, the optimal kernel parameter σ is 6.7775 and the optimal penalty parameter C is 6.6049.

[0055] Preferably, in step S235, the upper and lower limits of the two decision variables are restricted, wherein the value range of the Gaussian kernel function is [0.01, 200], and the value range of the penalty factor is [0.001, 30]. During initialization, the decision variables of each individual in the population are randomly assigned values ​​according to the upper and lower limit ranges.

[0056] Preferably, the neural network parameters in step S4 are set as follows: the input dimension of layer S1 is 400, and the input dimension of layer S2 is 30; the kernel size of layers C1 and C3 is 1×7, the number of kernels is 4, the stride is 1×1, and the edges are padded with 0; the kernel size of layers C2 and C4 is 1×3, the number of kernels is 8, the stride is 1×1, and the edges are padded with 0; the kernel size of layers M1 and M2 is 1×5, the stride is 3×3, and the edges are padded with 0; the kernel size of layers M3 and M4 is 1×3, the stride is 2×2, and the edges are padded with 0; layer L1 has 20 hidden units; layer L2 has 10 hidden units; layer FC1 has 128 neurons, and layer FC2 has 3 neurons; the maximum number of training epochs is set to 50, the initial learning rate is set to 0.001, the validation frequency is set to 20, and the optimizer is Adam optimizer.

[0057] Preferably, in the dynamic weighted decision fusion strategy model process in step S6, Z1 is assigned a weight of 0.8 and Z2 is assigned a weight of 0.2.

[0058] Preferably, in step S1, a portable physiological wristband is used to collect peripheral physiological signals and wrist acceleration data.

[0059] Preferably, the peripheral physiological signals in step S1 are physiological signals related to the peripheral nervous system, including electrocardiogram signals, skin conductance signals, skin temperature signals, respiratory signals, electrooculogram signals, and electromyogram signals.

[0060] Preferably, the portable physiological wristband collects peripheral physiological signals based on a skin conductance sensor and a photoplethysmography (PPG) sensor. A window size and step size are selected for signal processing, and a total of 89 features are extracted to construct a feature set, consisting of a GSR feature set containing 46 features and a PPG feature set containing 43 features. The portable physiological wristband also collects X, Y, and Z-axis acceleration data based on an accelerometer, and extracts a 400-length time-series wrist acceleration data set and a wrist acceleration feature set containing 30 features.

[0061] Preferably, the specific steps for dimensionality reduction of the feature set sample data using KPCA technology in step S1 are as follows:

[0062] Let the sample matrix be:

[0063] X = [x1, x2, ..., x N ]

[0064] Here, a single sample is a K-dimensional column vector, and X is the K×N-dimensional original sample space; through a nonlinear mapping φ, the column vector of a single sample is mapped to a high-dimensional space with dimension D;

[0065] φ(x):R K →R D D>>K

[0066] After mapping all samples to a high-dimensional space, a new matrix φ(X) is obtained, with dimensions D×N.

[0067] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0068] (1) This invention provides a pilot operation status monitoring method based on a multi-layer fusion recognition model. The monitoring method collects peripheral data and wrist acceleration data through a physiological wristband. Compared with physiological data collection technologies such as EEG and eye movement, the solution of collecting peripheral physiological signals through a portable physiological wristband is convenient and fast, does not require additional equipment, and is more suitable for long-term pilot status monitoring.

[0069] (2) The method of the present invention combines the SVM mental workload recognition model and the CNN-LSTM operational workload recognition model to construct a dynamic weighted fusion mode of the decision layer. The Sigmoid-Fitting method is used to obtain the result vector output by the SVM mental workload recognition model. Each of its sub-vectors represents the probability of predicting that the peripheral physiological data belongs to each label. The vector output by the Softmax layer of the CNN-LSTM operational workload recognition model is extracted. Each of its sub-vectors represents the probability of predicting that the wrist acceleration data belongs to each label. The vector output by the classifier is weighted using a linear combination method to obtain the final probability calculation result vector. The classification result obtained has a very high accuracy rate and is more suitable for monitoring the operational status of pilots. It also fully considers the influencing factors such as the operational characteristics of pilots and the quality of the acquired signals.

[0070] (3) When inputting data, the present invention uses KPCA dimensionality reduction technology to reduce the dimensionality of peripheral physiological signal feature set with a large number of features, removes useless noise information, overcomes the influence of high dimensionality, and improves the efficiency and performance of machine learning.

[0071] (4) This invention uses an optimization algorithm to optimize the key parameters of the SVM recognition model, obtains the optimized parameters, and reconstructs the SVM recognition model, avoiding the impact on model performance caused by selecting parameters based on experience, and ensuring the performance of the optimized SVM recognition model.

[0072] (5) The present invention utilizes the intermediate layer fused CNN-LSTM recognition model to fully extract and utilize the original information and statistical information in the wrist acceleration data, thus ensuring the performance of the CNN-LSTM recognition model.

[0073] (6) The method proposed in this invention can output accurate operational status classification results based on peripheral physiological characteristic data and wrist acceleration data, and thus be applied to operational status monitoring. It has good performance in monitoring pilot operational status and can be integrated into the cockpit of a new generation of aircraft, providing support for the design of a more intelligent human-machine interface, in order to better improve the level of human-machine collaboration. Attached Figure Description

[0074] Figure 1 This is a schematic diagram of the overall process of the present invention;

[0075] Figure 2 This is a schematic diagram of the fusion recognition model of the present invention;

[0076] Figure 3 This is a schematic diagram of the confusion matrix results in an embodiment of the present invention;

[0077] Figure 4 This is a schematic diagram of the confusion matrix result of the SVM scheme in an embodiment of the present invention. Detailed Implementation

[0078] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings.

[0079] This invention provides a method for monitoring pilot operational status, such as... Figure 1 , Figure 2 and Figure 4 As shown, it includes the following steps:

[0080] Specifically, the present invention provides a method for monitoring pilot operational status based on a multi-layer fusion recognition model, which includes the following steps:

[0081] S1. Collect peripheral physiological characteristic set data, wrist acceleration time series data, and wrist acceleration characteristic set data of pilots under different working conditions, and use KPCA technology to perform dimensionality reduction processing on the sample data of peripheral physiological characteristic set.

[0082] In the specific work process, the data acquisition equipment commonly used for personnel status monitoring in step S1 includes EEG electrode caps, eye trackers, cameras, etc., which collect EEG data, eye movement data, facial video data, etc. Considering the operational conditions of pilots and the feasibility of integrating relevant equipment in the aircraft cockpit, this method uses a physiological acquisition wristband to collect peripheral physiological characteristic data and wrist acceleration data. The physiological acquisition wristband will not interfere with the pilot's work tasks during data collection and is more suitable for long-term pilot status monitoring.

[0083] Peripheral physiological signals refer to physiological signals related to the peripheral nervous system, including electrocardiogram (ECG), electrodermal (TEG), skin temperature, respiration, electrooculogram (EOG), and electromyography (EMG). These peripheral physiological signals change when a pilot's operational status changes. By extracting and calculating the corresponding features of these physiological signals, the changes can be described, thus inferring the user's operational status. Peripheral physiological features are used to characterize the degree of mental workload on the pilot. Wrist acceleration data is collected via an accelerometer, and the X, Y, and Z-axis acceleration data are processed to obtain total acceleration time-series data and acceleration feature set data. Wrist acceleration data is used to characterize the degree of hand operation load on the pilot. By combining mental and operational load, a more accurate description of the pilot's operational status can be obtained.

[0084] The physiological data acquisition wristband of this method collects peripheral physiological signals based on two types of sensors: a ground-skin response (GSR) sensor and a photoplethysmography (PPG) sensor. A signal processing window of 20 seconds and a step size of 2 seconds are selected. A total of 89 features are extracted and constructed to form a feature set, which can effectively characterize the information collected by the GSR and PPG sensors. Specifically, the 46 features in the GSR feature set and the 43 features in the PPG feature set are shown in Table 1. The physiological data acquisition wristband of this method collects acceleration data through an accelerometer with a sampling frequency of 20Hz. A signal processing window of 20 seconds and a step size of 2 seconds are selected, resulting in each acceleration data point including 400 values. Specifically, the wrist acceleration time-series data and the 30-feature set are shown in Table 2.

[0085] Table 1. Feature sets of GSR and PPG

[0086]

[0087]

[0088]

[0089] Table 2. Time-series data and feature set of wrist acceleration

[0090]

[0091] The sample data of the peripheral physiological feature set were then subjected to dimensionality reduction processing, as follows:

[0092] Kernel Principal Component Analysis (KPCA) is used to reduce the dimensionality of the original data. The original feature dataset is mapped to a high-dimensional space using a non-linear mapping method, and then PCA is performed in this high-dimensional space to achieve dimensionality reduction. The mathematical model of the kernel technique of KPCA is as follows:

[0093] Let the sample matrix be:

[0094] X = [x1, x2, ..., x N ]

[0095] Here, a single sample is a K-dimensional column vector, so X is the original sample space of K×N dimensions. The main method of the kernel trick is to map the column vector of a single sample to a high-dimensional space of dimension D through a nonlinear mapping φ.

[0096] φ(x):R K →R D D>>K

[0097] In KPCA, the expression for φ(x) is not explicitly given. After mapping all samples to the new space, a new matrix φ(X) is obtained, with dimensions D×N.

[0098] S2. Construct an SVM recognition model and use optimization algorithms to iteratively find the optimal parameters of the SVM recognition model.

[0099] This step specifically includes the following sub-steps:

[0100] S21. Construct a linear model using the Gaussian kernel function method and introduce a penalty factor C;

[0101] S22. The parameters of the penalty factor C and the Gaussian kernel function are optimized using the Northern Eagle optimization algorithm.

[0102] In this embodiment, the specific steps are as follows:

[0103] Constructing an SVM recognition model based on the RBF kernel function

[0104] Set the input data as the training dataset:

[0105]

[0106] Where x i For single sample feature data, y i The state type set for the sample.

[0107] (a) Construct and solve the optimization problem.

[0108]

[0109]

[0110] 0≤α i ≤C, i=1,2,…,N

[0111] Where K(x) i ,x j ) represents the kernel function, α is the solution to the optimization problem, and C>0 is called the penalty parameter, which is generally determined by the specific application problem. The larger the value of C, the greater the penalty for misclassification, and the smaller the value of C, the smaller the penalty for misclassification.

[0112] In this embodiment, the optimal kernel parameter σ is 6.7775 and the optimal penalty parameter C is 6.6049.

[0113] The RBF kernel function can be further expressed as:

[0114]

[0115] In the formula, σ is the kernel function parameter.

[0116] When an appropriate kernel function K(x,z) and an appropriate parameter C are selected, the optimal solution is obtained:

[0117] Choose α * A positive component yields:

[0118]

[0119] (b) Construct the decision function.

[0120]

[0121] In the formula, x is the new sample input, and f(x) is the state result obtained by applying the model.

[0122] (2) Use the NGO algorithm for parameter optimization.

[0123] (a) Normalize the training feature data and labels, as well as the validation data and labels.

[0124] (b) Initialize the population.

[0125] The population size is randomly generated, and each population contains two decision variables: the Gaussian kernel function parameter and the penalty factor. Upper and lower bounds are imposed on the two decision variables: the Gaussian kernel function has a range of [0.01, 200], and the penalty factor has a range of [0.001, 30]. During initialization, the decision variables of each individual in the population are randomly assigned values ​​according to the upper and lower bounds.

[0126] (c) Parameter optimization.

[0127] The fitness of each individual in the initial population is calculated using the objective function:

[0128]

[0129] In the formula: Q is the number of validation samples, y q and These represent the true value and the predicted value of the validation sample, respectively.

[0130] After determining the initial optimal solution, multiple iterations are performed. In the first iteration, the current optimal solution is initialized to the best individual in the population. For each subsequent iteration, if the current optimal solution is "defeated" by any individual, the optimal solution is updated accordingly.

[0131] After multiple iterations, the optimal support vector machine (SVM) model parameters were obtained. These optimal parameters were then substituted into the SVM model to optimize the SVM recognition model, resulting in a final pilot operational status monitoring method model based on KPCA and NGO-SVM. This achieves the goal of optimizing the SVM model parameters using the Northern Eagle method.

[0132] The method in this embodiment employs a group behavior optimization approach to derive the Northern Goshawk Optimization (NGO) algorithm. This algorithm simulates the hunting process of the Northern Goshawk for parameter optimization. Overall, this strategy consists of two main behaviors:

[0133] (1) Prey identification and attack.

[0134] (2) Chase and escape operations.

[0135] The specific mathematical model of the algorithm is shown below:

[0136] (1) Model initialization.

[0137] The initial population using the NGO method can be represented in matrix form:

[0138]

[0139] Where X is the NGO race matrix, X i Let x represent the i-th Northern Goshawk and its location. i,j Let N represent the j-th decision variable and its position for the i-th Northern Goshawk. Furthermore, N represents the number of Northern Goshawks, and m represents the number of decision variables.

[0140] In this method, the model fitness of each individual can be determined by the function value of the objective function, which can be represented as a value vector, as follows:

[0141]

[0142] In the formula: F is the objective function vector of the NGO population, F i Let be the objective function value of the i-th Northern Eagle.

[0143] (2) Prey identification and attack stage.

[0144] The NGO method performs a global search of the decision variable space in the first stage to determine the optimal region. This stage simulates the prey identification and attack behavior of the northern goshawk. The specific formula for this stage is as follows:

[0145] P i =X k,i=1,2,…,N,k=1,2,…,i-1,i+1,…,N

[0146]

[0147]

[0148] In the formula: P i Let i be the location of the i-th Northern Goshawk's "prey". Let be the objective function value for the i-th Northern Goshawk. k is any integer in the range [1, N]. Indicates the possible new location of the i-th northern goshawk. This indicates the position of its j-th decision variable. Let r be the objective function value of the i-th Northern Eagle after each iteration, where r is a random value in the range [0, 1] and I is a random integer between 1 and 2.

[0149] (3) The chase and escape phase.

[0150] This stage improves the local search capability of the decision variable space by simulating the hunting process of a northern goshawk. The specific formula for this stage is as follows:

[0151]

[0152]

[0153]

[0154] In the formula: Indicates the possible new location of the i-th northern goshawk. This indicates the position of its j-th decision variable. Let j be the position of the i-th Northern Eagle's decision variable after the second stage update. Let t be the objective function value of the i-th Northern Eagle after the second stage update, t be the current iteration number, and T be the maximum iteration number.

[0155] S3. Based on the obtained optimal parameters, set the SVM mental workload recognition model, and train the SVM using the training dataset to obtain the trained SVM recognition model.

[0156] S4. Construct an operational load recognition model based on the fusion of CNN-LSTM and intermediate layers. The operational load recognition model consists of two fused neural networks. Wrist acceleration time-series data is input into the first network, and wrist acceleration feature set data is input into the second network. The connection structure of the first network is: S1, C1, R1, M1, C2, R2, M2, FL1, L1, L2, FL2. The connection structure of the second network is: S2, C3, R3, M3, C4, R4, M3, FL3. Then, the two network structures are concatenated and fused, resulting in the connection structure: CN1, FC1, FC2, SF1, P1. Here, letters represent module types, and numbers represent module numbers. S represents the input layer, C represents the convolutional layer, R represents the ReLU layer, M represents the max pooling layer, FL represents the Flatten layer, L represents the LSTM layer, CN represents the concatenated layer, FC represents the fully connected layer, SF represents the Softmax layer, and P represents the classification output layer.

[0157] In this embodiment, the input dimension of layer S1 is 400, and the input dimension of layer S2 is 30; the convolutional kernels of layers C1 and C3 are 1×7, with 4 kernels and a stride of 1×1, padded with zeros at the edges; the convolutional kernels of layers C2 and C4 are 1×3, with 8 kernels and a stride of 1×1, padded with zeros at the edges; the convolutional kernels of layers M1 and M2 are 1×5, with a stride of 3×3, padded with zeros at the edges; the convolutional kernels of layers M3 and M4 are 1×3, with a stride of 2×2, padded with zeros at the edges; layer L1 has 20 hidden units; layer L2 has 10 hidden units; layer FC1 has 128 neurons, and layer FC2 has 3 neurons. The maximum number of training epochs is set to 50, the initial learning rate is set to 0.001, the validation frequency is set to 20, and the Adam optimizer is selected.

[0158] S5. Use the collected data to train an operation load recognition model based on the fusion of CNN-LSTM and intermediate layers to obtain the trained CNN-LSTM operation load recognition model.

[0159] S6. Construct a dynamic weighted fusion model for the decision layer. Use the Sigmoid-Fitting method to obtain the output vector of the SVM mental load recognition model. Each sub-vector represents the probability of predicting that the peripheral physiological data belongs to each label. Extract the vector output by the Softmax layer of the CNN-LSTM operational load recognition model. Each sub-vector represents the probability of predicting that the wrist acceleration data belongs to each label. Use a linear combination method to design a dynamic weighted decision fusion strategy model, which includes the weights of pilot operation characteristics, the classification performance coefficients of sub-classifiers, and the quality of physiological signals. Based on the weight calculation results, perform weighted calculation on the vector output by the classifier to obtain the final probability calculation result vector.

[0160] The weight calculation formula is as follows:

[0161] P1 = A1 / (A1+A2);

[0162] P2 = A2 / (A1+A2);

[0163] Q1 = S1 / (S1+1);

[0164] Q2 = 1 / (S1+1);

[0165] W1 = (Z1 + P1 + Q1) / 3;

[0166] W2 = (Z2 + P2 + Q2) / 3;

[0167] In this equation, the SVM mental workload recognition model is defined as Model 1, and the CNN-LSTM operational workload recognition model is defined as Model 2; where W1 represents the weight of Model 1, and W2 represents the weight of Model 2; Z1 and Z2 represent the proportion of mental workload and operational workload in pilot operations, respectively; P1 and P2 represent the performance coefficients of Model 1 and Model 2, respectively; A1 represents the test set accuracy of Model 1, and A2 represents the test set accuracy of Model 2; Q1 and Q2 represent the physiological signal quality weights of Model 1 and Model 2, respectively; and S1 is the signal quality evaluation result of the PPG sensor.

[0168] S7. Perform state recognition using the trained recognition model, specifically including the following sub-steps:

[0169] S71. Real-time acquisition of peripheral physiological feature data and wrist acceleration data. For the peripheral physiological feature data, construct a feature set and perform dimensionality reduction using the method in step S1. Input the dimensionality-reduced data into the SVM mental load recognition model trained in step S3 to obtain the output result vector.

[0170] S72. For wrist acceleration data, input the wrist acceleration time series data and wrist acceleration feature set data into the CNN-LSTM operational load recognition model trained in step S5 to obtain the output result vector.

[0171] S73. The output vector is processed using the decision-level dynamic weighted fusion mode in step S6, and the category with the highest probability is selected as the final job status classification result.

[0172] The results of this embodiment will be verified below:

[0173] This embodiment provides a pilot operation status monitoring method based on a multi-layer fusion recognition model, which can identify the pilot's operation status based on a fusion model of KPCA-NGO-SVM and CNN-LSTM.

[0174] In this embodiment, the number of features after KPCA dimensionality reduction is set to 10, reducing the large number of 89 features to 10 dimensions while preserving as much of the original feature information as possible. Secondly, the dataset is divided into a training set (80%), a validation set (10%), and a test set (10%), resulting in 650 training data entries, 81 test data entries, and 81 validation data entries. The NGO algorithm is used to optimize the Gaussian kernel function parameters and penalty factor C of the SVM, with a population size of 10 and 10 iterations. The prediction accuracy of the validation machine is used as the fitness function to select the optimal model.

[0175] The model was tested using the test set data to validate its performance. Ultimately, the accuracy on the test set was 0.9753. The confusion matrix for the test set is shown below. Figure 3 As shown, the model made only one incorrect prediction for both the "resting state" and "high load state" data, indicating that the model performs well.

[0176] The results of this method are compared with the performance of other technical solutions.

[0177] To further validate the performance of the proposed model, this study conducted performance tests on other schemes, including: using only support vector machines; principal component analysis (PCA) and support vector machines; kernel principal component analysis and support vector machines; and principal component analysis and multilayer perceptron (MLP). The results are shown in Table 3.

[0178] Table 3 Comparison of performance results of this method with other models.

[0179] Technical solution accuracy SVM 0.6790 PCA+SVM 0.8148 KPCA+SVM 0.9259 PCA+MLP 0.7037 The fusion recognition model of this solution 0.9753

[0180] Examining the confusion matrix results of the SVM scheme reveals that the scheme using only support vector machines as classifiers struggles to distinguish between "resting state" and "low workload" data, resulting in numerous errors. Comparing PCA SVM and PCAMLP, SVM demonstrates superior performance in the classification task of this study. Furthermore, comparing PCA SVM and KPCA SVM schemes shows that KPCA dimensionality reduction is more effective at extracting valuable information in this classification task. To further improve monitoring accuracy, this method employs the NGO algorithm to optimize SVM hyperparameters and incorporates the operational load identification results from wrist acceleration data, achieving a high accuracy level (97.53%).

[0181] Pilot operational status monitoring has a wide range of applications in the aviation equipment field. For example, by monitoring the status of pilots during simulated training, their work efficiency and performance can be evaluated, and personalized mission suggestions and training programs can be provided to pilots. By monitoring the operational status of pilots during human-machine interface (HMI) tests, the ergonomic level of equipment can be indirectly evaluated, problems in the functionality and mission execution of equipment can be verified, and suggestions can be made for subsequent optimization design. By monitoring the operational status of pilots in the cockpit, input can also be provided for intelligent interactive technologies such as active push-to-display and adaptive display, optimizing human-machine interface design and improving the operational efficiency and safety of pilots.

[0182] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made by those skilled in the art to the technical solutions of the present invention without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A pilot working state monitoring method based on a multi-layer fusion recognition model, characterized in that: It comprises the following steps: S1, collecting peripheral physiological feature set data, wrist acceleration time series data and wrist acceleration feature set data of pilots in different operating states, and using KPCA technology to reduce the dimension of the peripheral physiological feature set data; S2, constructing an SVM mental workload recognition model based on an RBF kernel function, specifically comprising the following sub-steps: S21, taking the dimension-reduced peripheral physiological feature set data obtained in step S1 as input data, and setting the input data as a training data set: ， ； wherein, is a single sample, is a state type set for the sample; is a sample space, denotes that the sample space belongs to an n-dimensional Euclidean space, denotes a state type vector; S22, constructing an SVM mental workload recognition model based on an RBF kernel function: ； ； ； wherein, denotes a kernel function, is a solution of the optimization problem, is called a penalty parameter, the larger the value, the greater the penalty for misclassification, the smaller the value, the smaller the penalty for misclassification; where the RBF kernel function is further expressed as: ； In the formula, is a Gaussian kernel function parameter; S23, using the Northern Hawk Optimization Algorithm to iterate the parameters of the SVM recognition model to obtain the optimal parameters of the SVM recognition model; S3, setting the SVM brain load recognition model based on the obtained optimal parameters, and The training data set is used to train the SVM, and a trained SVM brain load recognition model is obtained. S4, constructing an operating load recognition model based on CNN-LSTM and intermediate layer fusion, the operating load recognition model is composed of two neural network fusion paths, the wrist acceleration time series data is input into the first neural network, and the wrist acceleration feature set data is input into the second neural network, the connection structure of the first neural network is: S1, C1, R1, M1, C2, R2, M2, FL1, L1, L2, FL2; the connection structure of the second neural network is: S2, C3, R3, M3, C4, R4, M3, FL3, and then the two network structures are spliced and fused, and the connection structure after fusion is: CN1, FC1, FC2, SF1, P1; wherein the letters represent the module type, and the numbers represent the module number; S represents the input layer, C represents the convolution layer, R represents the ReLU layer, M represents the maximum pooling layer, FL represents the Flatten layer, L represents the LSTM layer, CN represents the series splicing layer, FC represents the full connection layer, SF represents the Softmax layer, and P represents the classification output layer; S5, training the operating load recognition model based on CNN-LSTM and intermediate layer fusion using the collected data to obtain the trained CNN-LSTM operating load recognition model; S6, constructing a decision layer dynamic weighted fusion mode to obtain the result vector output by the SVM mental workload recognition model, each sub-vector of which represents the probability that the predicted peripheral physiological data belongs to each label, extracting the vector output by the Softmax layer of the CNN-LSTM operating load recognition model, each sub-vector of which represents the probability that the predicted wrist acceleration data belongs to each label, using a linear combination method to construct a dynamic weighted decision fusion strategy model, the dynamic weighted decision fusion strategy model including pilot operating characteristic weights, sub-classifier classification performance coefficient weights and physiological signal quality weights, calculating the weights using a weight calculation formula, and calculating the vector output by the classifier based on the weight calculation result to obtain the final probability calculation result vector; The weight calculation formula is as follows: P1=A1 / (A1+A2); P2=A2 / (A1+A2); Q1=S1 / (S1+1); Q2=1 / (S1+1); W1=(Z1+P1+Q1) / 3; W2=(Z2+P2+Q2) / 3; Wherein, the definition SVM brain load recognition model is model 1, and the CNN-LSTM operation load recognition model is model 2; In the formula, W1 represents the weight of model 1, and W2 represents the weight of model 2; Z1 and Z2 respectively represent the proportion of brain load and operation load in the pilot operation; P1 and P2 respectively represent the performance coefficients of model 1 and model 2, A1 represents the test set accuracy of model 1, and A2 represents the test set accuracy of model 2; Q1 and Q2 respectively represent the physiological signal quality weight of model 1 and model 2, and S1 is the signal quality evaluation result of the PPG sensor; S7, using the trained recognition model to recognize the state, specifically including the following sub-steps: S71, real-time acquisition of peripheral physiological characteristic data and wrist acceleration data, for the peripheral physiological characteristic data, the peripheral physiological characteristics are reduced dimensionally by the method of step S1, and the reduced dimension data is input into the SVM brain load recognition model trained in step S3 to obtain the output result vector; S72, for the wrist acceleration data, the wrist acceleration time series data and the wrist acceleration feature set data are input into the CNN-LSTM operation load recognition model trained in step S5 respectively to obtain the output result vector; S73, using the decision layer dynamic weighted fusion mode in step S6 to process the output vector, and selecting the class with the maximum probability as the final operation state classification result.

2. The pilot work state monitoring method based on the multi-layer fusion recognition model according to claim 1, characterized in that: Step S23 specifically includes the following sub-steps: S231, select the optimal kernel function and the optimal parameters , get the optimal solution: ; Selecting one positive component, we obtain: ； S232, constructing a decision function: ； In the formula, is a new sample input, is a state classification result obtained by applying the SVM identification model to the discrimination output. S233, using the North Hawk optimization algorithm to optimize parameters; S234, normalizing the training feature data and labels and the verification data and labels; S235, initializing the population: Randomly generate the number of populations, and each population contains 2 decision variables, which are Gaussian kernel function parameters and penalty factors; S236, optimizing parameters: Calculate the fitness of each individual in the initial population, and the objective function is: ； In the formula, To verify the number of samples, and are the true value and the predicted value of the verification sample, respectively; After determining the initial optimal solution, multiple iterations are performed, and in the first iteration, the current optimal solution is initialized as the optimal individual in the population. For each subsequent iteration, if the current optimal solution is defeated by any individual, the optimal solution is updated accordingly. After multiple iterations, the optimal parameters are obtained.

3. The pilot work state monitoring method based on the multi-layer fusion recognition model according to claim 1, characterized in that: optimal kernel parameters obtained in step S3 takes the value 6.7775, optimal penalty parameter takes the value 6.6049.

4. The pilot work state monitoring method based on the multi-layer fusion recognition model according to claim 2, characterized in that: In step S235, the upper and lower limits of the two decision variables are limited, wherein the value range of the Gaussian kernel function is [0.01, 200], and the value range of the penalty factor is [0.001, 30]. When initializing, the decision variables of each individual in the population are randomly assigned according to the upper and lower limit ranges.

5. The method of claim 1, wherein the method comprises: The neural network parameter setting in step S4 is as follows: the input dimension number of S1 layer is 400, and the input dimension number of S2 layer is 30; the convolution kernel size of C1 and C3 layers is 1*7, the number is 4, the convolution step length is 1*1, and the edge is supplemented with 0; the convolution kernel size of C2 and C4 layers is 1*3, the number is 8, the convolution step length is 1*1, and the edge is supplemented with 0; the convolution kernel size of M1 and M2 layers is 1*5, the convolution step length is 3*3, and the edge is supplemented with 0; the convolution kernel size of M3 and M4 layers is 1*3, the convolution step length is 2*2, and the edge is supplemented with 0; 20 hidden units are set in L1 layer; 10 hidden units are set in L2 layer; The number of neurons of FC1 layer is 128, and the number of neurons of FC2 layer is 3; the maximum training round is set to 50, the initial learning rate is set to 0.001, the verification frequency is set to 20, and the optimizer is selected as the Adam optimizer.

6. The method of claim 1, wherein the method comprises: In the linear combination process of step S6, the weight given to Z1 is 0.8, and the weight given to Z2 is 0.

2.

7. The method of claim 1, wherein the method comprises: In step S1, a portable physiological bracelet is used to collect peripheral physiological signals and wrist acceleration data.

8. The pilot work state monitoring method based on the multi-layer fusion recognition model according to claim 4, characterized in that: The peripheral physiological signals in step S1 are physiological signals related to the peripheral nervous system, including electrocardiogram signals, electrodermal signals, skin temperature signals, respiratory signals, electrooculogram signals and electromyogram signals.

9. The method of claim 7, wherein the method comprises: The portable physiological bracelet collects peripheral physiological signals based on a galvanic skin response sensor and a photoplethysmogram sensor, selects a window length and a step length for signal processing, calculates and extracts a total of 89 features to construct a feature set, including a GSR feature set containing 46 features and a PPG feature set containing 43 features; X, Y and Z three-axis acceleration data are collected based on an acceleration sensor, and 400-length wrist acceleration time series data and a wrist acceleration feature set containing 30 features are calculated and extracted.

10. The method of claim 1, wherein the method comprises: The specific steps of using KPCA technology to perform dimension reduction processing on the sample data of the feature set in step S1 are as follows: Let the sample matrix be ； wherein the single sample is a K-dimensional column vector, is a K x N-dimensional original sample space; through a nonlinear mapping the column vector of the single sample is mapped to a high-dimensional space with a dimension of ; ； After mapping all samples into high dimensional space, a new matrix is obtained with dimensions: × N.

Images