Pilot cognitive load monitoring and early warning method based on electroencephalogram and eye movement data

Abstract

The invention discloses a pilot cognitive load monitoring and early warning method based on electroencephalogram and eye movement data, and the method comprises the steps: 1) designing a pilot different cognitive load state induction experiment paradigm in a simulated flight task scene, and obtaining a multi-modal physiological signal and task performance of a testee during an experiment; wherein the multi-mode physiological signals comprise electroencephalogram signals and eye movement signals; 2) preprocessing the multi-modal physiological signals and task performance, extracting corresponding feature indexes, and fusing and selecting the feature indexes to construct an optimal feature subset; and 3) constructing a cognitive state classification and discrimination model based on the optimal feature subset so as to carry out monitoring and early warning on the cognitive load of the pilot. According to the invention, the cognitive load level of the pilot can be monitored and evaluated, especially the sudden change point of the overload state of the cognitive load is pre-warned, and the flight safety is improved.

Description

Technical Field

[0001] The present invention mainly relates to the technical field of pilot cognitive monitoring, and in particular to a pilot cognitive load monitoring and early warning method based on electroencephalogram and eye movement data. Background Art

[0002] With the increasing development of aviation technology, the automation level of civil aircraft, fighter jets and armed helicopters is constantly improving. The cockpit has become a special operating environment with highly intensive information and complex and changeable tasks. Safe flight has received more and more attention, and higher and higher requirements have been placed on the cognitive ability of pilots. The cognitive load level of pilots is an important factor affecting flight safety. Cognitive load can be understood as the amount of brain cognitive activity per unit time, the occupancy rate of brain cognitive resources, the psychological pressure of people at work or the ability to process information. During the flight, when the pilot's cognitive load level is high, it takes more effort to maintain the correct execution of the task, which will increase the pilot's fatigue and reduce his perception and reaction ability. When the flight encounters an emergency or the task demand increases sharply, the pilot quickly enters a state of cognitive load overload, which will cause errors in the acquisition and analysis of environmental information, decision-making errors, and a significant reduction in task performance, and even threaten the flight safety of the aircraft. Therefore, monitoring and evaluating the pilot's cognitive load level, especially warning of the mutation point of cognitive load overload, is of great significance to the appropriate allocation of flight system tasks, the handling of emergencies, the improvement of task execution efficiency and flight safety.

[0003] At present, most of the research on cognitive status focuses on the evaluation of cognitive load level and the detection of cognitive collapse using data before and after the collapse point. There is little advance prediction and analysis of cognitive collapse points. Pilot cognitive collapse warning research faces the following challenges: First, how to induce cognitive collapse state. Different paradigms and stimulus materials induce different cognitive load effects, and different individuals react differently to the same paradigm. At present, research is still not clear about how to define and judge cognitive collapse state; second, how to determine warning signals to distinguish different cognitive load levels and identify the critical slowing state before cognitive collapse; third, the data set samples are insufficient. Experimental collection in simulated flight mission scenarios is more complicated, and the experimental paradigm changes with the mission scenario. It is difficult to carry out large-scale data collection experiments, resulting in insufficient samples in the data set for the evaluation of cognitive load status in specific tasks and specific populations. Summary of the invention

[0004] The technical problem to be solved by the present invention is: in view of the technical problems existing in the prior art, the present invention provides a pilot cognitive load monitoring and early warning method based on EEG and eye movement data, which can monitor and evaluate the pilot's cognitive load level, especially provide early warning of the state mutation point of cognitive load overload.

[0005] In order to solve the above technical problems, the technical solution proposed by the present invention is:

[0006] A pilot cognitive load monitoring and early warning method based on electroencephalogram and eye movement data comprises the following steps:

[0007] 1) Design an experimental paradigm to induce different cognitive load states of pilots in simulated flight mission scenarios, and obtain the multimodal physiological signals and task performance of the subjects during the experiment; the multimodal physiological signals include EEG signals and eye movement signals;

[0008] 2) Preprocess the multimodal physiological signals and task performance and extract the corresponding feature indicators, fuse and select the feature indicators to construct the optimal feature subset;

[0009] 3) Based on the optimal feature subset, a cognitive state classification and discrimination model is constructed to monitor and warn the pilot's cognitive load.

[0010] Preferably, in step 2), the specific process of preprocessing the EEG signal in the multimodal physiological signal is:

[0011] Perform high-pass filtering, low-pass filtering and notch filtering on the EEG signals to extract EEG signals in a specific frequency range;

[0012] Perform blind source analysis on EEG signals in a specific frequency range to remove eye artifacts;

[0013] Then the EEG signal is averaged over the whole brain for reference;

[0014] Finally, the sliding window is used to segment the EEG signal.

[0015] Preferably, in step 2), the characteristic indicators of the EEG signal include frequency domain characteristics, nonlinear dynamic characteristics and functional connection characteristics of each frequency band of the EEG signal;

[0016] The frequency domain characteristics of the EEG signal are described as power spectral density. The power spectral density of the EEG signal is calculated by the Welch power spectrum estimation method, and the mean power spectral density of the corresponding frequency in each frequency band is calculated according to the following formula:

[0017]

[0018] Among them, N band is the number of frequency values ​​in the frequency domain, band high and band low is the range of frequency domain, PSD f is the power spectral density value corresponding to frequency f.

[0019] Preferably, in step 2), the characteristic index of the EEG signal also includes the square coherence of the EEG signal amplitude. This characteristic index evaluates and distinguishes the cognitive load level by calculating the connectivity within and between brain regions. Specifically, the EEG signal is first subjected to a Laplace transform based on the second-order spatial derivative of the scalp electric field to solve the common source problem, and then the square coherence of the EEG signal amplitude after Laplace transform is calculated to measure the relationship between the two EEG signals in a specific frequency band, so as to reveal the connection between different brain regions and within brain regions.

[0020] Preferably, in step 2), after extracting the corresponding feature indicators, they are cascaded and fused into a multimodal feature vector, and the fused feature indicators are screened through a three-stage feature selection strategy to construct an optimal feature subset; specifically:

[0021] Assume that the initial multimodal fusion feature set is S0. First, use variance analysis to preliminarily select the initial multimodal fusion feature set S0, remove the features with small inter-class differences, and select the feature subset S1 with large inter-class differences.

[0022] Secondly, the correlation coefficients between all the features in feature subset S1 are calculated; when the correlation coefficient exceeds the threshold, the feature with a higher overall correlation with other variables in feature subset S1 is removed until the correlation coefficients between all the features are lower than the given threshold, and feature subset S2 is obtained to solve the problem of collinear redundancy between different channel information of the same modality and different modal information;

[0023] Finally, the sequential backward feature selection method is used to select features related to the classification target, and the features that contribute greatly to the classification task are selected in the feature subset S2: the objective function is calculated using the N-1 dimensional features in the N-dimensional feature set in turn, and this process is repeated N times to find the features deleted in the best performance model, remove the features from the N-dimensional feature set, and repeat this process with the remaining N-1 dimensional features until deleting the features does not produce better classification results, and the optimal feature subset S3 is obtained.

[0024] Preferably, in step 1), three groups of tasks with different difficulty levels are designed to induce three cognitive load levels of low, medium and high in the subjects, and a cognitive collapse state is induced in the high cognitive load stage.

[0025] Preferably, in step 3), in the process of monitoring and warning the pilot's cognitive load, the cognitive load level is first determined, and then the high cognitive load signal is classified into a cognitive breakdown warning signal and a normal state signal.

[0026] Preferably, the determination process of the cognitive breakdown warning signal is as follows:

[0027] The multivariate mutation test method is used to calculate the difference values ​​of all window data on the left and right sides of each data point in the high cognitive load stage in turn; if the maximum difference value is greater than the set threshold, and there are at least 20 data before the data point and at least 10 data after the point, then this position is determined to be a cognitive collapse point, and the labels of the 10 window data before the cognitive collapse point are set as cognitive collapse warning signals.

[0028] Preferably, in step 3), the cognitive state classification and discrimination model includes an input layer, a domain adaptation layer and a cognitive state classification layer, and a new objective function is designed, specifically:

[0029] J=λ1J Domain +λ2J Calibration +λ||ω|| 2

[0030] J Domain =J HingeLoss +α(J MK-MMD +J BDC )

[0031] J Caliration =J MK-MMD

[0032] Among them, J represents the overall objective function of the model, J Domain Represents the loss function of Source and Target data, J Domain represents the prediction loss function of Target calibration data and Test calibration data, ||ω|| 2 represents the L2 regularization term, J HingeLoss HingeLoss loss representing domain adaptation transformation features, J MK-MMD represents the multi-core maximum mean difference of the adapted transformation features, J BCD represents the maximum mean difference of multiple kernels for adaptive transformation features; λ1, λ2, λ, and α represent the calculation weights between different losses.

[0033] Preferably, in step 3), during the training of the cognitive state classification and discrimination model, after the data is input into the cognitive state classification and discrimination model, domain adaptation feature transformation and classification decision are performed in sequence. For Source and Target data, the error between the true value and the predicted value is calculated, and the multi-core maximum mean difference and Bhattacharyya distance are calculated for the features of the domain adaptation layer to estimate their distribution gap; for Target and Test calibration data, only the MK-MMD distance of the domain adaptation layer is calculated to adapt to the distribution difference of data of the same subject at different times; and an L2 regularization term is added to the loss function to reduce overfitting, feedback propagation is performed after the error is calculated, and the weights are updated layer by layer.

[0034] Compared with the prior art, the advantages of the present invention are:

[0035] The present invention designs a cognitive load inducing experimental paradigm in a simulated flight scenario, collects multimodal physiological data and task performance, calculates and extracts relevant feature indicators and performs effective fusion selection to obtain the optimal feature subset, and then constructs a cognitive state classification and discrimination model based on the optimal feature subset, so as to monitor and evaluate the pilot's cognitive load level, especially to provide early warning of the state mutation point of cognitive load overload, thereby improving flight safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 The present invention is a flowchart of an embodiment of the method.

[0037] Figure 2 Schematic diagram of the experimental environment and data collection in the present invention.

[0038] Figure 3 It is the experimental flow chart of the present invention.

[0039] Figure 4 Schematic diagram of multimodal feature critical point detection in the present invention.

[0040] Figure 5 This is a structural diagram of the cognitive state classification and discrimination model in the present invention.

[0041] Figure 6 This is a result diagram of the cognitive state classification accuracy in the present invention.

[0042] Figure 7 It is the ROC curve diagram of different models in the present invention. DETAILED DESCRIPTION

[0043] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments.

[0044] like Figure 1 As shown, the pilot cognitive load monitoring and early warning method based on EEG and eye movement data of the embodiment of the present invention comprises the following steps:

[0045] 1) Design an experimental paradigm to induce different cognitive load states of pilots in simulated flight mission scenarios, and obtain the multimodal physiological signals and task performance of the subjects during the experiment; the multimodal physiological signals include EEG signals and eye movement signals;

[0046] 2) Preprocess the multimodal physiological signals and task performance and extract the corresponding feature indicators, fuse and select the feature indicators to construct the optimal feature subset;

[0047] 3) Based on the optimal feature subset, a cognitive state classification and discrimination model is constructed to monitor and warn the pilot's cognitive load.

[0048] In a specific embodiment, in step 1), an experimental paradigm is designed to induce different cognitive load states of pilots in a simulated flight mission scenario, and the subjective scores, multimodal physiological signals and task performance of the subjects during the experiment are recorded. The experimental environment and data collection are as follows: Figure 2 As shown, the details are as follows:

[0049] During the experiment, NeuSen W32 wireless digital EEG acquisition system was used to collect and record EEG signals in real time at a frequency of 1000Hz. Tobii Pro Navo eye tracker was used to collect and record eye movement signals of the subjects at a frequency of 60Hz. At the same time, the performance of the subjects' simulated flight missions was recorded in real time. After each stage of the task, the subjects gave subjective scores on the subjective questionnaire scale NASA-TLX, and the cognitive load level of the subjects was evaluated from six dimensions.

[0050] The experiment embedded some tasks in the Multi-Attribute Task Battery (MATB) in the DCS World platform to build a simulated flight mission environment. DCS World is an air combat flight simulation software that can use Tacview software to record and replay the flight process trajectory. The experimental driving aircraft model is Su-25T. There are a series of guidance frames distributed in the flight airspace. The subjects use the flight simulation peripheral Thrust Mater T-16000M to control the aircraft. MATB is a simulated aviation mission developed by NASA. It is a classic task paradigm in the study of pilot cognitive load. It simplifies real flight tasks into a format that allows experimental control levels, which is convenient for analyzing the flight behavior of subjects and studying the impact of tasks on decision-making, cognitive load and vigilance. The subjects use keyboard keys to execute and respond to the tasks. The main task of the experiment (Flight) is to control the aircraft to maintain safe flight in a simulated flight environment and pass through a series of guidance frames according to the route, such as Figure 2 As shown in the box [1] in the figure, the subtasks are the system monitoring task (Sysmon), air traffic control communication task (Com) and resource management task (Resman) in the MATB task paradigm, as shown in the figure. Figure 2 As shown in boxes [2]-[4] in the figure.

[0051] The subjects will conduct three experiments with exactly the same content, for a total of 30 experiments. 30 groups of samples are collected across subjects and time. Based on the number of tasks processed in parallel and the difficulty of a single task, three groups of tasks with different difficulties are designed to try to induce three levels of cognitive load in the subjects: low, medium, and high, and to induce cognitive collapse in the high cognitive load stage. Specifically, the experimental process is as follows: Figure 3As shown, stage 0 is the preparation and calibration stage, during which the subjects perform flight preparation tasks. Stage 1 is the single-task flight stage, during which the subjects only need to complete the main task - the route flight task. Stages 2 and 3 are multi-task flight stages, during which three sub-tasks need to be completed while completing the main task. The difficulty of the main task is the same in different task stages, and the difficulty of the sub-task is determined by the task frequency of a single task. The frequency of sub-tasks in each stage is shown in Table 1. The preparation and calibration stage is 3 minutes, and the experimental time for each stage of the formal experiment is 6 minutes. After each stage of flight, the subjects filled in the subjective questionnaire scale NASA-TLX and were forced to rest for 1 minute;

[0052] Table 1 Experimental task difficulty settings

[0053]

[0054] In a specific embodiment, in step 2), the process of preprocessing and providing features of the EEG signal is as follows: the EEG signal is subjected to a 1 Hz high-pass filter and a 30 Hz low-pass filter, and a 48-52 Hz notch filter, to extract the EEG signal with a frequency range of 1-30 Hz and remove the power frequency interference in the signal. Interpolation processing is performed on the lead channels with poor data validity to ensure the validity of the data. Blind source analysis is performed on the processed EEG signal to remove artifacts such as electrooculography, electrocardiography, and electromyography. The EEG signal is averaged for the whole brain to improve the accuracy of the data. Finally, a sliding window with a step size of 1 s and a window size of 10 s is used to segment the data for further extraction of relevant feature indicators.

[0055] The frequency domain characteristics, nonlinear dynamic characteristics and functional connection characteristics of each frequency band (δ (1-3Hz), θ (3-8Hz), α (8-13Hz), β (13-30Hz)) of the EEG signal are calculated respectively. The frequency domain characteristics of the EEG signal are usually described by the power spectral density (PSD). The power spectral density of the EEG signal is calculated based on the Welch power spectrum estimation method, and the PSD mean of the corresponding frequency in each frequency band is calculated according to formula (1):

[0056]

[0057] Among them, N band is the number of frequency values ​​in the frequency domain, band high and band low is the range of frequency domain, PSD f is the power spectral density value corresponding to frequency f.

[0058] In order to eliminate individual differences and differences in energy intensity and variation in each frequency band, the calculated absolute power spectrum is normalized, and the relative value of the power spectrum in each frequency band and the ratio between specific frequency bands are further calculated. The complexity of the information volume of the EEG signal is measured by calculating the differential entropy (DE) feature of the EEG signal, and then different cognitive load levels are identified.

[0059] In addition, the EEG signal amplitude squared coherence (Magnitude Squared Coherence, COH) is also used as a feature. This indicator evaluates and distinguishes the cognitive load level by calculating the connectivity (information interaction) within and between brain regions. Specifically, the EEG signal is first subjected to a Laplace transform based on the second-order spatial derivative of the scalp electric field to solve the common source problem, and the COH is calculated for the transformed EEG signal to measure the relationship between the two leads (channels) in a specific frequency band, thereby revealing the connection between different brain regions and within brain regions. The EEG signal is collected through 32 leads (channels), and COH is the calculation of the relationship between the EEG signals of two leads (channels).

[0060] The eye movement signals collected by the Tobii Pro Nano eye tracker include: eyeball coordinates, line of sight direction, screen viewpoint coordinates, and pupil diameter. The above eye movement signals can be used to indirectly calculate the blink frequency, blink time, delayed blink time, area of ​​interest, and eye movement event type. Because the experiment simulates the execution of tasks in a flight environment, the subjects need to focus on the flight and various tasks during the experiment. The subjects' areas of interest and eye movement events such as saccades and gazes are closely related to the task settings, and there are great differences in each task stage. Therefore, based on the validity of the eye movement signal, a blink threshold is set. If the duration of invalid data exceeds the threshold, it is considered that the eyes are looking at the area outside the screen, yawning, and other long-term eye closures that cannot capture eye movement signals. Otherwise, it is considered a blink event. The average closing time of the blinking process is calculated as the blink duration, the number of blinks per second is calculated as the blink frequency, and the average left and right pupil diameters of the valid data are calculated as the pupil diameter index of the subject.

[0061] For the main task, the flight records during the experiment are used to detect in real time whether there is a guide frame in the current picture. If there is, the size of the guide frame indirectly reflects the distance between the aircraft and the guide frame, and then calculates whether the aircraft has completed the guide frame crossing and the crossing accuracy. The larger the guide frame, the better the flight completion. The secondary task can reflect the task performance by calculating indicators such as task reaction time or accuracy based on the task record log. Because stage 1 only has the main task, the secondary task only has performance indicators for the last two stages.

[0062] When performing feature fusion and feature selection, single-modal signals are susceptible to various noises during the classification and recognition process, and it is difficult to fully reflect the cognitive state. Different physiological signals reflect the cognitive state from different levels. Therefore, in practical applications, multimodal data is fused and used, and different categories of signals are used to support each other and fuse complementary information to improve the classification effect. Each feature has different effects on cognitive state recognition. Simply splicing them may not be the best choice. Problems such as collinear redundancy between features and insensitivity of features to classification tasks will have a negative impact on the classification effect. Therefore, the feature-level fusion method is adopted to extract feature indicators from each modal data, cascade them into a multimodal feature vector, and further screen the fused multimodal feature vector through a three-stage feature selection strategy to construct the optimal feature subset for cognitive load recognition. Specifically:

[0063] Assume that the multimodal fusion feature set is S0. First, use variance analysis to preliminarily select features, remove features with small inter-class differences, and select feature subset S1 with large inter-class differences;

[0064] Secondly, the Pearson correlation coefficients between all the features in S1 are calculated. When the correlation coefficient exceeds the threshold, the feature with a higher overall correlation with other variables in S1 is removed until the correlation coefficients between all the features are lower than the given threshold. The feature subset S2 is obtained to solve the problem of collinear redundancy between different channel information of the same modality and different modal information.

[0065] Finally, the Sequential Backward Feature Selection (SBFS) method is used to select features related to the classification target, and the features that contribute most to the classification task are selected in S2: the objective function is calculated using the N-1 dimensional features in the N-dimensional feature set in turn, and this process is repeated N times to find the features deleted in the best performance model, remove the features from the N-dimensional features, and repeat the above process with the remaining N-1 dimensional features until deleting the features does not produce better classification results, thus obtaining the optimal feature subset S3.

[0066] In a specific embodiment, in step 3), the critical state of cognitive collapse of the subject is distinguished from the normal cognitive state signal. When an emergency situation occurs or the task load increases sharply, the warning signal is identified in time before the subject's cognitive collapse, thereby realizing cognitive collapse warning.

[0067] Specifically, firstly, cognitive load state labels are set for the multimodal feature dataset. The normal states of low, medium and high cognitive loads together constitute the normal cognitive load data. The 10 window data before the collapse point in high cognitive load are the cognitive collapse early warning signals (EWS). Considering the characteristics of experimental data, the experimental data of one experiment is divided into a data subset as the test set, and all other trial data are used as the training set to cross-validate and evaluate the classification accuracy of the model. Finally, the Two-Step early warning method of "first determining the cognitive load level, and then classifying the high cognitive load signal into EWS and normal state signals" is used, and a new cognitive state classification model is designed in each stage of the classification task. The domain adaptation layer is added to the model and a new objective function is designed to solve the overfitting problem caused by the small sample size and unstable features of the dataset, and to achieve accurate identification of EWS.

[0068] The low, medium and high levels of cognitive load are calibrated according to the experimental design stage, and stage1-stage3 are for low, medium and high cognitive load states respectively. The multivariate mutation test method is used to calculate the difference values ​​of all window data in the left and right parts of each point in the high cognitive load stage. If the maximum difference value is greater than the set threshold, and there are at least 20 data before the point and at least 10 data after the point, then this position is determined to be a cognitive collapse point, otherwise it is considered that no cognitive collapse occurred in this experiment (such as Figure 4 As shown in Figure 2.10, the data labels of the 10 windows before the crash point are set to EWS, and the labels of all data in the low cognitive load and medium cognitive load stages and other data before the crash point in the high cognitive load stage are set to normal.

[0069] Similar to the method of first extracting the target candidate area and then classifying and identifying the target in the target detection algorithm, in order to reduce the difficulty of crash warning and prevent the imbalance of positive and negative samples, a Two-Step warning method of "first determining the cognitive load level, and then classifying the high cognitive load signal into critical state and normal state" is adopted. For a certain experimental data, the step 1 stage classification model is first input, and the model output is the cognitive load level. The data of the high cognitive load stage is input into the step 2 crash warning classification model to achieve classification and identification of normal and EWS signals. The cognitive state classification model of the present invention includes a domain adaptation method that can solve the instability of feature signals and a classification objective function suitable for small sample data. The specific model structure is as follows: Figure 5 shown.

[0070] Based on the characteristics of experimental data, a cross-time and cross-subject cognitive state classification and discrimination model suitable for small sample multimodal data is designed. The model consists of a three-layer structure of input layer, domain adaptation layer and cognitive state classification layer, and a new objective function is designed, specifically:

[0071] J=λ1J Domain +λ2J Calibration +λ||ω|| 2 (4)

[0072] J Domain =J HingeLoss +α(J MK-MMD +J BCD ) (5)

[0073] J Caliration =J MK-MMD (6)

[0074] Among them, J represents the overall objective function of the model, J Domain Represents the loss function of Source and Target data, J Domain represents the prediction loss function of Target calibration data and Test calibration data, ||ω|| 2 represents the L2 regularization term, J HingeLoss HingeLoss loss representing domain adaptation transformation features, J MK-MMD represents the multi-core maximum mean difference of the adapted transformation features, J BCD represents the maximum mean difference of multiple kernels for adaptive transformation features; λ1, λ2, λ, and α represent the calculation weights between different losses.

[0075] During the training process of the model, after the data is input into the model, domain adaptation feature transformation and classification decision are performed in sequence. For Source and Target data, the error (HingeLoss) between the true value and the predicted value is calculated, and the multi-kernel maximum mean difference (Multiple Kernel Maximum Mean Discrepancy, MK-MMD) and Bhattacharyya Distance (BCD) are calculated for the features of the domain adaptation layer to estimate their distribution gap; for Target and Test calibration data, only the MK-MMD distance of the domain adaptation layer is calculated to adapt to the distribution difference of the data of the same subject at different times; and the L2 regularization term is added to the loss function to reduce overfitting, and feedback propagation is performed after the error is calculated, and the weights are updated layer by layer. When the model makes predictions, the data undergoes domain adaptation transformation and classification decision outputs the classification results.

[0076] The present invention designs a cognitive load inducing experimental paradigm in a simulated flight scenario, collects multimodal physiological data and task performance, calculates and extracts relevant feature indicators and performs effective fusion selection to obtain the optimal feature subset, and then constructs a cognitive state classification and discrimination model based on the optimal feature subset, so as to monitor and evaluate the pilot's cognitive load level, especially to provide early warning of the state mutation point of cognitive load overload, thereby improving flight safety.

[0077] The pilot cognitive load inducing experimental paradigm designed by the present invention in a simulated flight mission scenario has an environment and task settings close to a real flight mission scenario, and can effectively induce three cognitive load states with distinction and can induce a cognitive breakdown state in a high cognitive load stage.

[0078] The present invention optimizes the feature selection method, extracts and fuses the features of the data, and then optimizes and selects the features from multiple aspects in turn, obtains the optimal feature subset related to cognitive load, and constructs an effective early warning feature indicator vector.

[0079] The present invention proposes a cross-subject and cross-time cognitive collapse early warning method suitable for small sample multimodal data. Its cognitive state classification model consists of feature domain adaptation transformation and cognitive state classifier, and a new objective function is designed to solve the problems of feature signal instability across subjects and time and insufficient sample data.

[0080] In order to evaluate the model performance, the cognitive state classification and discrimination model proposed in the present invention is compared with an integrated model (such as XGBoost), a subspace learning method (such as subspace alignment SA) and a domain adaptation method (such as joint distribution adaptation JDA and domain adaptation neural network DaNN), and cross-validated based on a data set of 30 trials. In each step of cognitive state classification, the accuracy index can be used to analyze the effect of the model separately. The results of the cognitive state classification accuracy of each subject in the two steps are shown in Figure 2. Figure 6 As shown, the average accuracy is shown in Table 2:

[0081] Table 2 Performance comparison of different models

[0082]

[0083] It can be concluded from the results that the present invention achieved the highest accuracy for 70% of the subjects in the first step and for 60% of the subjects in the second step. The proposed model performed best in the first step classification, with an accuracy of 82.09% and a second step classification accuracy of 61.83%, second only to the classification accuracy of the JDA model (62.33%). However, due to the extremely unbalanced sample size of normal and EWS data used for overall early warning, the accuracy will be very high even if all samples are predicted to be normal. Therefore, the receiver operating characteristic (ROC) curve describes the discriminative ability of the classifier by adjusting the relationship between the true positive rate (TPR) and the false positive rate (FPR) under different thresholds. The ROC curve can be quantified by the area under the curve (AUC). The ROC curve of the early warning method is shown in Figure 2. Figure 7The quantitative results are shown in Table 3. The AUC of the model of the present invention is 0.78, which again achieves the best performance. The ROC curve of the model of the present invention is closer to the curves of other models. Figure 7 The upper left corner of the figure. From the qualitative description and quantitative comparison, it can be concluded that the model of the present invention is superior to other models in early warning of cognitive critical points. The classification threshold of 0.39, that is, the threshold corresponding to the maximum value of TPR minus FPR, is used to obtain the recall rate and false alarm rate of the model. It can be seen that the recall rate is 71.33% and the false alarm rate is 26.38%, which can achieve effective identification of EWS and timely warning of cognitive collapse.

[0084] The Chinese and English correspondences of relevant nouns are: cognitive workload: mental workload; cognitive collapse: cognitive collapse; critical slowing down: critical slowing down; early warning signal: early warning signal; tipping point: tipping point; electroencephalogram: electroencephalogram, EEG; eye movement signal: eye movement.

[0085] The above are only preferred embodiments of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should be regarded as the protection scope of the present invention.

Claims

1. A pilot cognitive load monitoring and early warning method based on EEG and eye movement data, characterized in that: Includes steps: 1) Design an experimental paradigm to induce different cognitive load states of pilots in simulated flight mission scenarios, and obtain the multimodal physiological signals and task performance of the subjects during the experiment; The multimodal physiological signals include EEG signals and eye movement signals; 2) Preprocess the multimodal physiological signals and task performance and extract the corresponding feature indicators, fuse and select the feature indicators to construct the optimal feature subset; 3) Based on the optimal feature subset, a cognitive state classification and discrimination model is constructed to monitor and warn the pilot's cognitive load.

2. The pilot cognitive load monitoring and early warning method based on EEG and eye movement data according to claim 1 is characterized in that: In step 2), the specific process of preprocessing the EEG signal in the multimodal physiological signal is as follows: Perform high-pass filtering, low-pass filtering and notch filtering on the EEG signals to extract EEG signals in a specific frequency range; Perform blind source analysis on EEG signals in a specific frequency range to remove eye artifacts; Then the EEG signal is averaged over the whole brain for reference; Finally, the sliding window is used to segment the EEG signal.

3. The pilot cognitive load monitoring and early warning method based on EEG and eye movement data according to claim 2 is characterized in that: In step 2), the characteristic indicators of the EEG signal include frequency domain characteristics, nonlinear dynamic characteristics and functional connection characteristics of each frequency band of the EEG signal; The frequency domain characteristics of the EEG signal are described as power spectral density. The power spectral density of the EEG signal is calculated by the Welch power spectrum estimation method, and the mean power spectral density of the corresponding frequency in each frequency band is calculated according to the following formula: Among them, N band is the number of frequency values ​​in the frequency domain, band high and band low is the range of frequency domain, PSD f is the power spectral density value corresponding to frequency f.

4. The pilot cognitive load monitoring and early warning method based on EEG and eye movement data according to claim 3 is characterized in that: In step 2), the characteristic indicators of the EEG signal also include the square coherence of the EEG signal amplitude. This characteristic indicator evaluates and distinguishes the cognitive load level by calculating the connectivity within and between brain regions. Specifically, the EEG signal is first subjected to a Laplace transform based on the second-order spatial derivative of the scalp electric field to solve the common source problem, and then the square coherence of the EEG signal amplitude after Laplace transform is calculated to measure the relationship between the two EEG signals in a specific frequency band, so as to reveal the connection between different brain regions and within brain regions.

5. The pilot cognitive load monitoring and early warning method based on EEG and eye movement data according to any one of claims 1 to 4, characterized in that: In step 2), after extracting the corresponding feature indicators, they are cascaded and fused into a multimodal feature vector, and the fused feature indicators are screened through a three-stage feature selection strategy to construct the optimal feature subset; specifically: Assume that the initial multimodal fusion feature set is S0. First, use variance analysis to preliminarily select the initial multimodal fusion feature set S0, remove the features with small inter-class differences, and select the feature subset S1 with large inter-class differences. Secondly, the correlation coefficients between all the features in feature subset S1 are calculated; when the correlation coefficient exceeds the threshold, the feature with a higher overall correlation with other variables in feature subset S1 is removed until the correlation coefficients between all the features are lower than the given threshold, and feature subset S2 is obtained to solve the problem of collinear redundancy between different channel information of the same modality and different modal information; Finally, the sequential backward feature selection method is used to select features related to the classification target, and the features that contribute greatly to the classification task are selected in the feature subset S2: the objective function is calculated using the N-1 dimensional features in the N-dimensional feature set in turn, and this process is repeated N times to find the features deleted in the best performance model, remove the features from the N-dimensional feature set, and repeat this process with the remaining N-1 dimensional features until deleting the features does not produce better classification results, and the optimal feature subset S3 is obtained.

6. The pilot cognitive load monitoring and early warning method based on EEG and eye movement data according to any one of claims 1 to 4, characterized in that: In step 1), three groups of tasks with different difficulty levels are designed to induce low, medium and high cognitive load levels in the subjects, and a cognitive collapse state is induced in the high cognitive load stage.

7. The pilot cognitive load monitoring and early warning method based on EEG and eye movement data according to claim 6 is characterized in that: In step 3), in the process of monitoring and warning the pilot's cognitive load, the cognitive load level is first determined, and then the high cognitive load signal is classified into a cognitive breakdown warning signal and a normal state signal.

8. The pilot cognitive load monitoring and early warning method based on EEG and eye movement data according to claim 7 is characterized in that: The determination process of cognitive breakdown warning signals is as follows: The multivariate mutation test method is used to calculate the difference values ​​of all window data on the left and right sides of each data point in the high cognitive load stage in turn; if the maximum difference value is greater than the set threshold, and there are at least 20 data before the data point and at least 10 data after the point, then this position is determined to be a cognitive collapse point, and the labels of the 10 window data before the cognitive collapse point are set as cognitive collapse warning signals.

9. The pilot cognitive load monitoring and early warning method based on EEG and eye movement data according to any one of claims 1 to 4, characterized in that: In step 3), the cognitive state classification model includes an input layer, a domain adaptation layer, and a cognitive state classification layer, and a new objective function is designed, specifically: J=λ1J Domain +λ2J Calibration +λ||ω|| 2 I Domain =J HingeLoss +α(J MK-MMD +J BDC ) I Caliration =J MK-MMD Among them, J represents the overall objective function of the model, J Domain Represents the loss function of Source and Target data, J Domain represents the prediction loss function of Target calibration data and Test calibration data, ||ω|| 2 represents the L2 regularization term, J HingeLoss HingeLoss loss representing domain adaptation transformation features, J MK-MMD represents the multi-core maximum mean difference of the adapted transformation features, J BCD represents the maximum mean difference of multiple kernels for adaptive transformation features; λ1, λ2, λ, and α represent the calculation weights between different losses.

10. The pilot cognitive load monitoring and early warning method based on EEG and eye movement data according to claim 9 is characterized in that: In step 3), during the training process of the cognitive state classification and discrimination model, after the data is input into the cognitive state classification and discrimination model, domain adaptation feature transformation and classification decision are performed in sequence. For Source and Target data, the error between the true value and the predicted value is calculated, and the multi-core maximum mean difference and Bhattacharyya distance are calculated for the features of the domain adaptation layer to estimate their distribution gap. For Target and Test calibration data, only the MK-MMD distance of the domain adaptation layer is calculated to adapt to the distribution difference of the data of the same subject at different times. In addition, the L2 regularization term is added to the loss function to reduce overfitting. After calculating the error, feedback propagation is performed to update the weights layer by layer.

Images