Multi-mode electroencephalogram characteristic cognitive ability evaluation system based on neural network

Abstract

The invention relates to the field of electroencephalogram signal processing, and particularly discloses a multi-mode electroencephalogram characteristic cognitive ability evaluation system based on a neural network, comprising: an acquisition module used for synchronously acquiring whole brain signals and eye movement data during pilot simulation training; the data preprocessing module is used for primarily processing the received whole brain signals and the eye movement data, monitoring the primarily processed data in real time by utilizing a preset monitoring mechanism, and triggering to generate interaction test starting signals if preset symbolic fluctuation occurs in the whole brain signals or the eye movement data; and the test interaction sub-module is used for generating arithmetic test questions according to a preset rule after receiving the interaction test starting signal, and synchronously recording answer correlation parameters of the pilot. By adopting the technical scheme of the invention, the deep-level reaction mechanism of the brain under the stimulation of the special complex cognitive task of flight training can be fully considered, and the cognitive load change of the pilot in the simulated training can be comprehensively and accurately reflected.

Description

Technical Field

[0001] The present invention relates to the field of electroencephalogram signal processing, and particularly to a multi-modal electroencephalogram feature cognitive ability evaluation system based on a neural network. Background Art

[0002] With the rapid development of technologies in the aviation field, the complexity of flight missions has been increasing day by day, and the requirements for pilots' cognitive abilities have become more and more stringent. During the process of pilot training, simulated flight training has become a key link. Accurately evaluating the cognitive state of pilots during training is of great significance for optimizing training programs and preventing flight risks.

[0003] Traditional cognitive load evaluation methods have many limitations. In the early stage, subjective evaluation methods were mostly relied on. For example, based on questionnaires and scales, pilots were asked to self-evaluate their perception of task complexity. However, this method is only applicable to experimental environments and has serious deficiencies in real-time performance. It cannot provide immediate feedback to pilots and ground crew during the dynamic process of flight training and is difficult to meet the actual training needs.

[0004] Subsequently, evaluation means based on objective indicators have been developed. Some studies focus on electroencephalogram signals. For example, in the technology disclosed in Chinese Patent Application No.: 202210757622.3, electroencephalogram signals are collected through a wearable electrode cap, and power spectral density, functional connectivity, and microstate features are extracted by using Fourier transform, clustering algorithms, etc. Then, cognitive load is judged through feature fusion and traditional classification algorithms. Another example is in the literature of Chinese Application No. 202010234303.5, which attempts to combine eye movement and electroencephalogram data, select specific eye movement indicators (such as 7 indicators including gaze duration, saccade count, etc.) and electroencephalogram alpha wave data, and calculate the cognitive load based on the support vector machine algorithm.

[0005] Although the above technologies have improved real-time performance to a certain extent, in the actual use process, they have not fully considered the deep reaction mechanism of the brain under the complex cognitive tasks unique to flight training and are difficult to comprehensively and accurately reflect the cognitive changes of pilots during simulated training.

[0006] Therefore, there is an urgent need for a multi-modal electroencephalogram feature cognitive ability evaluation system based on a neural network that can fully consider the deep reaction mechanism of the brain under the complex cognitive tasks unique to flight training and comprehensively and accurately reflect the cognitive load changes of pilots during simulated training on the premise of ensuring real-time performance. Summary of the Invention

[0007] The present invention provides a multi-modal electroencephalogram feature cognitive ability evaluation system based on a neural network, which can fully consider the deep reaction mechanism of the brain under the complex cognitive tasks unique to flight training and comprehensively and accurately reflect the cognitive load changes of pilots during simulated training.

[0008] In order to solve the above technical problems, this application provides the following technical solutions:

[0009] The multimodal EEG feature cognitive ability assessment system based on neural network includes:

[0010] An acquisition module, used to simultaneously acquire whole-brain signals and eye movement data during pilot simulation training;

[0011] The data preprocessing module is used to perform preliminary processing on the received whole-brain signals and eye movement data, and use the preset monitoring mechanism to monitor the preliminarily processed data in real time. If the whole-brain signals or eye movement data show a preset characteristic fluctuation, it triggers the generation of an interactive test start signal;

[0012] The test interaction submodule is used to generate arithmetic test questions according to preset rules after receiving the interactive test start signal, and synchronously record the pilot's answer correlation parameters, and then summarize and integrate the answer correlation parameters with the pre-processed whole brain signal and eye movement data into the test data signal;

[0013] The feature extraction module is used to extract the power spectrum density features of different frequency bands from the whole brain signal using the time-frequency analysis method; identify different eye movement patterns through the clustering algorithm and extract the statistical results of each pattern to obtain the visual attention allocation characteristics; perform feature analysis on the answer correlation parameters to obtain the answer response characteristics;

[0014] The neural network model module is connected to the feature extraction module. It uses a combination of long short-term memory network (LSTM) and convolutional neural network (CNN). The power spectrum density features, visual attention allocation features, and answer response features are first input into CNN for spatial feature extraction. The output of CNN is used as the input of LSTM, and finally the evaluation result is output by LSTM.

[0015] Among them, LSTM pre-learns the dependencies and changing trends of whole-brain signals and eye movement data in the time dimension, maps the output of LSTM to the results of cognitive ability assessment through the fully connected layer, and pre-trains CNN and LSTM using a large amount of historical flight training data with real cognitive assessment labels, and continuously adjusts the weights and biases of CNN and LSTM through the back-propagation algorithm.

[0016] Furthermore, the whole brain signal is a multi-channel EEG signal collected by a 10-10 standard lead system. The preliminary processing is: for the whole brain signal, a linear interpolation method is used to process some missing values. Suppose a channel has a missing value at time t, and its signal value at the previous time t-1 is x t -1, the signal value at the next moment t+1 is x t+1 , then by the formula Fill in the missing values; for outliers, identify and process them based on the 3σ principle of statistics. First, calculate the mean μ and standard deviation σ of the whole-brain signal of this channel. When the signal value x at a certain moment satisfies ∣x - μ∣> 3σ, it is determined as an outlier. At this time, replace the outlier with the median of the signal values at k moments before and after it in its channel.

[0017] Furthermore, the preliminary processing is as follows: for eye movement data, use the mean filtering method to remove noise interference. Let the eye movement data sequence be y i , i = 1, 2, …, n, the size of the filtering window is m, then the filtered eye movement data y i ′ satisfies: where the value range of i is to to ensure that the filtering window can completely cover the sequence and does not exceed the boundary.

[0018] Furthermore, the preset monitoring mechanism is implemented based on the sliding window algorithm and the principle of statistical hypothesis testing. Set a sliding time window with a length of T, which slides along the time axis at a fixed time interval Δt, and perform real-time acquisition and analysis on the whole-brain signal or eye movement data within the window; for the whole-brain signal, decompose it into Delta, Theta, Alpha, Beta, and Gamma frequency bands, and calculate the mean μ i and standard deviation σ i of the power spectral density in each frequency band within the window, where i represents different frequency bands. Use the one-sample t-test, set the significance level as α, when is satisfied, it is determined that there is a signature fluctuation in this frequency band, where μ 0i is the mean of the power spectral density pre-acquired and statistically obtained when the pilot is in a normal cognitive state, n is the number of data points within the window, and t α / 2,n-1 is the two-sided t-quantile corresponding to the α level with a degree of freedom of n - 1;

[0019] For eye movement data, the eye movement data includes fixation duration and saccade count. Calculate the mean μ g and standard deviation σ g of the fixation duration and saccade count within the window respectively. Also use the one-sample t-test. When is satisfied, it is determined that there is a signature fluctuation in the eye movement data, and μ 0g is the mean of the corresponding eye movement index in the normal cognitive state; when it is detected that either the whole-brain signal or the eye movement data satisfies the above signature fluctuation condition, the interactive test start signal is triggered.

[0020] Furthermore, monitor the change rate R f of the power spectral density of each frequency band of the whole-brain signal. The calculation formula of the power spectral density change rate is: where, Pf (t) is the power spectral density of a certain frequency band f at time t, P f (t - Δt) is the power spectral density of this frequency band at time t - Δt; the characteristic fluctuations of the whole-brain signal are divided into levels L1, L2, and L3 according to the magnitude of the power spectral density change rate; when ∣Rf∣ ≤ 0.1, it is level L1; when 0.1 < ∣Rf∣ ≤ 0.3, it is level L2; when ∣Rf∣ > 0.3, it is level L3;

[0021] The eye movement data also includes the pupil area. Taking the gaze duration T, the saccade count S, and the pupil area A as input parameters, let T(t), S(t), and A(t) be the gaze duration, saccade count, and pupil area at time t respectively, then T(t - Δt), S(t - Δt), and A(t - Δt) are the corresponding index values at time t - Δt, and then calculate the change rates of each index respectively:

[0022] Gaze duration change rate:

[0023] Saccade count change rate:

[0024] Pupil area change rate:

[0025] Then, the comprehensive change rate R c The calculation formula is: R c = w T R T + w S R S + w A R A ; where w T 、w S 、w A are the weight coefficients of the gaze duration, saccade count, and pupil area change rates respectively, and satisfy w T + w S + w A = 1; the weight coefficients can be obtained through training with a large amount of experimental data and multiple linear regression; according to the magnitude of the comprehensive change rate R c The characteristic fluctuations of the eye movement data are divided into levels E1, E2, and E3: when ∣R c ∣ ≤ 0.15, it is level E1; when 0.15 < ∣R c ∣ ≤ 0.35, it is level E2; when ∣R c ∣ > 0.35, it is level E3;

[0026] According to the difficulty of arithmetic test questions, they are divided into simple difficulty questions D1, medium difficulty questions D2, and high difficulty questions D3;

[0027] A mapping matrix M is established to represent the correlation between the whole-brain signal and the signature fluctuation levels of eye movement data and the difficulty levels of arithmetic test questions:

[0028]

[0029] Among them, the rows represent the signature fluctuation level L of the whole-brain signal i (i = 1, 2, 3), and the columns represent the signature fluctuation level E of the eye movement data j (j = 1, 2, 3), and the matrix element m ij represents the corresponding difficulty level of the arithmetic test question;

[0030] When i = 1 and j = 1, m 11 = D1, that is, when both the whole-brain signal and the eye movement data are mildly fluctuating, the simple-difficulty question D1 is selected;

[0031] When i = 1 and j = 2, or i = 2 and j = 1, the simple-difficulty question D1 or the medium-difficulty question D2 is randomly selected with a preset probability;

[0032] When i = 2 and j = 2, that is, when both the whole-brain signal and the eye movement data are moderately fluctuating, the medium-difficulty question D2 is selected;

[0033] When i = 2 and j = 3, or i = 3 and j = 2, the medium-difficulty question D2 or the high-difficulty question D3 is randomly selected with a preset probability;

[0034] When i = 3 and j = 3, that is, when both the whole-brain signal and the eye movement data are severely fluctuating, the high-difficulty question D3 is selected.

[0035] Furthermore, the feature extraction module extracts features from the whole-brain signal by using a method based on wavelet packet transform combined with adaptive threshold screening; first, the collected whole-brain signal x(t) is decomposed into N layers by wavelet packet transform to obtain a series of wavelet packet coefficients c i，j , where i represents the decomposition layer number, and j represents the sub-band serial number at this layer;

[0036] For each sub-band, its energy E i，j is calculated, and the calculation formula is:

[0037]

[0038] where k is the sample point index; then, an adaptive threshold T i，j is set and determined by the following formula:

[0039] T i，j = μ i，j + α·σ i，j

[0040] where μi，j is the mean value of sub - band E i，j , σ i，j is the standard deviation, α is an adjustable threshold coefficient, and its value range is optimized according to experimental data;

[0041] Only retain the wavelet packet coefficients corresponding to the sub - bands with energy greater than the threshold T i，j , and reconstruct the signal to obtain the optimized whole - brain signal x′(t); then, extract the power spectral density features PSD of different frequency bands from x′(t) f . Using the Welch method, segment the signal and apply a window. The window function is selected as the Hamming window, and calculate the power spectral density through the fast Fourier transform:

[0042]

[0043] where U is the number of segments, N FFT is the number of FFT points, f s is the sampling frequency, and x′ u (n) is the u - th segment of the signal.

[0044] This method can more accurately focus on the frequency bands containing effective cognitive information, remove the noise - interference frequency bands, and the extracted whole - brain signal features are more distinguishable.

[0045] Furthermore, the feature extraction module extracts features from the eye movement data using a differential clustering algorithm based on the features extracted from the whole - brain signal; the frequency bands of the whole - brain signal are re - divided into a low - frequency band of 1 - 10Hz, a middle - frequency band of 11 - 30Hz, and a high - frequency band of 31 - 80Hz, which respectively correspond to different cognitive states: the low - frequency band is related to maintaining basic cognition, the middle - frequency band reflects the dynamic changes of cognitive load, and the high - frequency band is involved in processing complex cognitive tasks.

[0046] Furthermore, when the energy proportion P low of the low - frequency band of the whole - brain signal, that is, the ratio of the sum of the energies of each sub - band in the low - frequency band to the total energy, is greater than 60%, use the K - means++ clustering algorithm for the eye movement data, and perform clustering based on two indicators: the gaze duration T and the saccade count S; let the cluster centers be μ k (k = 1, 2, …, K), and the distance d i from the sample point x i，k to the cluster center is calculated using the Euclidean distance:

[0047]

[0048] By iteratively updating the cluster centers to minimize the sum of the distances from the sample points to their respective cluster centers, different eye movement pattern clusters are obtained; extract the statistical results of each pattern cluster: the average gaze duration of each cluster average saccade count As a visual attention allocation feature, it reflects the visual attention law in the basic cognitive state.

[0049] Further, when the proportion of medium-frequency energy P in the whole-brain signal mid is between 30% and 60%, the hierarchical clustering algorithm is applied to the eye movement data, and the pupil area A index is added to construct the sample vector x i =[T i , S i , A i ; The Euclidean distance is used to measure the distance between samples. By continuously merging the closest clusters, a hierarchical clustering structure is formed; according to the given number of clusters or distance threshold, different eye movement patterns are divided, and the statistical features of each pattern are extracted: the pattern duration ratio ρ m , that is, the ratio of the duration of a certain pattern to the total observation time, and the conversion frequency ω between different patterns mn , where m and n represent different patterns; it is used to characterize the visual attention transfer characteristics when the cognitive load changes dynamically.

[0050] Further, when the proportion of high-frequency energy P in the whole-brain signal high is greater than 30%, the Gaussian mixture model GMM clustering algorithm is used to process the eye movement data. Considering the three indicators T, S, and A comprehensively, a multi-dimensional Gaussian distribution model is constructed; assuming there are M Gaussian components, the model parameters θ = {π m , μ m , Σ m} (m = 1, 2,..., M), where π m is the mixing weight, μ m is the mean vector, and Σ m is the covariance matrix. The optimal parameters are iteratively solved through the expectation maximization EM algorithm, and the eye movement data is divided into patterns with different probability distributions, and the high-order statistical features of each pattern are extracted: the information entropy H m , which measures the pattern uncertainty, and the determinant value of the covariance matrix |Σ m |, which reflects the correlation strength between each indicator; this reflects the fine allocation and change characteristics of visual attention under complex cognitive tasks.

[0051] Principle and technical effect of the solution: During the pilot simulation training, the acquisition module synchronously acquires the whole-brain signal through professional equipment. The 64-channel (10-10) whole-brain signal acquisition device used strictly follows the international standard for electrode positioning and can capture the electrophysiological activities of each region of the pilot's cerebral cortex with high resolution and all-round. When the brain performs flight training tasks, different regions will generate electro-signal changes of different frequencies and amplitudes due to the needs of cognitive processing. These whole-brain signals contain rich information, reflecting the overall neural activity state of the brain and providing basic data for subsequent in-depth analysis of the cognitive process.

[0052] At the same time, high-precision eye trackers work closely together to record eye movement data. As an important organ for obtaining external information, the movement pattern of the eye is closely linked to the cognitive processing of the brain. For example, the duration of gaze reflects the degree of concentration of the pilot's attention in a specific area, the number of glances shows the frequency of his visual search, and the change in pupil area is related to the excitement and cognitive load of the brain. These eye movement data reflect the brain's processing of flight training scene information from the perspective of visual attention allocation.

[0053] The data preprocessing module uses filtering algorithms to effectively remove environmental noise, electromagnetic interference, and artifacts caused by body movement for the collected whole-brain signals, ensuring the purity of the signal so that subsequent analysis can focus on the changes in electrical signals that are truly generated by brain cognitive activities. For eye movement data, smoothing and other technologies are used to correct data deviations caused by factors such as slight head shaking and blinking, restoring the true eye movement trajectory.

[0054] Using the preset monitoring mechanism, the module analyzes the preliminarily processed data in real time. This monitoring mechanism is based on the continuous tracking of the frequency band characteristics of the whole-brain signal and the changes in key indicators of the eye movement data. For example, abnormal fluctuations in the power spectrum density of the frequency bands of the whole-brain signal (such as Delta, Theta, Alpha, Beta, and Gamma waves), or sudden deviations of the gaze duration and number of glances in the eye movement data from the normal range may indicate changes in the pilot's cognitive state. Once the preset landmark fluctuations are detected, the interactive test start signal is immediately triggered to provide an opportunity to capture cognitive changes in a timely manner.

[0055] When the trigger signal is received, the test interaction submodule generates arithmetic test questions according to carefully designed preset rules. These questions are closely centered around key cognitive tasks in flight training, such as flight parameter calculations, navigation decisions, instrument recognition feedback, and other scenarios, and are designed to simulate complex cognitive challenges during flight. When pilots answer these questions, the brain needs to mobilize a variety of cognitive resources, including memory, calculation, logical reasoning, and attention allocation. By synchronously recording the pilot's answer-related parameters, such as answer accuracy and answer time, the current cognitive ability state is quantitatively reflected from the task performance level. These answer data are then aggregated and integrated with the pre-processed whole-brain signals and eye movement data to form a multi-dimensional and more representative test data signal that reflects the full picture of the cognitive process in all directions.

[0056] The feature extraction module uses time-frequency analysis methods, such as wavelet transform, on the whole-brain signals, converting the whole-brain signals in the time domain into frequency-domain information and accurately extracting the power spectral density features of different frequency bands. The power spectral density of different frequency bands is closely related to specific cognitive functions of the brain. For example, the Alpha wave is often related to the relaxed and standby state of the brain, and the Beta wave increases when attention is concentrated and thinking is active. By analyzing the changes in the power spectral density of each frequency band, the activity level and functional state switching of the brain at different flight training stages and different task difficulties can be understood.

[0057] The clustering algorithm is used to process the eye movement data. Different eye movement patterns, such as focused gaze, rapid saccade, and frequent refixation, are identified based on features such as eye movement trajectory, speed, and pause. Corresponding statistical results are extracted for each pattern, such as the time proportion and frequency of a certain eye movement pattern, so as to clearly show the distribution strategy, transfer path, and focus of visual attention in the flight training scenario, and assist in understanding the cognitive orientation of the brain.

[0058] A deep analysis is carried out on the question-answering relevance parameters. The answering accuracy rate is calculated to reflect the accuracy of knowledge mastery and application, the average answering time reflects the reaction speed, and the standard deviation of the time measures the stability of the reaction. These features reverse-infer key cognitive traits such as information processing efficiency and decision-making accuracy of the brain when dealing with complex cognitive tasks from the task execution results.

[0059] The neural network model module adopts an innovative architecture that combines long short-term memory network (LSTM) and convolutional neural network (CNN). CNN is good at automatically extracting the spatial features of data. For the features of the whole-brain signals in the spatial distribution of the cerebral cortex and the features of the eye movement trajectory images, it can efficiently capture them through the convolutional layer and pooling layer. And because of its special gating structure, LSTM can effectively process data with time series characteristics, learn the dynamic change laws of the whole-brain signals and eye movement data over time, and remember the key information in the long time series, such as the gradual change process of the brain cognitive state and the stage adjustment of the eye movement pattern.

[0060] A large amount of historical flight training data with real cognitive assessment labels is used to train CNN and LSTM in advance. During the training process, through the backpropagation algorithm, the weights and biases of the network are continuously adjusted according to the error between the prediction result and the real label. This enables the model to gradually learn the complex non-linear mapping relationship between multi-modal data features and pilots' cognitive abilities, and be able to summarize the typical patterns of data features in different cognitive states from a large amount of historical experience, and then accurately identify the current cognitive state of the pilot in practical applications.

[0061] By synchronously collecting whole-brain signals and eye movement data, and combining the brain stress response stimulated by arithmetic tests, the system realizes multi-dimensional collaborative analysis from neuroelectrophysiological activities, visual attention allocation to cognitive task performance. Data from different modalities corroborate and complement each other, avoiding the limitations of single-modal information, and comprehensively exploring the deep-level responses of the brain under the stimulation of complex cognitive tasks in flight training. For example, when the whole-brain signal shows an increase in the activity of a certain frequency band, and at the same time the eye movement data shows an extended fixation duration in a specific area and a decrease in the answering accuracy, it can be comprehensively judged that the pilot may face an overload of cognitive load in this task link, and the brain is concentrating resources to cope with difficulties.

[0062] Based on the excellent processing ability of LSTM for time series data and the real-time data collection and analysis process, the system can dynamically track the cognitive changes of pilots throughout the simulation training. Whether it is the cognitive shock caused by sudden situations in an instant or the progressive cognitive load adjustment caused by the increasing difficulty of training tasks, they can all be captured in a timely manner and reflected in the continuously updated evaluation results.

[0063] From the frequency band characteristics of whole-brain signals, the statistical analysis of eye movement patterns to the quantitative indicators of answering data, the fine feature extraction provides rich materials for accurate evaluation. The slight changes in the power spectral density of each frequency band, the capture of detailed eye movement features, and the accurate quantification of answering performance have greatly improved the sensitivity of the system to cognitive load changes. For example, a slight increase in the eye movement scanning frequency, combined with an increase in the power spectral density of the Beta wave in the whole-brain signal and a slightly extended answering time, can be keenly detected, indicating that the cognitive load may start to climb.

[0064] The CNN-LSTM composite model trained with a large amount of historical data has powerful discriminant ability. It can accurately distinguish different levels of cognitive load states, whether it is the low load in the normal cruise state, the high load when dealing with complex weather or emergencies, or the transitional state between the two, and can give accurate positioning. This accurate reflection not only provides real-time feedback for the pilots themselves to help them adjust their states in a timely manner, but also provides a scientific basis for flight training instructors to optimize the training plan and ensure flight safety. Brief Description of the Drawings

[0065] Figure 1 It is the front view / cross-sectional view of the first embodiment of the multi-modal electroencephalogram feature cognitive ability evaluation system based on neural network. Detailed Description of the Invention

[0066] The following is a further detailed description through specific embodiments:

[0067] The multi-modal electroencephalogram feature cognitive ability evaluation system based on neural network includes:

[0068] An acquisition module, used to simultaneously acquire whole-brain signals and eye movement data during pilot simulation training;

[0069] The data preprocessing module is used to perform preliminary processing on the received whole-brain signals and eye movement data, and use the preset monitoring mechanism to monitor the preliminarily processed data in real time. If the whole-brain signals or eye movement data show a preset characteristic fluctuation, it triggers the generation of an interactive test start signal;

[0070] The test interaction submodule is used to generate arithmetic test questions according to preset rules after receiving the interactive test start signal, and synchronously record the pilot's answer correlation parameters, and then summarize and integrate the answer correlation parameters with the pre-processed whole brain signal and eye movement data into the test data signal;

[0071] The feature extraction module is used to extract the power spectrum density features of different frequency bands from the whole brain signal using the time-frequency analysis method; identify different eye movement patterns through the clustering algorithm and extract the statistical results of each pattern to obtain the visual attention allocation characteristics; perform feature analysis on the answer correlation parameters to obtain the answer response characteristics;

[0072] The neural network model module is connected to the feature extraction module. It uses a combination of long short-term memory network (LSTM) and convolutional neural network (CNN). The power spectrum density features, visual attention allocation features, and answer response features are first input into CNN for spatial feature extraction. The output of CNN is used as the input of LSTM, and finally the evaluation result is output by LSTM.

[0073] Among them, LSTM pre-learns the dependencies and changing trends of whole-brain signals and eye movement data in the time dimension, maps the output of LSTM to the results of cognitive ability assessment through the fully connected layer, and pre-trains CNN and LSTM using a large amount of historical flight training data with real cognitive assessment labels, and continuously adjusts the weights and biases of CNN and LSTM through the back-propagation algorithm.

[0074] When used specifically, the whole brain signal is a multi-channel EEG signal collected by a 10-10 standard lead system. The preliminary processing is: for the whole brain signal, a linear interpolation method is used to process some missing values. Suppose a channel has a missing value at time t, and its signal value at the previous time t-1 is xt-1, and its signal value at the next time t+1 is xt+1, then the formula Fill in missing values; for outliers, identify and process them based on the statistical 3σ principle. First, calculate the mean μ and standard deviation σ of the whole brain signal of the channel. When the signal value x at a certain moment satisfies |x-μ|>3σ, it is judged as an outlier. At this time, the outlier is replaced by the median of the signal values ​​of the channel before and after k moments.

[0075] The preliminary processing is as follows: for the eye movement data, the mean filtering method is used to remove the noise interference. Let the eye movement data sequence be yi, where i = 1, 2, …, n, and the size of the filtering window be m. Then, the filtered eye movement data yi′ satisfies: where the range of values of i is to to ensure that the filtering window can completely cover the sequence and does not exceed the boundary.

[0076] The preset monitoring mechanism is implemented based on the sliding window algorithm and the principle of statistical hypothesis testing. A sliding time window with a length of T is set, which slides along the time axis at a fixed time interval Δt, and the whole-brain signal or eye movement data within the window is collected and analyzed in real time; for the whole-brain signal, it is decomposed into Delta, Theta, Alpha, Beta, and Gamma frequency bands, and the mean value μ i and standard deviation σ i of the power spectral density in each frequency band within the window are calculated, where i represents different frequency bands. The one-sample t-test is used, and the significance level is set as α. When is satisfied, it is determined that a characteristic fluctuation occurs in this frequency band, where μ 0i is the mean value of the power spectral density pre-collected and statistically obtained when the pilot is in a normal cognitive state, n is the number of data points within the window, and t α / 2，n-1 is the two-sided t-quantile corresponding to the α level with a degree of freedom of n - 1;

[0077] For the eye movement data, the eye movement data includes the fixation duration and the saccade count. The mean values μ g and standard deviation σ g of the fixation duration and the saccade count within the window are calculated respectively. The one-sample t-test is also used. When is satisfied, it is determined that a characteristic fluctuation occurs in the eye movement data, and μ 0g is the mean value of the corresponding eye movement index in the normal cognitive state; when it is detected that either the whole-brain signal or the eye movement data satisfies the above characteristic fluctuation condition, the interactive test start signal is triggered.

[0078] The change rate R f of the power spectral density of each frequency band of the whole-brain signal is monitored. The calculation formula for the change rate of the power spectral density is: where P f (t) is the power spectral density of a certain frequency band f at time t, and P f (t - Δt) is the power spectral density of this frequency band at time t - Δt; according to the magnitude of the change rate of the power spectral density, the characteristic fluctuations of the whole-brain signal are divided into level L1, level L2, and level L3; when ∣Rf∣ ≤ 0.1, it is level L1; when 0.1 < ∣Rf∣ ≤ 0.3, it is level L2; when ∣Rf∣ > 0.3, it is level L3;

[0079] The eye movement data also includes the pupil area. Taking the gaze duration T, the saccade count S, and the pupil area A as input parameters, let T(t), S(t), and A(t) be the gaze duration, saccade count, and pupil area at time t respectively. Then T(t - Δt), S(t - Δt), and A(t - Δt) are the corresponding index values at time t - Δt. Then, calculate the change rates of each index respectively:

[0080] Change rate of gaze duration:

[0081] Change rate of saccade count:

[0082] Change rate of pupil area:

[0083] Then, the comprehensive change rate R c The calculation formula is: R c = w T R T + w S R S + w A R A ; where, w T , w S , w A are the weight coefficients of the change rates of gaze duration, saccade count, and pupil area respectively, and satisfy w T + w S + w A = 1; The weight coefficients can be obtained through training with a large amount of experimental data and multiple linear regression; According to the magnitude of the comprehensive change rate R c , the characteristic fluctuations of eye movement data are divided into level E1, level E2, and level E3: When |R c | ≤ 0.15, it is level E1; When 0.15 < |R c | ≤ 0.35, it is level E2; When |R c | > 0.35, it is level E3;

[0084] According to the difficulty of arithmetic test questions, they are divided into easy-difficulty questions D1, medium-difficulty questions D2, and high-difficulty questions D3;

[0085] Establish a mapping matrix M to represent the correlation between the whole-brain signal and the characteristic fluctuation level of eye movement data and the difficulty level of arithmetic test questions:

[0086]

[0087] Among them, the rows represent the characteristic fluctuation level L of the whole-brain signal i(i = 1, 2, 3), the column represents the iconic fluctuation level E of eye movement data j (j = 1, 2, 3), the matrix element m ij represents the corresponding difficulty level of the arithmetic test questions;

[0088] When i = 1 and j = 1, m 11 = D1, that is, when both the whole-brain signal and eye movement data are mildly fluctuating, select the simple difficulty question D1;

[0089] When i = 1 and j = 2, or i = 2 and j = 1, randomly select the simple difficulty question D1 or the medium difficulty question D2 with a preset probability;

[0090] When i = 2 and j = 2, that is, when both the whole-brain signal and eye movement data are moderately fluctuating, select the medium difficulty question D2;

[0091] When i = 2 and j = 3, or i = 3 and j = 2, randomly select the medium difficulty question D2 or the high difficulty question D3 with a preset probability;

[0092] When i = 3 and j = 3, that is, when both the whole-brain signal and eye movement data are severely fluctuating, select the high difficulty question D3.

[0093] The feature extraction module extracts features from the whole-brain signal by using a method based on wavelet packet transform combined with adaptive threshold screening; First, perform N-layer wavelet packet decomposition on the collected whole-brain signal x(t) to obtain a series of wavelet packet coefficients c i，j , where i represents the decomposition layer number, and j represents the sub-band number in this layer;

[0094] For each sub-band, calculate its energy E i,j , and the calculation formula is:

[0095]

[0096] where k is the sample point index; Then, set the adaptive threshold T i，j , and determine it through the following formula:

[0097] T i，j = μ i，j + α·σ i，j

[0098] where μ i,j is the mean value of the sub-band E i,j , σ i,j is the standard deviation, and α is an adjustable threshold coefficient, and its value range is optimized according to experimental data;

[0099] Only retain the energy greater than the threshold T i，jThe wavelet packet coefficients corresponding to the sub-bands are used to reconstruct the signal to obtain the optimized whole-brain signal \(x'(t)\); then, the power spectral density features PSD of different frequency bands are extracted from \(x'(t)\). f , using the Welch method, the signal is segmented and windowed. The window function is selected as the Hamming window, and the power spectral density is calculated by the fast Fourier transform:

[0100]

[0101] where \(U\) is the number of segments, \(N\) FFT is the number of FFT points, \(f\) s is the sampling frequency, \(x'\) u (n) is the \(u\)-th segment of the signal.

[0102] Based on the features extracted from the whole-brain signal, the feature extraction module extracts features from the eye movement data using a differential clustering algorithm; the frequency bands of the whole-brain signal are re-divided into a low-frequency band of 1 - 10 Hz, a mid-frequency band of 11 - 30 Hz, and a high-frequency band of 31 - 80 Hz, corresponding to different cognitive states: the low-frequency band is related to the maintenance of basic cognition, the mid-frequency band reflects the dynamic changes of cognitive load, and the high-frequency band is involved in the processing of complex cognitive tasks.

[0103] When the energy ratio \(P\) low of the low-frequency band of the whole-brain signal, that is, the ratio of the sum of the energies of each sub-band in the low-frequency band to the total energy, is greater than 60%, the K-means++ clustering algorithm is used for the eye movement data, and clustering is performed based on two indicators of gaze duration \(T\) and saccade count \(S\); let the cluster centers be \(\mu\) k (k = 1, 2,..., K), and the distance \(d\) i from the sample point \(x\) i,k to the cluster center is calculated using the Euclidean distance:

[0104]

[0105] By iteratively updating the cluster centers to minimize the sum of the distances from the sample points to their respective cluster centers, different eye movement pattern clusters are obtained; the statistical results of each pattern cluster are extracted: the average gaze duration average saccade count are used as visual attention allocation features to reflect the visual attention rules in the basic cognitive state.

[0106] When the energy ratio \(P\) mid of the mid-frequency band of the whole-brain signal is between 30% - 60%, the hierarchical clustering algorithm is used for the eye movement data, and the pupil area \(A\) index is added to construct the sample vector \(x\) i = [T i , S i , A i; The Euclidean distance is used to measure the distance between samples, and by continuously merging the clusters with the closest distance, a hierarchical clustering structure is formed. According to the given number of clusters or distance threshold, different eye movement patterns are divided, and the statistical features of each pattern are extracted: the proportion of pattern duration ρ m , that is, the ratio of the duration of a certain pattern to the total observation time, and the conversion frequency ω between different patterns mn , where m and n represent different patterns; it is used to characterize the visual attention transfer characteristics during the dynamic change of cognitive load.

[0107] When the proportion of the high-frequency band energy P of the whole-brain signal high is greater than 30%, the Gaussian mixture model GMM clustering algorithm is used to process the eye movement data. Considering the three indicators T, S, and A comprehensively, a multi-dimensional Gaussian distribution model is constructed; assuming there are M Gaussian components, the model parameters θ = {π m , μ m , ∑ m}(m = 1, 2,..., M), where π m is the mixing weight, μ m is the mean vector, and ∑ m is the covariance matrix. The optimal parameters are iteratively solved through the expectation-maximization EM algorithm, and the eye movement data is divided into patterns with different probability distributions, and the high-order statistical features of each pattern are extracted: the information entropy H m , which measures the uncertainty of the pattern, and the determinant value of the covariance matrix |∑ m |, which reflects the correlation strength between each indicator; this reflects the fine allocation and change characteristics of visual attention under complex cognitive tasks.

[0108] Specifically, when in use: Before the pilot conducts simulation training, a 64-channel (10-10) whole-brain signal acquisition device is worn for him. The electrodes of this device are accurately positioned at each key position on the pilot's scalp according to international standards, ensuring that the electrophysiological activity signals of the cerebral cortex, that is, the whole-brain signals, can be collected in all directions and with high resolution. At the same time, a high-precision eye tracker is arranged in the simulation training environment so that it can clearly and accurately capture the eye movement situation of the pilot.

[0109] During the simulation training process, the acquisition module synchronously collects the whole-brain signal and eye movement data in real time. The sampling frequency of the whole-brain signal acquisition is set to 1000Hz to ensure that the subtle changes in brain electrical activity can be captured. The eye movement data recorded by the eye tracker includes key indicators such as gaze duration, gaze count, saccade duration, saccade count, saccade angle, pupil area, and blink rate. The sampling frequency is also set to a relatively high 100Hz to ensure the timeliness and accuracy of the data.

[0110] For the collected whole-brain signals, first, a band-pass filtering algorithm is adopted, with the passband range set to 0.5 Hz - 100 Hz, to remove low-frequency noise (such as power interference) and high-frequency noise (such as artifacts generated by muscle activity) in the environment.

[0111] Then, the adaptive threshold adjustment technology is applied to dynamically adjust the amplitude of the signals according to the statistical characteristics of the whole-brain signals, enhancing the clarity of the signals. For example, calculate the mean and standard deviation of the signals, and appropriately scale the part of the signal amplitude that exceeds the mean ± 3 times the standard deviation.

[0112] For the eye movement data, a median filtering algorithm is used for smoothing to correct the errors caused by small head movements. The filter window size is set to 5 data points, that is, the median of the 2 data points before and after the current data point is taken as the corrected value of the current data point.

[0113] The data preprocessing module has a built-in intelligent monitoring mechanism to monitor the preliminarily processed data in real-time. For the whole-brain signals, a sliding window algorithm is adopted, with the window length set to 1 second and the window sliding at a time interval of 0.1 second. Within each window, calculate the mean and standard deviation of the power spectral density of each frequency band (Delta: 1 - 3 Hz, Theta: 4 - 7 Hz, Alpha: 8 - 12 Hz, Beta: 13 - 30 Hz, Gamma: 31 - 80 Hz). When the mean of the power spectral density of a certain frequency band exceeds ± 2 times the standard deviation of its normal range (obtained through statistics of a large amount of previous experimental data), it is determined that a characteristic fluctuation occurs in that frequency band.

[0114] For the eye movement data, the sliding window algorithm is also adopted, with the window length of 2 seconds and the sliding interval of 0.2 second. Within each window, calculate the mean and standard deviation of key indicators such as the fixation duration and saccade count. When the mean of a certain indicator exceeds ± 2 times the standard deviation of its normal range, it is determined that a characteristic fluctuation occurs in the eye movement data. Once a preset characteristic fluctuation occurs in the whole-brain signals or the eye movement data, an interactive test start signal is immediately triggered.

[0115] When the interactive test start signal is received, the test interaction sub-module generates arithmetic test questions according to the preset rules. The preset rules are classified according to the characteristic fluctuation levels of the whole-brain signals and the eye movement data. For example, when both the whole-brain signals and the eye movement data are mildly fluctuating, arithmetic test questions of simple difficulty are generated, such as one-digit or two-digit addition, subtraction, multiplication, and division operations (2 + 3, 15 - 7, etc.); when both are moderately fluctuating, questions of medium difficulty are generated, including four arithmetic operations of multi-digit numbers ((23 + 17) × 4 ÷ 5, etc.); when both are severely fluctuating, questions of high difficulty are generated, involving operations of decimals, fractions, and complex operation sequences (3.5 × 2 / 7 + 1.2 ÷ 0.3, etc.).

[0116] The arithmetic test questions generated will be presented to the pilot through the human-machine interaction interface of the simulation training system, and the timing function will be started synchronously. Parameters such as the pilot's answer to the questions and the time taken to answer the questions will be recorded. After the answering is completed, the answer relevance parameters will be aggregated and integrated with the preprocessed whole-brain signals and eye movement data into test data signals and stored in the system database for subsequent analysis.

[0117] Wavelet transform is used for time-frequency analysis of the whole-brain signals, and the signals are decomposed into different frequency bands. The Daubechies wavelet basis is selected, and the decomposition level is set to 5 layers.

[0118] Calculate the power spectral density characteristics of each frequency band. Using the Welch method, the signal is segmented, with each segment having a length of 1 second and an overlap rate of 50%, and processed with a Hamming window. The power spectral density is calculated through the fast Fourier transform (FFT) to obtain the power spectral density values of each frequency band of Delta, Theta, Alpha, Beta, and Gamma waves.

[0119] The K-means clustering algorithm is used to process the eye movement data, and clustering is performed according to indicators such as gaze duration, saccade count, and pupil area. The K-means++ algorithm is used to select the initial clustering centers, and the number of clusters is set to 3 - 5 categories according to the actual situation.

[0120] Extract the statistical results of each clustering pattern, such as average gaze duration, average saccade count, average pupil area, etc., as visual attention allocation features. Perform feature analysis on the answer relevance parameters, and calculate features such as answer accuracy rate, average answering time, and standard deviation of answering time as answer reaction features.

[0121] An architecture combining a long short-term memory network (LSTM) and a convolutional neural network (CNN) is adopted. In the CNN part, 3 convolutional layers are set, with the convolutional kernel sizes being 3×3, 5×5, and 7×7 respectively, and the number of convolutional kernels being 16, 32, and 64 respectively. The ReLU function is used as the activation function. After each convolutional layer, a max-pooling layer is connected, and the pooling window size is 2×2.

[0122] In the LSTM part, 2 layers of LSTM units are set, with 128 units in each layer. The output of the CNN is used as the input of the LSTM, and finally, the output of the LSTM is mapped to the result of cognitive ability assessment through a fully connected layer. The number of output nodes of the fully connected layer is 1 (representing the cognitive ability assessment score).

[0123] Pre-collect a large amount of historical flight training data with real cognitive assessment labels, and divide it into a training set, a validation set, and a test set according to a ratio of 7:2:1. Use the training set to train the CNN and LSTM, adopt the Stochastic Gradient Descent (SGD) optimization algorithm, set the learning rate to 0.001, and the momentum to 0.9. During the training process, after every 10 batches of training, use the validation set to evaluate the model, and adjust the hyperparameters of the model, such as the learning rate, the number of convolutional kernels, etc., according to the loss function value of the validation set. After multiple iterations of training until the loss function value of the model converges on the validation set, a trained neural network model is obtained.

[0124] The trained neural network model evaluates the currently collected test data signal and outputs the cognitive ability evaluation score of the pilot. The evaluation result output module displays the evaluation score in an intuitive chart form, such as a line chart showing the change trend of cognitive ability in different training stages, and a bar chart comparing the scores of different cognitive ability indicators. At the same time, the evaluation result is output in a standardized data format (such as CSV format), which is convenient for integration with the flight training management system, providing a decision-making basis for training personnel and pilots to adjust the training strategy in a timely manner.

[0125] The above are only embodiments of the present invention. The present invention is not limited to the fields involved in this embodiment. Common knowledge such as the specific structures and characteristics known in the art are not described in detail here. Those of ordinary skill in the art know all the common technical knowledge in the technical field to which the invention belongs before the application date or the priority date, can know all the existing technologies in this field, and have the ability to apply the conventional experimental means before this date. Those of ordinary skill in the art can, under the inspiration given in this application, combine their own abilities to improve and implement this solution. Some typical well-known structures or well-known methods should not become an obstacle for those of ordinary skill in the art to implement this application. It should be noted that for those skilled in the art, without departing from the structure of the present invention, several deformations and improvements can still be made, and these should also be regarded as the protection scope of the present invention, and these will not affect the implementation effect of the present invention and the practicality of the patent. The protection scope required by this application should be subject to the content of its claims, and the specific implementation manners described in the specification can be used to interpret the content of the claims.

Claims

1. A multi-modal electroencephalogram feature cognitive ability evaluation system based on a neural network, characterized in that, include: An acquisition module, used to simultaneously acquire whole-brain signals and eye movement data during pilot simulation training; The data preprocessing module is used to perform preliminary processing on the received whole-brain signals and eye movement data, and use the preset monitoring mechanism to monitor the preliminarily processed data in real time. If the whole-brain signals or eye movement data show a preset characteristic fluctuation, it triggers the generation of an interactive test start signal; The test interaction submodule is used to generate arithmetic test questions according to preset rules after receiving the interactive test start signal, and synchronously record the pilot's answer correlation parameters, and then summarize and integrate the answer correlation parameters with the pre-processed whole brain signal and eye movement data into the test data signal; The feature extraction module is used to extract the power spectrum density features of different frequency bands from the whole brain signal using the time-frequency analysis method; identify different eye movement patterns through the clustering algorithm and extract the statistical results of each pattern to obtain the visual attention allocation characteristics; perform feature analysis on the answer correlation parameters to obtain the answer response characteristics; The neural network model module is connected to the feature extraction module. It uses a combination of long short-term memory network (LSTM) and convolutional neural network (CNN). The power spectrum density features, visual attention allocation features, and answer response features are first input into CNN for spatial feature extraction. The output of CNN is used as the input of LSTM, and finally the evaluation result is output by LSTM. Among them, LSTM pre-learns the dependencies and changing trends of whole-brain signals and eye movement data in the time dimension, maps the output of LSTM to the results of cognitive ability assessment through the fully connected layer, and pre-trains CNN and LSTM using a large amount of historical flight training data with real cognitive assessment labels, and continuously adjusts the weights and biases of CNN and LSTM through the back-propagation algorithm.

2. The multi-modal EEG feature cognitive ability evaluation system based on a neural network according to claim 1, wherein The whole-brain signal is a multi-channel electroencephalogram signal collected by a 10-10 standard lead system. The preliminary processing is as follows: for the whole-brain signal, the linear interpolation method is used to process some missing values. Suppose a missing value appears at time t in a certain channel, and the signal value at the previous adjacent time t-1 is x t -1, and the signal value at the next adjacent time t+1 is x t+1 , then the missing value is filled through the formula ; for outliers, they are identified and processed based on the 3σ principle of statistics. First, calculate the mean μ and standard deviation σ of the whole-brain signal of this channel. When the signal value x at a certain moment satisfies ∣x-μ∣>3σ, it is determined as an outlier. At this time, the outlier is replaced with the median of the signal values of the k moments before and after it in its channel.

3. The multi-modal EEG feature cognitive ability evaluation system based on a neural network according to claim 2, characterized in that, The preliminary processing is as follows: for the eye movement data, the mean filtering method is used to remove noise interference. Let the eye movement data sequence be y i If the size of the filtering window is m, then the eye movement data y i ' after filtering satisfies: where the value range of i is to This ensures that the filtering window can completely cover the sequence without exceeding the boundary.

4. The multi-modal EEG feature cognitive ability evaluation system based on a neural network according to claim 3, characterized in that, The preset monitoring mechanism is implemented based on the sliding window algorithm and the principle of statistical hypothesis testing. A sliding time window with a length of T is set and slides along the time axis at a fixed time interval Δt to collect and analyze the whole-brain signals or eye movement data in the window in real time. For whole-brain signals, they are decomposed into Delta, Theta, Alpha, Beta, and Gamma frequency bands, and the mean value μ of the power spectral density of each frequency band in the window is calculated. i and the standard deviation σ i , where i represents different frequency bands. A one-sample t-test is used, and the significance level is set as α. When is satisfied, it is determined that a signature fluctuation appears in this frequency band, where μ 0i is the mean value of the power spectral density pre-collected and statistically obtained when the pilot is in a normal cognitive state for this frequency band, n is the number of data points in the window, and t α / 2，n-1 is the two-sided t-quantile corresponding to the α level with a degree of freedom of n - 1. For eye movement data, the eye movement data includes the fixation duration and the number of saccades. Calculate the mean μ of the fixation duration and the number of saccades within the window respectively g and the standard deviation σ g . Also use the one-sample t-test. When , it is determined that the eye movement data has a signature fluctuation, and μ 0g is the mean of the corresponding eye movement index in the normal cognitive state; when it is detected that either the whole-brain signal or the eye movement data meets the above signature fluctuation condition, the interactive test start signal is triggered 5. The multimodal EEG feature cognitive ability evaluation system based on a neural network according to claim 4, wherein The rate of change R of the power spectral density of each frequency band of the whole-brain signal f is monitored. The calculation formula for the rate of change of the power spectral density is as follows: where P f (t) is the power spectral density of a certain frequency band f at time t, and P f (t - Δt) is the power spectral density of this frequency band at time t - Δt; according to the magnitude of the rate of change of the power spectral density, the characteristic fluctuations of the whole-brain signal are divided into level L1, level L2, and level L3; when |Rf| ≤ 0.1, it is level L1; when 0.1 < |Rf| ≤ 0.3, it is level L2; when |Rf| > 0.3, it is level L3; Eye movement data also includes pupil area, with gaze duration T, saccade number S, and pupil area A as input parameters. Assume that T(t), S(t), and A(t) are gaze duration, saccade number, and pupil area at time t, respectively. Then T(t-Δt), S(t-Δt), and A(t-Δt) are the corresponding index values ​​at time t-Δt, respectively. Then calculate the change rate of each index respectively: Gaze duration change rate: Saccade rate of change: Pupil area change rate: Then, the comprehensive change rate R c is calculated by the formula: R c = w T R T + w S R S + w A R A ; where, w T 、w S 、w A are the weight coefficients of the fixation duration, saccade count, and pupil area change rate respectively, and satisfy w T + w S + w A = 1; the weight coefficients can be obtained through training with a large amount of experimental data and multiple linear regression; according to the magnitude of the comprehensive change rate R c , the characteristic fluctuations of eye movement data are divided into levels E1, E2, and E3: when |R c | ≤ 0.15, it is level E1; when 0.15 < |R c | ≤ 0.35, it is level E2; when |R c | > 0.35, it is level E3; According to the difficulty of arithmetic test questions, they are divided into simple difficulty questions D1, medium difficulty questions D2 and high difficulty questions D3; A mapping matrix M is established to represent the correlation between the level of characteristic fluctuations of whole-brain signals and eye movement data and the level of difficulty of arithmetic test questions: Among them, the rows represent the whole-brain signal iconic fluctuation level L i (i = 1, 2, 3), and the columns represent the eye movement data iconic fluctuation level E j (j = 1, 2, 3), and the matrix element m ij represents the corresponding arithmetic test question difficulty level; When i = 1 and j = 1, m 11 = D1, that is, when both the whole-brain signal and the eye movement data show mild fluctuations, the simple-difficulty question D1 is selected; When i=1 and j=2, or i=2 and j=1, a simple difficulty question D1 or a medium difficulty question D2 is randomly selected with a preset probability; When i = 2 and j = 2, that is, when both the whole brain signal and the eye movement data fluctuate moderately, select the medium difficulty question D2; When i=2 and j=3, or i=3 and j=2, a medium-difficulty question D2 or a high-difficulty question D3 is randomly selected with a preset probability; When i=3 and j=3, that is, when both the whole brain signal and the eye movement data fluctuate severely, the high-difficulty question D3 is selected.

6. The multi-modal EEG feature cognitive ability evaluation system based on a neural network according to claim 5, characterized in that, The feature extraction module extracts features from the whole-brain signal by using a method based on wavelet packet transform combined with adaptive threshold screening; first, the collected whole-brain signal x(t) is decomposed by N-layer wavelet packet transform to obtain a series of wavelet packet coefficients c i,j , where i represents the decomposition layer number and j represents the sub-band serial number at this layer; For each sub-band, calculate its energy E i，j , and the calculation formula is: where k is the sample point index; then, set the adaptive threshold T i,j , which is determined by the following formula: T i,j = μ i，j + α·σ i，j where μ i,j is the mean value of sub - band E i,j , σ i，j is the standard deviation, α is an adjustable threshold coefficient, and its value range is optimized according to experimental data; Only retain the wavelet packet coefficients corresponding to the sub-bands with energy greater than the threshold T i,j to reconstruct the signal and obtain the optimized whole-brain signal x′(t); then, extract the power spectral density features PSD of different frequency bands from x′(t) f , using the Welch method, segment and window the signal, select the Hamming window as the window function, and calculate the power spectral density through the fast Fourier transform: where U is the number of segments, N FFT is the number of FFT points, f s is the sampling frequency, x′ u (n) is the signal of the u-th segment. This method can more precisely focus on the frequency bands containing effective cognitive information, remove the noise interference frequency bands, and the extracted whole-brain signal features are more distinguishable.

7. The multimodal EEG feature cognitive ability evaluation system based on a neural network according to claim 6, wherein Based on the features extracted from the whole-brain signal, the feature extraction module uses a differential clustering algorithm to extract features from the eye movement data; the frequency bands of the whole-brain signal are re-divided into a low-frequency band of 1-10 Hz, a middle-frequency band of 11-30 Hz, and a high-frequency band of 31-80 Hz, which respectively correspond to different cognitive states: the low-frequency band is related to the maintenance of basic cognition, the middle-frequency band reflects the dynamic changes of cognitive load, and the high-frequency band is involved in the processing of complex cognitive tasks.

8. The multi-modal EEG feature cognitive ability evaluation system based on a neural network according to claim 7, wherein When the proportion of the energy of the low-frequency band in the whole-brain signal P low , that is, the ratio of the sum of the energies of each sub-band in the low-frequency band to the total energy, is greater than 60%, the K-means++ clustering algorithm is used for the eye movement data, and clustering is performed based on two indicators: the gaze duration T and the saccade count S; let the cluster center be μ k (k = 1, 2,..., K, the distance d i from the sample point x i,k to the cluster center is calculated using the Euclidean distance: By iteratively updating the cluster centers to minimize the sum of the distances from the sample points to their respective cluster centers, different clusters of eye movement patterns are obtained; extract the statistical results of each pattern cluster: the average gaze duration of each cluster Average saccade count As visual attention allocation features, they reflect the visual attention patterns in the basic cognitive state.

9. The multi-modal EEG feature cognitive ability evaluation system based on a neural network according to claim 8, characterized in that, When the proportion P of the mid-frequency energy in the whole-brain signal mid is between 30% and 60%, apply the hierarchical clustering algorithm to the eye movement data, add the pupil area A index, and construct the sample vector x i = [T i , S i , A i ; Use the Euclidean distance to measure the distance between samples, and form a hierarchical clustering structure by continuously merging the closest clusters; According to the given number of clusters or distance threshold, divide different eye movement patterns, and extract the statistical features of each pattern: the pattern duration ratio ρ m , that is, the ratio of the duration of a certain pattern to the total observation time, the conversion frequency ω mn between different patterns, where m and n represent different patterns; It is used to characterize the visual attention transfer characteristics during the dynamic change of cognitive load.

10. The multi-modal EEG feature cognitive ability evaluation system based on a neural network according to claim 9, wherein, When the high-frequency band energy ratio P of the whole-brain signal high is greater than 30%, the Gaussian mixture model GMM clustering algorithm is used to process the eye movement data. Considering the three indicators T, S, and A comprehensively, a multi-dimensional Gaussian distribution model is constructed. Assuming there are M Gaussian components, the model parameters θ = {π m , μ m , ∑ m} (m = 1, 2,..., M), where π m is the mixing weight, μ m is the mean vector, and ∑ m is the covariance matrix. The optimal parameters are iteratively solved by the expectation maximization EM algorithm, and the eye movement data is divided into patterns with different probability distributions, and the high-order statistical features of each pattern are extracted: the information entropy H m , which measures the pattern uncertainty, and the determinant value of the covariance matrix |∑ m |, which reflects the correlation strength between each indicator; this reflects the fine allocation and change characteristics of visual attention under complex cognitive tasks.

Images