A cognitive load evaluation method based on brain intelligence technology

Abstract

The application relates to the field of cognitive neuroscience and discloses a cognitive load evaluation method based on brain intelligence technology, which comprises the following steps: S1. 4-channel electroencephalogram (EEG) signal and electrooculogram (EOG) signal acquisition; S2. EEG signal and EOG signal preprocessing and feature extraction; S3. A variational autoencoding generative adversarial enhancement method for the EEG signal can expand and enrich an original data set by generating synthetic data, thereby improving the performance of a training model; S4. A dual-mode feature fusion model for the EEG and EOG based on a denoising autoencoder can obtain complementary features, enriches a feature space and is helpful to more accurately describe the cognitive state of an individual; and S5. An improved cognitive load classification method of a support vector machine (SVM) optimized by a seagull algorithm can better capture cognitive load-related features, improve the accuracy and robustness of a classifier, and the optimized model can more accurately judge the cognitive load level of an individual, thereby providing a reliable basis for cognitive load evaluation.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of cognitive neuroscience, and particularly relates to a cognitive load evaluation method based on brain intelligence technology. BACKGROUND

[0002] With the advancement of digitization process, many traditional manual operations are gradually replaced by computer automation. In large-scale and wide-range automated operation processes, the tasks that people are responsible for are gradually changed from manual operation to monitoring and decision-making. The cognitive load level of people in the process of decision-making is an important influencing factor.

[0003] The commonly used cognitive load evaluation method at present is mainly based on subjectivity. However, whether it is a scale method or a task method, it is necessary to measure before and after the operation, which does not have real-time performance and is affected by subjective factors, and has great limitations. Cognitive load evaluation based on physiological signals can collect the physiological signals of the operator during the operation to evaluate the cognitive load, which has better objectivity and can avoid the influence of some subjective factors. This can make the application range of cognitive load more universal. Therefore, cognitive load evaluation based on physiological signals has gradually become the focus of research.

[0004] However, the existing cognitive load evaluation has the following problems:

[0005] (1) Cognitive load-related EEG data is less: Compared with image processing or other fields, it is more difficult to obtain EEG data. On the one hand, EEG acquisition requires more professional acquisition equipment and more complex acquisition environment. On the other hand, EEG-related research requires higher requirements for subjects. These difficulties have caused the problem of less EEG-related data. A common method is to generate new data through some data enhancement algorithm, and try to make the difference between the generated data and the original data as small as possible, especially in the characteristics that are highly related to the problem. Common data enhancement methods generally include traditional methods and machine learning-based methods. Traditional data enhancement mainly includes flipping, cutting and adding noise, etc. Unlike the wide application of traditional data enhancement in the field of image processing, traditional data enhancement methods are not suitable for EEG signals, because EEG signals are generally time series that cannot be translated or rotated, and due to the relative weakness of EEG signals, it is difficult to add appropriate noise for data generation. Miri (Yanshan University, 2022) selected a data enhancement method based on generative adversarial network to expand the motor imagery EEG data of the Berlin team in Germany by 1.5 times the amount of real data, and to analyze and evaluate the mixed data. The artificial EEG data generated by the generative adversarial network improved the accuracy of motor intention analysis to some extent, and to some extent, it made up for the problem of insufficient EEG data. However, for EEG signals, the physiological mechanism of the generative adversarial network model for generating EEG signals is not clear enough. And the generative adversarial network has certain limitations. If not properly trained, the discriminator will be trained too well, causing the generator to have a gradient vanishing problem, and if the discriminator is trained too poorly, the generator will not be able to obtain information to adjust the generated data. In addition, the training process of the generative adversarial network cannot be like a general neural network model, which can determine the convergence state of the network through the value of the loss function.

[0006] (2) Limitations of single EEG signal in cognitive load evaluation: EEG records the overall performance of brain electrical activity on the scalp, and cannot provide accurate information about different regions in the brain. Therefore, it is difficult to determine whether a specific brain region is processing a specific cognitive task, and it is easily affected by many external interference factors, such as eye movement, muscle activity, electromagnetic interference, etc., which may mask or confuse the signals related to cognitive load. In some cases, the EEG signal may be disturbed by multiple tasks at the same time when performing a cognitive task, which makes the use of single EEG signal to evaluate and classify cognitive load more complex. Xiong Ronglong (Southwest University, 2021) proposed a method of combining ECG and EEG to evaluate cognitive load, and constructed a cognitive load evaluation model of dual modal ECG and EEG signals. In view of the cognitive load situation of classroom teaching and learning in the teaching process, the two states of resting baseline state and cognitive load state were classified by using the feature extraction and classification of two modalities in the research, which was used for the adjustment of teaching strategies. However, due to the relatively small number of samples and subjects, the lack of subject diversity limits the further exploration of the generalization performance of the model. In addition, the distinction effect of cognitive load matching / non-matching is still far from practical application, and the verification result of dual modal has no improvement compared with the use of HRV features only.

[0007] (3) Classifier selection problem of cognitive load evaluation: The performance of the classifier depends on the selected features. For electroencephalogram and electrooculogram signals, the selection of features is complex, and there are many possible choices. Different feature selection may lead to differences in the performance of the classifier. If the classifier is over-fitted on the training data, it may lead to poor generalization performance on new data. Some classifiers may have strong prediction performance, but it may be difficult to explain how the model makes decisions. In some application scenarios, the interpretability of the model is also an important consideration. Therefore, the selection of the classifier is a very important problem for the evaluation of cognitive load. Tan Tingyi (Chongqing University of Posts and Telecommunications, 2021) proposed a cognitive load evaluation method based on electroencephalogram signals, which uses the characteristics of the covariance matrix to analyze the electroencephalogram data related to cognitive load in time and frequency, and then uses a neural network to implement end-to-end cognitive load detection. Finally, the convolutional neural network is used to classify the cognitive load. The proposed transfer learning model effectively alleviates the transfer learning problem between subjects, but there is still a significant distribution difference between subjects. Chakladar D D et al. (Biomedical Signal Processing and Control, 2020) proposed a deep hybrid model based on bidirectional long short-term memory and long short-term memory for the scenario of multiple tasks operating simultaneously, which divides the cognitive load level into low, medium and high. The study mainly uses the STEW dataset, and uses the grey wolf optimization algorithm for feature selection in the final classification. Finally, six features required by the study are selected after screening to divide the cognitive load state into two parts. SUMMARY

[0008] In view of the above problems existing in the prior art, the purpose of the present application is to provide a cognitive load evaluation method based on brain intelligence technology.

[0009] To achieve the above purpose, the present application adopts the following technical solutions:

[0010] A cognitive load evaluation method based on brain intelligence technology, comprising the following steps:

[0011] S1.4-channel electroencephalogram and electrooculogram signal acquisition;

[0012] S2. Preprocessing and feature extraction of the electroencephalogram and electrooculogram signals collected in step S1;

[0013] S3. Variational auto-encoding generation of electroencephalogram signals;

[0014] The feature of the electroencephalogram signal processed through step S2 is enhanced by a data enhancement algorithm WVAEGP algorithm; the WVAEGP algorithm comprises: adding a discriminator link in the VAE network; and using the Wasserstein distance and gradient penalty to solve the problems of gradient explosion and gradient disappearance;

[0015] S4. A dual-mode feature fusion model of electroencephalogram and electrooculogram based on a denoising autoencoder

[0016] The dual-mode feature fusion model of electroencephalogram and electrooculogram based on the denoising autoencoder comprises constructing an explicit layer fusion strategy or an implicit layer fusion strategy to judge the cognitive load, wherein the step of constructing the explicit layer fusion is as follows:

[0017] Step 41: merging the features of the electroencephalogram and electrooculogram signals to obtain fused features;

[0018] Step 42: adding noise to the fused features according to the idea of the denoising autoencoder to obtain a signal containing noise;

[0019] Step 43: encoding the signal containing noise to an implicit layer space by an encoder to generate implicit layer data;

[0020] Step 44: finally generating reconstructed data similar to the original noise-free data by a decoder;

[0021] In the training process, the parameters of the decoder and the decoder are constantly adjusted, and when the average reconstruction error between the fused features and the reconstructed data is the smallest, the implicit layer data containing the original electroencephalogram and electrooculogram signal features are taken as the input of a subsequent classifier;

[0022] When the implicit layer fusion strategy is adopted, the electroencephalogram and electrooculogram signal features are processed respectively, the implicit layer data corresponding to the electroencephalogram and electrooculogram signals are fused to form new fused implicit layer features, and the fused implicit layer data are taken as the input of a subsequent classifier; S5. Cognitive load classification method of improved albatross algorithm optimizing support vector machine

[0023] The steps of the improved albatross-support vector machine algorithm are as follows:

[0024] Step 51: generating an initial population and setting related parameters;

[0025] Step 52: calculating the fitness value of the initial albatross population, calculating the position of the optimal albatross individual, and sorting the albatross individuals according to the fitness value, and saving the optimal albatross position information in the whole space as

[0026] Step 53: adjusting the control factor using the nonlinear characteristics of the cosine function;

[0027]

[0028] Wherein c represents the control parameter in the sea gull algorithm needs to be adjusted, t represents the iteration number of the algorithm at present, MaxT is the maximum iteration number of the algorithm, c max And c min Is the maximum and minimum value of the parameter variation range to be adjusted;

[0029] Step 54: updating the sea gull position according to the historical evolution strategy;

[0030] Step 55: calculating the fitness value of the updated sea gull population, and recalculating the position of the best sea gull;

[0031] Step 56: judging whether the maximum iteration number is reached, yes, stopping iteration and outputting the optimal parameters C and g, otherwise, jumping to step 3 to continue iteration.

[0032] Preferably, in step S1, all electroencephalogram data and electrooculogram data of the user are collected by using the 14-channel electroencephalogram acquisition device Emotive.

[0033] Further, in step S2, in the preprocessing stage of the electroencephalogram signal, the fast Fourier transform method is used for band-pass filtering W to correct linear drift of the electroencephalogram data to be processed, and high-frequency environmental noise is removed to a certain extent, and low-frequency signals of 0.1-40 Hz are retained, and signals of 50 Hz are notch processed, and the slope is 40 dB / oct; in the feature extraction of the electroencephalogram signal, wavelet entropy, permutation entropy and differential entropy are selected as the features of the electroencephalogram signal.

[0034] Further, in step S2, in the preprocessing of the electrooculogram signal, the fast independent component analysis method is used to separate the principal components of the electrooculogram signal from the electrooculogram signal; in the feature extraction of the electrooculogram signal, the scan rate, scan rate variance, scan amplitude variance, scan amplitude power, scan times in the scan category, and the blink rate, amplitude variance, average of blink amplitude, power of blink amplitude, number of blinks, average of blink duration variance, blink duration, average of scan times in the blink category are selected as the features of the electrooculogram signal.

[0035] Further, in step S3, the WVAEGP algorithm process includes:

[0036] Input: training sample; maximum iteration number T;

[0037] Output: learned parameters θ E , θ G , θ Dis And θ gp Used for the encoder E, the generator G, the discriminator Dis,

[0038] Step 1: initialize parameters θ E , θ G , θ Dis and θ gp , set the iteration unit ITER = 1;

[0039] Step 2: judge whether ITER < T;

[0040] Step 3: randomly extract the batch training sample number and Gaussian value, noise Z ~ N(μ, σ);

[0041] Step 4: calculate the latent vector z = E(x real ), generate the feature x' real =G(z) and x gen =G(z p );

[0042] Step 5: calculate various losses L Rec , L KL , L G , L Dis , L gp ;

[0043] Step 6: use the ADAM optimizer to optimize parameters θ E , θ G , θ Dis and θ gp : θ E = arg min θ (L Rec +L KL +L G ); θ G = arg min θ (L Rec +L G ); θ Dis = arg max θ (L Dis ); θ gp = arg max θ (L gp ); wherein θ E represents the learning parameter of the original sample input into the hidden layer feature, θ G represents the learning parameter of the new output data generated from the hidden layer space data distribution, θ Dis represents the learning parameter between the original data input into the discriminator and the new data, and θ gp represents the learning parameter between the original data distribution and the generated output data distribution measured by the gradient penalty;

[0044] Step 7: ITER = ITER + 1;

[0045] Step 8: End.

[0046] Further, in step S4, the electroencephalogram feature processing step is as follows:

[0047] Step 1: Add noise to the electroencephalogram feature to obtain an electroencephalogram signal containing noise;

[0048] Step 2: Encode the electroencephalogram signal containing noise to the hidden layer space through the encoder to generate electroencephalogram hidden layer data;

[0049] Step 3: Generate electroencephalogram reconstruction data similar to the original noise-free data through the decoder;

[0050] During the training process, the parameters of the decoder and the encoder need to be adjusted, and when the average reconstruction error between the electroencephalogram feature and the electroencephalogram reconstruction data is the smallest, the data at this time is called electroencephalogram hidden layer processing data.

[0051] Further, in step S4, the electroencephalogram feature processing step is as follows:

[0052] Step 1: Add noise to the electroencephalogram feature to obtain an electroencephalogram signal containing noise;

[0053] Step 2: Encode the electroencephalogram signal containing noise to the hidden layer space through the encoder to generate electroencephalogram hidden layer data;

[0054] Step 3: Generate electroencephalogram reconstruction data similar to the original noise-free data through the decoder;

[0055] During the training process, the parameters of the decoder and the encoder need to be adjusted, and when the average reconstruction error between the electroencephalogram feature and the electroencephalogram reconstruction data is the smallest, the data at this time is called electroencephalogram hidden layer processing data.

[0056] Further, in step S5, the implementation step of the improved seagull algorithm includes:

[0057] Step 1: Initialize the population and set the control parameters related to the algorithm, and each seagull represents a solution;

[0058] Step 2: Calculate the fitness value of the initial seagull individual and find the optimal individual position;

[0059] Step 3: Use the cosine function nonlinear feature proposed in formula (7) to improve the control factor;

[0060]

[0061] The control factor of the seagull algorithm at the tth iteration is defined as f c (t), f max is the maximum value among them, and f minis the minimum value in it, and MaxT is the maximum iteration number of the algorithm;

[0062] Step 4: updating the gull position according to the historical evolution strategy of formula (8);

[0063] HS i (t+1)=r d ×HS i (t)+(1-r d )×P b,i -(1-r d )×HS w (t) (8)

[0064] Assume that the updated position of the i-th gull individual is HS i (t+1), the position of the gull individual before updating is defined as HS i (t), P b,i is the historical optimal position of the i-th gull individual, HS w (t) represents the global worst position of the gull population as a whole, and r d is a random number in [0.1];

[0065] Step 5: calculating the updated gull population fitness value, and recalculating the position of the optimal gull;

[0066] Step 6: judging the iteration number, if the maximum value is reached, stopping optimization and outputting the optimal value, otherwise continuing iteration.

[0067] Compared with the prior art, the present application has the following beneficial effects:

[0068] (1) The present application proposes a variational auto-encoding generative adversarial enhancement method based on electroencephalogram data: the variational auto-encoding generative adversarial enhancement method based on electroencephalogram data can expand and enrich the original data set by generating synthetic data, thereby improving the performance of the training model. The synthetic data has similar statistical characteristics as the real data, which can help the model to learn more comprehensive feature representation and enhance the generalization ability of the model to different samples.

[0069] (2) The application preprocesses the electrooculogram signal through principal component analysis, extracts and selects characteristics such as blink frequency, blink amplitude and blink power. A fusion model of electroencephalogram and electrooculogram signals is designed based on denoising autoencoder (DAE), which is a surface layer fusion and a hidden layer fusion. Electroencephalogram signals and electrooculogram signals represent different physiological processes and cognitive activities. Electroencephalogram signals reflect the electrical activity of the brain and can provide information about brain function activities; electrooculogram signals reflect eye movement and eye-related physiological processes and can provide information about attention and visual processing. By fusing the two signals, complementary features can be obtained, enriching the feature space and helping to more accurately describe the cognitive state of individuals.

[0070] (3) The application proposes a cognitive load classification method of optimizing support vector machine by improved seagull algorithm, which can better capture the characteristics related to cognitive load and improve the accuracy and robustness of the classifier. The optimized model can more accurately judge the cognitive load level of individuals and provide a reliable basis for cognitive load evaluation.

[0071] (4) The improved seagull algorithm proposed in the application overcomes the defects of single search and dependence on optimal individuals of the seagull algorithm, adopts a historical population evolution strategy to update the optimal seagull position, and introduces a cosine function to adjust the parameters by using its nonlinear characteristics, thereby greatly improving the search efficiency and optimization accuracy of the seagull algorithm. BRIEF DESCRIPTION OF DRAWINGS

[0072] Other features, objects and advantages of the application will become more apparent through reading the following detailed description of the non-limiting embodiments with reference to the accompanying drawings:

[0073] Figure 1 The scheme flowchart of Example 1 of the application;

[0074] Figure 2 The method structure framework diagram of the application;

[0075] Figure 3 The network structure diagram of the WVAEGP data generation model;

[0076] Figure 4 The surface layer fusion strategy and hidden layer fusion strategy framework;

[0077] Figure 5 The parameter change curve of the improved algorithm;

[0078] Figure 6 The flowchart of the improved seagull algorithm (HCSOA);

[0079] Figure 7 The flowchart of the improved HCSOA-SVM algorithm;

[0080] Figure 8 For the average accuracy comparison of several classifiers in the training sample in the ten-fold cross-validation method;

[0081] Figure 9 For the average accuracy comparison of several classifiers in the test sample in the ten-fold cross-validation method;

[0082] Figure 10 For the EEG data enhancement framework diagram;

[0083] Figure 11 For the comparison of real data and generated data wavelet time-frequency diagram;

[0084] Figure 12 For the accuracy comparison of each data enhancement method under different proportions;

[0085] Figure 13 For the accuracy comparison before and after fusion;

[0086] Figure 14 For the cognitive load classification effect comparison. DETAILED DESCRIPTION

[0087] The application will be described in detail below with specific embodiments. The following examples will help those skilled in the art to further understand the application, but do not limit the application in any form. It should be noted that for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made. These are within the scope of the present application.

[0088] Example 1

[0089] A cognitive load classification method based on data enhancement and dual-mode fusion uses Emotive devices to collect electroencephalogram (EEG) and electrooculogram (EOG). The corresponding electrodes are placed according to the size of the human brain and the electrode positions of the human brain specified by the International Federation of Societies for Electroencephalography and Clinical Neurophysiology (10-20 standard system). The Emotive device converts the brain wave signal into digital information, and after pairing with a desktop computer, it can send data to the computer program in a wireless Bluetooth manner. In the program, the data can be processed and analyzed to obtain the features required for cognitive load evaluation, and finally the cognitive load is evaluated. The brain and eye data are used as input to obtain feature data, and finally the purpose of evaluating cognitive load through the fusion of brain and eye signals is achieved. As shown in Figure 1 and Figure 2 The specific steps are as follows:

[0090] (1) 4-channel EEG signal and EOG signal acquisition

[0091] The collected data is mainly all the brain electrical data and the electrooculogram data of the user, and the electroencephalogram and electrooculogram signals are collected by using a 14-lead electroencephalogram collection device Emotive. The Emotive collection device has 14 electrodes, which correspond to different points of the human brain. The AF3, AF4, F7 and F8 electrodes are relatively close to the position of the eyes, and the signals collected by the four electrodes contain more EOG signals. The signals of the AF3, AF4, F7 and F8 electrodes are selected for electrooculogram cognitive load detection.

[0092] (2) Electroencephalogram signal and electrooculogram signal preprocessing and feature extraction

[0093] In the preprocessing stage of the electroencephalogram signal, the fast Fourier transform method is used for band-pass filtering of the electroencephalogram data to be processed, W correction of linear drift, and removal of environmental noise to a certain extent. The low-frequency signal of 0.1 to 40 Hz is retained, and the signal of 50 Hz is processed by wave filtering with a slope of 40 dB / oct.

[0094] Regarding the feature extraction of the electroencephalogram signal, wavelet entropy reflects the subtle changes in a small range of the signal by calculating at multiple scales, and permutation entropy can represent the information purity of the sample set and the complexity of the specific time series. Differential entropy is an important expression of Shannon information entropy, which can better understand and analyze the electroencephalogram signal. Therefore, wavelet entropy, permutation entropy and differential entropy are selected as the features of the electroencephalogram signal.

[0095] Regarding the electrooculogram signal preprocessing, the EOG signal may be disturbed by noise in the laboratory environment, which may come from the equipment inside the laboratory. In addition, since physiological signals usually do not exist alone, these electrooculogram signals may contain irrelevant artifacts. Therefore, the fast independent component analysis method is used to separate the principal components of the electrooculogram from the electrooculogram.

[0096] Regarding the feature extraction of the electrooculogram signal, by analyzing the blinking, saccade and fixation of the eyes, the corresponding amplitude, frequency and maximum / minimum value can be obtained, and based on CFS, a part of the features with high correlation with the judgment of cognitive load are selected. The results show that the saccade rate, saccade rate variance, saccade amplitude variance, saccade amplitude power, saccade frequency in the saccade category, the blink rate, amplitude variance, average blink amplitude, blink amplitude power, blink frequency, and the average of the average blink duration variance, blink duration, and the average of the saccade frequency are related to the judgment of cognitive load. Finally, the above features are selected as the electrooculogram signal features.

[0097] (3) Variational auto-encoding generative adversarial enhancement method of electroencephalogram signal

[0098] After the original electroencephalogram signal is collected, the original electroencephalogram signal is preprocessed and feature extraction is performed, and then the electroencephalogram features are data enhanced through a data enhancement algorithm. The operation is as follows:

[0099] WVAEGP: WVAE uses the Wasserstein distance as the loss function, and the Wasserstein distance gradually decreases during training, which can adjust the effect of the generator, and make the quality of the generated samples better than the original VAE. The Wasserstein distance is generally used to judge the similarity between two distributions. The formula of the Wasswerstein distance is as follows:

[0100]

[0101] ∏(p r ,p g ) represents the joint distribution of all possible γ(x,y), where x ~ p r , y ~ p g . γ(x,y) represents the unit distance moved to change the distribution p r to the distribution p g .

[0102] The Wasswerstein distance is difficult to calculate in practice, and can be transformed into the following formula:

[0103]

[0104] A set of functions f depending on the parameter w is defined, where f satisfies the K-Lipschitz condition, denoted as ||f||≤K, that is, |f(a)-f(b)|≤K|a-b|, K is a real number greater than zero, and the function f can be any function. The above can be approximated as the following formula:

[0105]

[0106] At this time, the function of the discriminator is no longer to separate true and false samples, but to learn a function that satisfies the K-Lipschitz condition to calculate the W distance. As the W distance gradually approaches during the training process, the generator can generate more real samples that satisfy the real sample distribution. In the above derivation, a necessary assumption is that the discriminator f w satisfies the K-Lipschitz condition, which can be realized by using the gradient penalty method. Gradient penalty is realized by adding a penalty term to the loss function of the discriminator. At this time, the discriminator satisfies the K-Lipschitz condition, and the loss function of the discriminator is:

[0107]

[0108] The penalty term sampling in the gradient penalty is just sampling in the real sample data distribution p r The generated sample data distribution p g Each of the two points is sampled once, and then a random sampling is performed between the two points After this operation, the penalty term sampling is obtained The value of the default hyperparameter λ in the entire sampling process is 10, represent The distribution of α is a random number in [0, 1], x is a random real sample, and z is noise input.

[0109] By changing the function of the W distance, the discriminator from distinguishing true and false samples to approximating the W distance. The generator continuously learns to reduce the W distance, effectively solving the gradient disappearance problem that may occur during the training of the generator when the training effect of the discriminator is too effective. And the W distance can effectively evaluate the distance between the true and false sample distributions. During the training process, the W distance is continuously reduced, which can also serve as a reference for the training process.

[0110] WVAEGP: (1) The discriminator is added to the VAE network, so that the VAE network feedback adjusts the relationship between the original data and the hidden layer space after encoding. (2) Use the Wasserstein distance and gradient penalty to solve the problems of gradient explosion and gradient disappearance. As shown in Figure 3 The network structure diagram of the WVAEGP data generation model is shown in

[0111] WVAEGP algorithm flow:

[0112] Input: training samples; maximum iteration number T;

[0113] Output: learned parameters θ E , θ G , θ Dis and θ gp for the encoder E, the generator G, the discriminator Dis,

[0114] Step 1: initialize parameters θ E , θ G , θ Dis and θ gp . Set the iteration unit ITER = 1;

[0115] Step 2: judge whether ITER < T;

[0116] Step 3: randomly extract the batch training sample number and Gaussian value, noise Z ~ N(μ, σ);

[0117] Step 4: calculate the latent vector z = E(x real ), generated feature x'real = G(z) and x gen = G(z p );

[0118] Step 5: Calculate various losses L Rec , L KL , L G , L Dis , L gp ;

[0119] Step 6: Optimize parameters θ E , θ G , θ Dis and θ gp : θ E = arg min θ (L Rec + L KL + L G ); θ G = arg min θ (L Rec + L G ); θ Dis = arg max θ (L Dis ); θ gp = arg max θ (L gp ); where θ E represents the learning parameters of the original sample input mapped into the hidden layer features, θ G represents the learning parameters of the new output data generated by sampling from the hidden layer space data distribution, θ Dis represents the learning parameters between the original data input into the discriminator and the new data, and θ gp represents the learning parameters between the original data distribution and the generated output data distribution measured by the gradient penalty.

[0120] Step 7: ITER = ITER + 1;

[0121] Step 8: End.

[0122] (4) Dual-mode feature fusion model of electroencephalogram and electrooculogram based on denoising autoencoder

[0123] As shown in Figure 4 , two fusion strategies of explicit layer and hidden layer are adopted to judge the cognitive load, and the steps of constructing explicit layer fusion are as follows:

[0124] Step 1: Merge the features of electroencephalogram and electrooculogram to obtain the fusion features;

[0125] Step 2: Add noise to the fused features according to the idea of denoising autoencoder to obtain a signal containing noise;

[0126] Step 3: Encode the signal containing noise into the hidden layer space through the encoder to generate hidden layer data;

[0127] Step 4: Finally, generate reconstruction data similar to the original noise-free data through the decoder;

[0128] During the training process, the parameters of the decoder and the encoder are constantly adjusted. When the average reconstruction error between the fused features and the reconstruction data is minimized, the hidden layer data containing the original EEG and EOG signal features are used as the input of the subsequent classifier.

[0129] When using the hidden layer fusion strategy, the features of EEG and EOG are processed separately, and finally the hidden layer data corresponding to EEG and EOG are fused. The steps of processing EEG features are as follows:

[0130] Step 1: Add noise to the EEG feature to obtain an EEG signal containing noise;

[0131] Step 2: Encode the signal containing noise EEG through the encoder to generate EEG hidden layer data in the hidden layer space;

[0132] Step 3: Generate EEG reconstruction data similar to the original noise-free data through the decoder

[0133] During the training process, the parameters of the decoder and the encoder need to be adjusted. When the average reconstruction error between the EEG feature and the EEG reconstruction data is minimized, the data at this time best represents the original input EOG data, which is called EEG hidden layer processing data.

[0134] The same as the processing steps of EEG features, the processed EOG hidden layer processing data can be obtained through the above training steps. Finally, the hidden layer data of the two are merged to form new fused hidden layer features, which are used as the input of the subsequent classifier.

[0135] (5) Improved gull algorithm optimization support vector machine cognitive load classification method

[0136] ① About the improvement of gull algorithm optimization

[0137] The original EEG signal collected from the human brain and the signal fed back from the robot will be acquired and processed in the Android APP. Using the parameter control method of the cosine function, a method using the nonlinear characteristics of the cosine function is introduced to adjust the parameter changes of the original gull. The formula is as follows:

[0138]

[0139] where c represents the control parameter that needs to be adjusted in the seagull algorithm, t represents the iteration number of the algorithm at present, MaxT is the maximum iteration number of the algorithm, c max and c min are the maximum and minimum values of the parameter variation range to be adjusted. In the experiment, c max = 1, c min = 0.004, and MaxT = 500, the parameter control formula obtained by adjustment is:

[0140]

[0141] The control parameter of the seagull algorithm is substituted into the above formula to obtain the new control factor expression:

[0142]

[0143] The control factor of the seagull algorithm at the tthiteration is defined as f c (t), and the maximum value is f max , and the minimum value is f min . When f max = 2, f min = 0.004, and the maximum iteration number MaxT = 500, the improved parameter variation curve is shown in FIG. 2, and the dashed line is the original linear transformation. Figure 5

[0144] Based on the position update formula of the proposed historical evolution strategy, this position update method introduces the historical optimal solution of a single seagull individual and the global worst solution of the entire seagull as a reference, which means that the seagull position update is no longer limited to the global optimal individual, but is realized by approaching the historical optimal solution of the individual and moving away from the global worst solution of the population. The new update formula is as follows:

[0145] HS i (t+1) = r d ×HS i (t) + (1-r d )×P b,i -(1-r d )×HS w (t) (8)

[0146] First, assume that the updated position of the ithseagull individual is HS i (t+1), and the position of the seagull individual before updating is defined as HS i (t). Let P b,i be the historical optimal position of the ithseagull individual, and HS w (t) represent the global worst position of the entire seagull population, r d ​is a random number within [0.1].

[0147] As shown in Figure 6 , the improved HCSOA realizes the steps as follows:

[0148] Step 1: initialize the population, set the control parameters related to the algorithm, and each seagull represents a solution;

[0149] Step 2: calculate the fitness value of the initial seagull individual, and find the optimal individual position;

[0150] Step 3: use the cosine function to improve the control factor;

[0151] Step 4: update the seagull position according to the historical evolution strategy of formula (8);

[0152] Step 5: calculate the fitness value of the updated seagull population, and recalculate the position of the optimal seagull;

[0153] Step 6: judge the iteration number, if it reaches the maximum value, stop optimization and output the optimal value, otherwise continue iteration.

[0154] As shown in Figure 7 , the improved HCSOA-SVM seagull algorithm searches for the optimal solution to search for the optimal parameters, and the highest classification accuracy is obtained through optimal model training, so as to improve the classification performance of SVM. The steps of the improved seagull-support vector machine algorithm are as follows:

[0155] Step 1: generate the initial population and set the related parameters;

[0156] Step 2: calculate the fitness value of the initial seagull population, calculate the position of the optimal seagull individual, and sort the seagull individuals according to the fitness value. Save the optimal seagull position information in the whole space as

[0157] Step 3: use the nonlinear characteristics of the cosine function to adjust the control factor;

[0158] Step 4: update the seagull position according to the historical evolution strategy;

[0159] Step 5: calculate the fitness value of the updated seagull population, and recalculate the position of the optimal seagull;

[0160] Step 6: judge whether the maximum iteration number is reached, if yes, stop iteration and output the optimal parameters C and g, otherwise jump to step 3 and continue iteration.

[0161] ②Performance evaluation of the classifier based on UCI data set

[0162] To verify the effect of the improved classifier, the UCI dataset is used to compare and verify the classifier. Thus, the purpose of evaluating the machine learning classification model is achieved. There are a total of 559 different datasets in this dataset. To test the proposed classification method, the Wine dataset is used in this embodiment, which mainly contains 178 samples and has been labeled.

[0163] In the experiment, the samples are divided into training set and test set. In addition to the improved HCSOA-SVM, the Extreme Learning Machine (ELM) algorithm, Kernel Based Extreme Learning Machine (KELM) algorithm, and SOA-ELM algorithm are selected for simulation comparison, so as to highlight the stability and effectiveness of the improved algorithm. The ten-fold cross-validation diagram of the average accuracy of the training sample classifier is as shown in Figure 8 The average accuracy of the test sample classifier is compared as shown in Figure 9 .

[0164] The selected comparison parameters are mainly the average accuracy of the training effect, the average accuracy of the test effect, the average Kappa coefficient of the training effect, the average JACCARD coefficient of the training effect, the average Kappa coefficient of the test effect, and the average JACCARD coefficient of the test effect. The comparison data of the effects of several classifiers is shown in Table 1. From the table, it can be seen that the HCSOA-SVM has a high classification accuracy, and the accuracy in the training sample and the test sample is 93.444% and 98.222% respectively. At the same time, the performance in the Kappa coefficient and the JACCARD coefficient is not worse than other classification algorithms, proving that the HCSOA-SVM used in the present application has good classification effect.

[0165] Table 1 Comparison of classifier effects

[0166]

[0167]

[0168] Example 2

[0169] A cognitive load evaluation method based on brain intelligence technology, first, the EEG data is enhanced through WVAEGP, second, the features of EEG and EOG signals are fused through a denoising autoencoder, and finally, the SVM classifier is optimized through an improved SOA algorithm. The results show that the classification effect of cognitive load is improved, and the specific implementation is as follows:

[0170] (1) Variational autoencoder generation based on EEG data

[0171] This embodiment validates the effect of EEG enhancement using a public dataset of cognitive load. The mental arithmetic task dataset selected from a public database is the dataset publicly available on Physiobank by the author, Igor Zyma. This dataset primarily contains EEG data from 36 participants. The experiment recruited 66 healthy right-handed volunteers, including 7 women and 19 men. Due to environmental interference and other factors, some invalid data were removed from the final dataset. The experiment used a Neurocom 23-channel EEG acquisition system to collect the EEG data.

[0172] A brainwave data augmentation scheme based on the WVAEGP network was designed. To verify the effect of data augmentation, support vector machine was used to analyze the changes in classification accuracy after data augmentation. Figure 10 This is a WVAEGP data augmentation validation framework diagram, where the upper part is the data augmentation part and the lower part is the data classification validation part after data augmentation.

[0173] Figure 11 This paper compares the wavelet time-frequency plots of real and generated EEG signals. The first row shows the generated signal, and the second row shows the real signal. Visually, the structure and shape of the wavelet time-frequency plots of the EEG signals generated by the WVAEGP model are very similar to the real samples. Even if they are not entirely natural, they are very close to the real samples. Visual effects can only serve as a partial evaluation of the quality of the generated data. To further test the quality of the EEG data generated by the model, the generated samples and real samples are combined and fed into a classifier for classification training. The final classification accuracy is then observed on the test set.

[0174] Compared with the data augmentation effects of WVAEGP, and compared with other data augmentation methods, Figure 12 A visualization comparison of the experimental results is provided. On the test dataset, neither VAE nor WGAN improved accuracy; in fact, increasing their proportion in the training data led to a decrease in accuracy. This indicates that this deterioration is understandable as the network begins to learn more irrelevant features from non-existent artificial data. When the generated data proportion is 30%, WVAEGP achieves the best average accuracy of 75.35%; when the generated data proportion is 40%, DCGAN achieves the best average accuracy of 76.24%; and when the generated data proportion is 40%, WVAEGP achieves the best average accuracy of 79.87%. The accuracy using WVAEGP, DCGAN, and WVAEGP is 2.1%, 2.99%, and 6.62% higher than the accuracy without data augmentation, respectively. This demonstrates that WVAEGP has the best performance among all methods.

[0175] (2) Dual-mode feature fusion model based on EEG and EEG

[0176] To verify the effectiveness of the fusion model, the data of six subjects are selected based on the SVM classifier to compare the cognitive load judgment accuracy, and the effectiveness of the fusion model is verified by comparing single-mode electroencephalogram, single-mode electrooculogram, double-mode explicit layer fusion and double-mode implicit layer fusion, Figure 13 The experimental results are shown in the figure. Through analysis of the experimental results, it can be seen that the accuracy of the two fusion models is higher than that of the single-mode model based on electroencephalogram or electrooculogram in different subjects, and the accuracy of explicit and implicit fusion is similar. The highest accuracy of explicit layer fusion strategy is 82.76%, and the highest accuracy of implicit layer fusion strategy is 81.67%.

[0177] (3) Improved cognitive load classification method of SVM optimized by ternar sea gull algorithm

[0178] The influence of brain electrical data enhancement ratio and different strategies of electroencephalogram and electrooculogram fusion is compared, the improved HCSOA-SVM classifier is used to classify the cognitive load state, based on explicit layer fusion and implicit layer fusion, the classification effect of different proportion of data enhancement generated data in the improved SVM classifier is verified and analyzed, and the final classification effect is shown in Figure 14 When the classifier is HCSOA-SVM, the proportion of brain electrical data generated data to training data is 30%, the explicit layer fusion strategy has the best cognitive load judgment accuracy effect, and the classification accuracy reaches 90.23%. The improved algorithm can effectively improve the cognitive load judgment accuracy, and can better classify the cognitive load, which is an effective cognitive load judgment method.

[0179] The specific embodiments of the application are described above. It should be understood that the application is not limited to the above specific embodiments, and those skilled in the art can make various modifications or changes within the scope of the claims, which does not affect the essential content of the application.

Claims

1. A cognitive load evaluation method based on brain intelligence technology, characterized in that, The method comprises the following steps: S1.4 channel electroencephalogram signal and electrooculogram signal acquisition; S2. Preprocessing and feature extraction of the electroencephalogram signal and electrooculogram signal collected in step S1; S3. Variational autoencoder generated adversarial enhancement method of electroencephalogram signal; The data of the electroencephalogram signal processed in step S2 is enhanced by a data enhancement algorithm WVAEGP algorithm; The WVAEGP algorithm comprises: adding a discriminator link in the VAE network; and using the Wasserstein distance and gradient penalty to solve the problems of gradient explosion and gradient disappearance; S4. Double-mode feature fusion model of electroencephalogram and electrooculogram based on a denoising autoencoder The double-mode feature fusion model of electroencephalogram and electrooculogram based on the denoising autoencoder comprises constructing an explicit layer fusion strategy or an implicit layer fusion strategy to judge the cognitive load, wherein the step of constructing the explicit layer fusion is as follows: Step 41: merging the features of the electroencephalogram and electrooculogram signals to obtain fused features; Step 42: adding noise to the fused features according to the idea of the denoising autoencoder to obtain signals containing noise; Step 43: encoding the signals containing noise to an implicit layer space by an encoder to generate implicit layer data; Step 44: finally generating reconstructed data similar to the original noise-free data by a decoder; In the training process, the parameters of the encoder and the decoder are continuously adjusted, and when the average reconstruction error between the fused features and the reconstructed data is the smallest, the implicit layer data containing the original electroencephalogram and electrooculogram signal features are taken as the input of a subsequent classifier; When the implicit layer fusion strategy is adopted, the electroencephalogram and electrooculogram signal features are processed respectively, the implicit layer data corresponding to the electroencephalogram and electrooculogram signals are fused respectively to form new fused implicit layer features, and the fused implicit layer data are taken as the input of a subsequent classifier; S5. Cognitive load classification method of improved albatross algorithm optimized support vector machine The steps of the improved albatross-support vector machine algorithm are as follows: Step 51: generating an initial population and setting related parameters; Step 52: calculating the fitness value of the initial albatross population, calculating the position of the optimal albatross individual, and sorting the albatross individuals according to the fitness value, and saving the optimal albatross position information in the whole space; Step 53: adjusting the control factor using the nonlinear characteristics of the cosine function; （7） represents the current iteration number of the algorithm, is the maximum iteration number of the algorithm, the first iteration of the albatross algorithm is defined as , is the maximum value among them, is the minimum value among them; Step 54: updating the albatross position according to the historical evolution strategy of formula (8); （8） First assume the first The updated location of the individual seagull is The position of the seagull before the update is defined as ,set up For the first The historical best position of an individual seagull, at the same time This represents the worst-case scenario for the entire seagull population. It is a random number within the range [0.1]. Step 55: calculating the fitness value of the updated albatross population and recalculating the position of the best albatross; Step 56: determining whether the maximum number of iterations is reached, if yes, stopping iteration and outputting the optimal parameters C and g, otherwise, jumping to step 53 for continuous iteration.

2. The cognitive load evaluation method based on brain intelligence technology according to claim 1, characterized in that, In step S1, all electroencephalogram data and electrooculogram data of the user are collected by using a 14-channel electroencephalogram acquisition device Emotive. 3.The cognitive load evaluation method based on brain intelligence technology according to claim 1, wherein, In step S2, the pre-processing stage of the electroencephalogram signal, filtering uses a fast Fourier transform method to perform band-pass filtering on the electroencephalogram data to be processed, corrects linear drift, and removes environmental noise to a certain extent, retains low-frequency signals of 0.1 to 40 Hz, and performs notch processing on 50 Hz signals with a slope of 40 dB / oct; the feature extraction of the electroencephalogram signal selects wavelet entropy, permutation entropy and differential entropy as the features of the electroencephalogram signal. 4.The cognitive load evaluation method based on brain intelligence technology according to claim 1, wherein, In step S2, the pre-processing of the electrooculogram signal uses a fast independent component analysis method to separate the principal components of the electrooculogram signal from the electrooculogram signal; the feature extraction of the electrooculogram signal selects the saccade rate, saccade rate variance, saccade amplitude variance, saccade amplitude power, saccade number in the saccade category, and the blink rate, amplitude variance, average blink amplitude, blink amplitude power, blink number, and average blink duration variance, blink duration, and average saccade number in the blink category as the electrooculogram signal features. 5.The cognitive load evaluation method based on brain intelligence technology according to claim 1, wherein, In step S3, the WVAEGP algorithm process includes: Input: training samples; maximum number of iterations T; Output: learned parameters , , and for the encoder , generator , discriminator , Step 1: Initialize parameters , , and Set iteration unit ITER = 1; Step 2: Determine whether ITER < T; Step 3: Randomly sample the number of training samples and Gaussian values, noise ; Step 4: Compute latent vectors , generated features and ; Step 5: Compute various losses , , , , ; Step 6: Optimizing parameters using ADAM optimizer , , and : ; ; ; ; wherein represents learning parameters that map the input original samples into the hidden layer features, represents learning parameters that sample from the hidden layer space data distribution to generate new output data, represents learning parameters between the original data and the new data input into the discriminator, represents learning parameters that measure the distance between the original data distribution and the generated output data distribution from the gradient penalty. Step 7: ITER = ITER + 1; Step 8: End. 6.The cognitive load evaluation method based on brain intelligence technology according to claim 1, wherein, In step S4, the electroencephalogram feature processing steps are as follows: Step 1: Add noise to the electroencephalogram features to obtain an electroencephalogram signal containing noise; Step 2: Encode the noise-containing electroencephalogram signal to the hidden layer space through the encoder to generate electroencephalogram hidden layer data; Step 3: Generate electroencephalogram reconstruction data similar to the original noise-free data through the decoder; During the training process, the parameters of the decoder and the encoder need to be adjusted, and when the average reconstruction error between the electroencephalogram features and the electroencephalogram reconstruction data is the smallest, the data at this time is called electroencephalogram hidden layer processing data. 7.The cognitive load evaluation method based on brain intelligence technology according to claim 1, wherein, In step S4, the electrooculogram feature processing steps are as follows: Step 1: Add noise to the electrooculogram features to obtain an electrooculogram signal containing noise; Step 2: Encode the noise-containing electrooculogram signal to the hidden layer space through the encoder to generate electrooculogram hidden layer data; Step 3: Generate electrooculogram reconstruction data similar to the original noise-free data through the decoder; During the training process, the parameters of the decoder and the encoder need to be adjusted, and when the average reconstruction error between the electrooculogram features and the electrooculogram reconstruction data is the smallest, the data at this time is called electrooculogram hidden layer processing data.

Images