A cognitive load classification method and system

Abstract

The application provides a cognitive load classification method and system. The method comprises obtaining an electroencephalogram signal of a target user; inputting the electroencephalogram signal into a trained cognitive load classification model to obtain a cognitive load level of the target user at this time; and the classification process of the cognitive load classification model specifically comprises: generating an energy topographic map according to the input electroencephalogram signal of the user; extracting spatial features in the energy topographic map; performing spectral time feature extraction on the extracted spatial features to locate features in a most discriminative time period; and predicting a cognitive load level of the user according to the features in the most discriminative time period; wherein the extraction of the spatial features in the energy topographic map specifically comprises: step A1: independently extracting and optimizing information of each frequency band in the energy topographic map to obtain corresponding optimized spatial features of each frequency band; and step A2: fusing feature maps of different scales to obtain final spatial features.

Description

Technical Field

[0001] This invention relates to the field of electroencephalography (EEG) signal technology, and more particularly to a method and system for classifying cognitive load. Background Technology

[0002] With the development of brain-computer interface (BCI), human-computer interaction (HCI), and virtual reality technologies, integrating EEG classification into related technologies to improve human-computer interaction capabilities has become a new development direction.

[0003] Since human cognitive abilities are limited, when the cognitive demands of a task exceed these limits, cognitive overload occurs, leading to decreased task performance and slower learning. Therefore, extracting EEG signals from cognitive states and constructing efficient and accurate classification and recognition models is crucial for many applications, including brain-computer interfaces, human-computer interaction, and tutoring services.

[0004] Electroencephalography (EEG) is a widely used non-invasive neuroimaging technique that works by measuring voltage changes on the scalp caused by cortical activity. It represents the spontaneous and rhythmic electrical activity of brain cell groups recorded by electrodes and has been widely used in daily life and research.

[0005] With the development of convolutional neural networks (CNNs), the recognition of EEG signals has become more versatile. Convolutional structures can leverage spatial relationships to better extract features from multi-channel data for EEG decoding, reducing the number of parameters that need to be learned and improving the training efficiency of backpropagation algorithms. However, the downsampling inherent in CNNs can lead to information loss during feature acquisition. Furthermore, due to low signal-to-noise ratios and small amplitudes, feature loss significantly increases the loss rate, severely impacting the model's classification results. In addition, although convolutional layers can extract features at different levels of the input data, simply stacking multiple convolutional layers may result in limited expressive power of the feature representation.

[0006] Considering the temporal continuity of EEG signals, there are currently EEG signal recognition methods based on LSTM networks. At each time step, the LSTM network compresses the input data and generates a hidden state to be passed to the next time step. Therefore, LSTM can capture the temporal and long-term dependencies in the EEG signal. However, in this process, some detailed information of the input data may be lost, and only the summary information in the hidden state is retained. This information compression may lead to the loss of some original features, especially when the input data contains important local details. Summary of the Invention

[0007] To improve the accuracy of identifying users' cognitive load status, this invention provides a cognitive load classification method and system.

[0008] On one hand, the present invention provides a cognitive load classification method, comprising:

[0009] Acquire the target user's brainwave signals;

[0010] The EEG signal is input into a trained cognitive load classification model to obtain the cognitive load level of the target user at that time. The classification process of the cognitive load classification model specifically includes: generating an energy topography map based on the input user's EEG signal; extracting spatial features from the energy topography map; extracting spectral temporal features from the extracted spatial features to locate the features within the most discriminative time period; and predicting the user's cognitive load level based on the features within the most discriminative time period.

[0011] Specifically, the spatial features extracted from energy topographic maps include:

[0012] Step A1: Extract and optimize the information of each frequency band in the energy topography map independently to obtain the optimized spatial features corresponding to each frequency band;

[0013] Step A2: Obtain feature maps of different scales based on the optimized spatial features, and fuse the feature maps of different scales to obtain the final spatial features.

[0014] Further, step A1 specifically includes: using a set of Depthwise convolutions to independently extract information from each frequency band and amplify the feature map of each frequency band; performing global max pooling and average pooling operations on the feature map of each frequency band according to formula (1) to obtain the spatial features corresponding to that frequency band; using a shared one-dimensional convolution to calculate the weights of the spatial features of that single frequency band to obtain the attention weights F of that spatial feature; optimizing the feature map of each frequency band according to the attention weights F, and combining the residual idea to obtain the optimized spatial features corresponding to each frequency band;

[0015] F=σ(C1D k (MaxPool(X)))+σ(C1D k (AvgPool(X))) (1)

[0016] Where X represents the feature map of a single frequency band, MaxPool and AvgPool represent the global max pooling and average pooling operations, respectively, and C1D k σ represents a one-dimensional convolution with kernel K, and σ represents the softmax function.

[0017] Furthermore, step A2 specifically includes: the optimized spatial features are sequentially passed through two sets of convolutional layers to obtain feature maps C at two different scales. i and C i+1 Next, according to formula (2), the intermediate feature maps C output by the two sets of convolutional layers are first processed. i C i+1 Each feature map is convolved pointwise to obtain two new feature maps. The high-level feature map is then upsampled and added to the other new feature map. Finally, pointwise convolution is performed to fuse the feature maps and obtain the final spatial features.

[0018] P i ,P i+1 =f(Con 1*1 (C i C i+1 (2)

[0019] Where i represents the i-th convolutional layer, Con 1*1 denoted as pointwise convolution, and f denotes the fusion operation including upsampling, addition, and pointwise convolution.

[0020] Furthermore, spectral temporal features are extracted from the extracted spatial features to locate the features within the most discriminative time period, specifically including:

[0021] The extracted spatial features are input into a BILSTM, which is then used to extract forward and backward temporal features. and And according to formula (3), the forward and reverse time characteristics are... and The concatenation process yields a new output vector H;

[0022]

[0023] Where cat represents the concatenation operation;

[0024] According to formula (4), the most discriminative feature h within the time period is extracted from the new output vector H using an attention mechanism. * :

[0025]

[0026] Where α is the attention score obtained during training.

[0027] Furthermore, based on the features within the most discriminative time period, the user's cognitive load level is predicted, specifically including: inputting the features within the most discriminative time period into a fully connected layer for classification to obtain the user's cognitive load level.

[0028] On the other hand, the present invention provides a cognitive load classification system, comprising:

[0029] The EEG signal acquisition unit is used to acquire the EEG signals of the target user;

[0030] A cognitive load classification model unit is used to store a trained cognitive load classification model, so as to obtain the cognitive load level of the target user at this time based on the target user's EEG signal through the trained cognitive load classification model; wherein, the cognitive load classification model includes:

[0031] An energy topography generation unit is used to generate an energy topography map based on the input EEG signals of the user.

[0032] Spatial feature extraction unit, used to extract spatial features from energy topographic maps;

[0033] The spectral temporal feature extraction unit is used to extract spectral temporal features from the extracted spatial features in order to locate the features within the most discriminative time period.

[0034] The prediction unit is used to predict the user's cognitive load level based on the characteristics within the most discriminative time period.

[0035] The spatial feature extraction unit includes a frequency band feature adaptive optimization module and a multi-scale feature fusion module.

[0036] The frequency band feature adaptive optimization module is used to independently extract and optimize the information of each frequency band in the energy topography map to obtain the optimized spatial features corresponding to each frequency band.

[0037] The multi-scale feature fusion module is used to obtain feature maps of different scales based on the optimized spatial features, and to fuse the feature maps of different scales to obtain the final spatial features.

[0038] The beneficial effects of this invention are:

[0039] The FFBAO module significantly enriches frequency band features, capturing the most representative frequency band information. The MSFF module, by introducing feature fusion, successfully preserves information overlooked by high-dimensional features, improving the model's expressive power. In the TPE module, BiLSTM is used to extract bidirectional temporal information, combined with an attention mechanism, to better capture key features in temporal patterns, enhancing the model's performance on memory load classification tasks. This invention, by accurately identifying cognitive load EEG signal levels, can play a crucial role in applications including brain-computer interfaces, computer-aided education, and medical rehabilitation. Attached Figure Description

[0040] Figure 1A flowchart illustrating a cognitive load classification method provided in an embodiment of the present invention;

[0041] Figure 2 This is a network structure diagram of the cognitive load classification model provided in an embodiment of the present invention;

[0042] Figure 3 This is the Sternberg memory task paradigm;

[0043] Figure 4 This is a schematic diagram of a cognitive load classification system provided in an embodiment of the present invention. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of the embodiments of this invention will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0045] Example 1

[0046] like Figure 1 As shown, this embodiment of the invention provides a cognitive load classification method, including the following steps:

[0047] S101: Obtain the target user's EEG signal according to the 10-10 International Leads Association;

[0048] S102: Input the target user's EEG signal into a trained cognitive load classification model (e.g., ... Figure 2 As shown in the figure, the cognitive load level of the target user at this time is obtained; wherein, the classification process of the cognitive load classification model specifically includes:

[0049] S1021: Generate an energy topography map based on the user's input EEG signals;

[0050] Specifically, electroencephalogram (EEG) signals consist of multiple time series, each corresponding to a measurement at a spatial location on the cortex. The most prominent features of EEG signals reside in the frequency domain, therefore, the spectra of EEG signals are typically used for analysis.

[0051] In this embodiment, the power spectrum of the electroencephalogram (EEG) is extracted using Fast Fourier Transform (FFT). Then, the energy value of each frequency band is obtained by summing the squares of the signal amplitudes within different frequency bands based on the power spectrum. The electrode coordinates according to the 10⁻¹⁰ International Lead Standard (ICS) are projected from three-dimensional space onto a two-dimensional plane. The energy value of each frequency band is then projected onto the corresponding electrode location, thus obtaining an energy topography map for each frequency band. In this embodiment, the energy values ​​of three frequency bands are mainly extracted, resulting in energy topography maps for three frequency bands, thus forming three RGB images for each channel.

[0052] S1022: Extract spatial features from energy topographic maps; specifically including:

[0053] Step A1: Extract spatial features independently from the information of each frequency band in the energy topography map and optimize them to obtain the optimized spatial features corresponding to each frequency band;

[0054] Specifically, this embodiment proposes a Frequency Band Feature Adaptive Optimization Module (FBFAO), which consists of a set of Depthwise convolutions, a global max pooling layer, an average pooling layer, and a shared one-dimensional convolution for all frequency bands. The optimization process is as follows: information for each frequency band is extracted independently using a set of Depthwise convolutions, and the feature map of each frequency band is amplified (the initial 3-channel image is amplified to 33 through Depthwise convolution, that is, the feature map of each frequency band is amplified by 11 times); global max pooling and average pooling operations are performed on the feature map of each frequency band according to formula (1) to obtain the spatial features corresponding to that frequency band; the shared one-dimensional convolution is used to calculate the weights of the spatial features of a single frequency band to obtain the attention weights F of that spatial features; the feature map of each frequency band is optimized according to the attention weights F to obtain the optimized spatial features corresponding to each frequency band.

[0055] F=σ(C1D k (MaxPool(X)))+σ(C1D k (AvgPool(X))) (1)

[0056] Where X represents the feature map of a single frequency band, MaxPool and AvgPool represent the global max pooling and average pooling operations, respectively, and C1D k σ represents a one-dimensional convolution with kernel K, and σ represents the softmax function.

[0057] By extracting information from each frequency band independently through convolution without fusing the information, rich feature representations containing only information from a single frequency band can be obtained. Max pooling and average pooling operations can be used to extract different spatial features of a single frequency band. Using shared one-dimensional convolutions to calculate weights on the feature maps of all frequency bands allows for the acquisition of the most representative frequency band features.

[0058] Step A2: Fuse feature maps of different scales to obtain the final spatial features.

[0059] Specifically, this embodiment proposes a Multi-Scale Feature Fusion Module (MSFF). The feature map optimized by the FFBAO module is fed into the MSFF for processing. The MSFF processing procedure is as follows:

[0060] The feature map optimized by the FFBAO module is passed through two sets of convolutional layers to obtain two feature maps C at different scales. i and C i+1 Then, according to formula (2), the intermediate feature maps C output by the two sets of convolutional layers are processed. i C i+1 Each feature map is convolved pointwise to obtain two new feature maps. The high-level feature map is then upsampled and added to the other new feature map. Finally, pointwise convolution is performed to fuse the feature maps and obtain the final spatial features.

[0061] P i ,P i+1 =f(Con 1*1 (C i C i+1 (2)

[0062] Where i represents the i-th convolutional layer, Con 1*1 denoted as pointwise convolution, and f denotes the fusion operation including upsampling, addition, and pointwise convolution.

[0063] Ultimately, the size of the feature map generated by the fusion module is doubled compared to the size of the original input feature map, thus retaining more information. Through MSFF processing, information that was previously ignored can be preserved, and the expressive power of the features can be improved.

[0064] S1023: Extract spectral and temporal features from the extracted spatial features in order to locate the features within the most discriminative time period;

[0065] Specifically, this embodiment proposes a Temporal Pattern Enhancement Module (TPE). The TPE module mainly consists of a BiLSTM and an attention module. Its working process is as follows: the extracted spatial features are input into the BiLSTM, and the BiLSTM is used to extract forward and backward temporal features. and And the forward and reverse time features and The new output vector H is obtained by concatenation to ensure that the forward and reverse time patterns are completely preserved; the concatenation process can be represented by formula (3):

[0066]

[0067] Here, cat represents the cat function in the torch computing framework.

[0068] According to formula (4), the most discriminative feature h within the time period is extracted from the new output vector H using an attention mechanism. * :

[0069]

[0070] Where α is the attention score obtained during training.

[0071] S1024: Based on the characteristics within the most discriminative time period, predict the user's cognitive load level;

[0072] Specifically, in this embodiment, a fully connected layer is used as the prediction module. The features of the most discriminative time period are input into the fully connected layer for classification to obtain the user's cognitive load level.

[0073] Based on the above embodiments, in this embodiment of the invention, the training process of the cognitive load classification model specifically includes:

[0074] For each level of cognitive load, participants were tested using a modified version of the Sternberg memory task, and their EEG signals were collected during the experiment; for example... Figure 3 As shown, the experimental procedure includes: displaying a set of characters (composed of different English characters around a central point) on the screen, denoted as SET, for the subject to memorize; this display process lasts for 500 milliseconds; wherein, the different numbers of characters contained in the SET represent different cognitive load levels (for example, 2 characters, 4 characters, 6 characters, and 8 characters correspond to the first, second, third, and fourth cognitive load levels, respectively); after a 3-second interval, a character is displayed on the screen, and the subject is asked to judge whether the character displayed at this time belongs to the SET;

[0075] The cognitive load classification model was trained using EEG signals at different cognitive load levels.

[0076] In this embodiment of the invention, the FFBAO module can greatly enrich frequency band features and obtain the most representative frequency band information. The MSFF module introduces the concept of feature fusion, successfully preserving information ignored by high-dimensional features and improving the model's expressive power. In the TPE module, BiLSTM is used to extract bidirectional temporal information, combined with an attention mechanism, which can better capture key features in temporal patterns and improve the model's performance on memory-loaded classification tasks.

[0077] Example 2

[0078] To implement the above method embodiments, this invention also provides a cognitive load classification system, such as... Figure 4 As shown, it includes an EEG signal acquisition unit, a cognitive load classification model unit, an energy topography generation unit, a spatial feature extraction unit, a spectral time feature extraction unit, and a prediction unit; the spatial feature extraction unit includes a frequency band feature adaptive optimization module and a multi-scale feature fusion module.

[0079] The system comprises the following components: an EEG signal acquisition unit for acquiring the target user's EEG signal; a cognitive load classification model unit for storing a trained cognitive load classification model to determine the target user's cognitive load level based on their EEG signal; an energy topography map generation unit for generating an energy topography map based on the input user's EEG signal; a spatial feature extraction unit for extracting spatial features from the energy topography map; a spectral temporal feature extraction unit for extracting spectral temporal features from the extracted spatial features to locate the most discriminative time period; a prediction unit for predicting the user's cognitive load level based on the most discriminative time period; a frequency band feature adaptive optimization module for independently extracting and optimizing spatial features from each frequency band in the energy topography map to obtain optimized spatial features for each frequency band; and a multi-scale feature fusion module for fusing feature maps at different scales to obtain the final spatial features.

[0080] It should be noted that the embodiments of the present invention are for implementing the above method embodiments, and their specific functions can be referred to the above method embodiments, which will not be repeated here.

[0081] To verify the role of each module in improving the overall model, the following comparative experiment was also conducted.

[0082] (I) Evaluation Indicators:

[0083] Accuracy, precision, recall, and F1 score are used as four metrics to evaluate model performance. Their corresponding formulas are as follows:

[0084]

[0085] Where TP is the number of correctly predicted positive samples, TN is the number of correctly predicted negative samples, FP is the number of incorrectly predicted positive samples, and FN is the number of incorrectly predicted negative samples.

[0086] (II) Experimental Results

[0087] The comparison methods include four existing classification models: CNN+1D-Conv, CNN+LSTM, CNN+LSTM / 1D-Conv, and ARCNN. The experimental results of this invention and the existing classification models are shown in Table 1.

[0088] Table 1. Average classification performance of various deep learning models under 10x cross-validation.

[0089] CNN+1D-Conv 0.9053 0.9176 0.9061 0.9025 CNN+LSTM 0.9077 0.9223 0.9116 0.9057 CNN+LSTM / 1D-Conv 0.9148 0.9297 0.9166 0.9142 ARCNN 0.9207 0.9322 0.9236 0.9197 This invention 0.9355 0.9435 0.9353 0.9333

[0090] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A cognitive load classification method, characterized in that, include: Acquire the target user's brainwave signals; The EEG signal is input into a trained cognitive load classification model to obtain the cognitive load level of the target user at this time. The classification process of the cognitive load classification model specifically includes: generating an energy topography map based on the input user's EEG signal; extracting spatial features from the energy topography map; extracting frequency band time features from the extracted spatial features to locate the features within the most discriminative time period; and predicting the user's cognitive load level based on the features within the most discriminative time period. Specifically, the spatial features extracted from energy topographic maps include: Step A1: Extract and optimize the information of each frequency band in the energy topography map independently to obtain the optimized spatial features corresponding to each frequency band; Step A2: Obtain feature maps at different scales based on the optimized spatial features, and fuse the feature maps at different scales to obtain the final spatial features; The extracted spatial features are subjected to frequency band temporal feature extraction in order to locate the features within the most discriminative time period, specifically including: The extracted spatial features are input into a BILSTM, which is then used to extract forward and backward temporal features. and And according to formula (3), the forward and reverse time characteristics are... and The concatenation process yields a new output vector H; (3) in, Indicates a splicing operation; According to formula (4), the attention mechanism is used to obtain the new output vector. Extract the most discriminative features within the time period : (4) in, It is the attention score obtained during training.

2. The cognitive load classification method according to claim 1, characterized in that, Step A1 specifically includes: using a set of Depthwise convolutions to independently extract information from each frequency band and amplify the feature map of each frequency band; performing global max pooling and average pooling operations on the feature map of each frequency band according to formula (1) to obtain the spatial features corresponding to that frequency band; using a shared one-dimensional convolution to calculate the weights of the spatial features of that frequency band to obtain the attention weights F of that spatial features; optimizing the feature map of each frequency band according to the attention weights F, and combining the residual idea to obtain the optimized spatial features corresponding to each frequency band; (1) in, A feature map representing a single frequency band. and These represent global max pooling and average pooling operations, respectively. This represents a one-dimensional convolution with kernel K. This represents the softmax function.

3. The cognitive load classification method according to claim 1, characterized in that, Step A2 specifically includes: the optimized spatial features are sequentially passed through two sets of convolutional layers to obtain feature maps at two different scales. and Next, according to formula (2), the intermediate feature maps output by the two sets of convolutional layers are first processed. Each feature map is convolved pointwise to obtain two new feature maps. The high-level feature map is then upsampled and added to the other new feature map. Finally, pointwise convolution is performed to fuse the feature maps and obtain the final spatial features. ， ， (2) in, The final spatial feature is represented by i, where i represents the i-th convolutional layer. This represents pointwise convolution. This indicates operations including upsampling, addition, and pointwise convolution fusion.

4. The cognitive load classification method according to claim 1, characterized in that, Based on the features within the most discriminative time period, the user's cognitive load level is predicted, specifically by inputting the features within the most discriminative time period into a fully connected layer for classification to obtain the user's cognitive load level.

5. A cognitive load classification system, characterized in that, include: The EEG signal acquisition unit is used to acquire the EEG signals of the target user; A cognitive load classification model unit is used to store a trained cognitive load classification model, so as to obtain the cognitive load level of the target user at this time based on the target user's EEG signal through the trained cognitive load classification model; wherein, the cognitive load classification model includes: An energy topography generation unit is used to generate an energy topography map based on the input EEG signals of the user. Spatial feature extraction unit, used to extract spatial features from energy topographic maps; The spectral temporal feature extraction unit is used to extract frequency band temporal features from the extracted spatial features in order to locate the features within the most discriminative time period; specifically, it includes: extracting frequency band temporal features from the extracted spatial features in order to locate the features within the most discriminative time period, specifically including: The extracted spatial features are input into a BILSTM, which is then used to extract forward and backward temporal features. and And according to formula (3), the forward and reverse time characteristics are... and The concatenation process yields a new output vector H; (3) in, Indicates a splicing operation; According to formula (4), the attention mechanism is used to obtain the new output vector. Extract the features within the most discriminative time period : (4) in, It is the attention score obtained during training; The prediction unit is used to predict the user's cognitive load level based on the characteristics within the most discriminative time period. The spatial feature extraction unit includes a frequency band feature adaptive optimization module and a multi-scale feature fusion module. The frequency band feature adaptive optimization module is used to independently extract and optimize the information of each frequency band in the energy topography map to obtain the optimized spatial features corresponding to each frequency band. The multi-scale feature fusion module is used to obtain feature maps of different scales based on the optimized spatial features, and to fuse the feature maps of different scales to obtain the final spatial features.

Images