Brain load assessment method and system based on BEAM

Abstract

The invention discloses a brain load assessment method and system based on BEAM. The method comprises the steps that EEG data sets corresponding to high, medium and low symbol loads of a subject are collected, and labels are edited; carrying out data preprocessing on the collected data to reduce the calculation amount and reserve effective frequency band information; calculating the power spectrum density by adopting a Welch method; constructing a color BEAM image by using the obtained power spectrum density and the electrode coordinates; inputting the BEAM sequence into a convolutional neural network to extract deep spatial features; the image sequence is transmitted into a BiLSTM module to capture time sequence features; constructing a GUI (Graphical User Interface) to visually display the spatial distribution condition of the electroencephalogram signals; compared with the traditional electroencephalogram frequency band feature vector research, the method has the advantages that the position information between the electrodes can be better reserved; the gradient disappearance problem in the deep network is effectively solved; the prediction accuracy is well improved; the active condition of each functional area of the brain is visually displayed, and the convenience of man-machine interaction is improved.

Description

Technical Field

[0001] The present invention relates to the field of brain-computer interface, and in particular to a method and system for evaluating mental workload based on electroencephalographic mapping (BEAM). Background Art

[0002] Mental workload (MWL) can be used as a measure of the mental effort required to complete a task. Analyzing MWL helps us understand the physiological and psychological state of human-computer interaction users and the complexity of the operations they perform. Categorizing the mental workload under task conditions and optimizing task difficulty based on these findings to avoid unexpected situations are crucial for studying MWL classification.

[0003] Application number "CN116756657A" discloses a fNIRS mental workload detection method based on CNN and Transformer. This method preprocesses the raw data to obtain signals of oxyhemoglobin and deoxyhemoglobin concentrations, combines the two convolved signals, extracts state features through a convolutional neural network and a Transformer module, and inputs the classification layer to obtain the classification results; Application number "CN118902472A" provides a cross-task mental workload evaluation method based on EEG tensor transfer learning. This method analyzes the EEG time domain information of multiple channels under different tasks, constructs a target domain three-dimensional tensor containing the time-frequency features of the multi-channel EEG signals, and uses the screening features in the three-dimensional tensor as the basis for mental workload classification. In the above methods, there is a problem that the signal feature extraction method is single and not intuitive enough. Although the second method comprehensively considers the signal features of various functional areas of the brain, it cannot effectively extract the spatial position relationship between multiple channels. Summary of the Invention

[0004] In order to solve the above problems, the present invention provides a BEAM-based mental workload assessment method and system, the purpose of which is to provide a highly efficient, accurate and intuitive mental workload classification method by analyzing the spatial and temporal characteristics in the BEAM image sequence, and to design a corresponding graphical user interface (GUI).

[0005] To achieve the above objectives, the technical implementation scheme of the present invention is as follows:

[0006] A BEAM-based mental workload assessment method, comprising:

[0007] Collect EEG datasets corresponding to high, medium, and low loads of the subjects and edit the labels;

[0008] The power spectral density was calculated using the Welch method.

[0009] The obtained power spectrum density and electrode coordinates are used to construct a color BEAM image;

[0010] Input the BEAM sequence into the convolutional neural network to extract deep spatial features;

[0011] The image sequence is passed into the BiLSTM module to capture temporal features;

[0012] Construct a GUI to intuitively display the spatial distribution of EEG signals.

[0013] Furthermore, the power spectral density is calculated using the Welch method, specifically including:

[0014] Analyze the 0.5-34Hz frequency band in the EEG signal, mark N data observation points in a signal segment, and record the i-th signal segment as The power spectrum calculation formula of this signal is:

[0015]

[0016] Where M is the total number of data; as a normalization factor;

[0017] Average the L segmented periodograms to get x N The power spectral density estimation of (n) is calculated as follows:

[0018]

[0019] Furthermore, constructing a color BEAM image using the obtained power spectrum density and electrode coordinates specifically includes:

[0020] The three-dimensional electrode information was projected onto a two-dimensional surface using the equidistant projection method; the blank areas between electrodes were filled using the bicubic interpolation method to construct a color topographic map.

[0021] Furthermore, the step of inputting the BEAM sequence into the convolutional neural network to extract deep spatial features specifically includes: inputting the image sequence into the ResNet module; if the original expected mapping is defined as H(u), the residual block reconstructs it as:

[0022] H(u)=F(u)+u

[0023] Where u is the input, and F(u) is the residual function to be learned;

[0024] Furthermore, the image sequence is fed into the BiLSTM module to capture temporal features, specifically including: the spatial features extracted by ResNet18 are fed into the BiLSTM network for processing; the cell state is combined with the input of the input gate and the forgotten content of the forget gate to update the current cell state, which is expressed as:

[0025] c t =f t ⊙c t-1 +i t ⊙tanh(W c ·u t +U c ·h t-1 +b c )

[0026] Where c t and c t-1 are the cell states at the current moment and the previous moment, respectively, f t is the output of the forget gate at the current moment, with a value between 0 and 1, ⊙ represents element-by-element multiplication, i t is the output of the input gate at the current moment, with a value between 0 and 1, tanh(·) is the hyperbolic tangent activation function, u t is the input vector at the current moment, W c Is the cell state update acting on u t The weight matrix, h t-1 is the hidden state of the previous moment, U c Is the cell state update acting on h t-1 The weight matrix, b c is the bias term in the cell state update.

[0027] Furthermore, the GUI is constructed to intuitively display the spatial distribution of the EEG signal, specifically by constructing a main interface framework based on QMainWindow of PyQt5.

[0028] Furthermore, before using the Welch method to calculate the power spectral density, the collected data is preprocessed to reduce the amount of calculation and retain the effective frequency band information.

[0029] A BEAM-based mental workload assessment system, comprising:

[0030] The data collection module is used to collect EEG data sets corresponding to high, medium and low loads of the subjects and edit labels;

[0031] The power spectrum density calculation module is used to calculate the power spectrum density using the Welch method;

[0032] Construct a color BEAM image module to construct a color BEAM image using power spectral density and electrode coordinates;

[0033] The spatial feature extraction module is used to input the BEAM sequence into the convolutional neural network to extract deep spatial features;

[0034] The module for capturing temporal features is used to pass the image sequence into the BiLSTM module to capture temporal features;

[0035] The display module is used to build a GUI to intuitively display the spatial distribution of EEG signals.

[0036] Furthermore, the BEAM-based mental workload assessment system also includes:

[0037] The preprocessing module is used to preprocess the collected data to reduce the amount of calculation and retain the effective frequency band information.

[0038] A computer medium stores a computer program, which implements the method described above when executed by a processor.

[0039] Beneficial effects:

[0040] 1. The present invention uses bicubic interpolation to construct color topographic maps, which can more intuitively reflect the spatial distribution of EEG data. Compared with traditional EEG frequency band feature vector research, it can better preserve the position information between electrodes;

[0041] 2. The ResNet18 in this invention adopts a residual network structure, combined with skip connections, which can effectively solve the gradient vanishing problem in deep networks. The prediction accuracy of the CNN-BiLSTM model is significantly improved compared with the single CNN model and the traditional EEG analysis method.

[0042] 3. The present invention constructs a QMainWindow main interface framework based on PyQt5, integrates EEG preprocessing functions and provides a BEAM display window, which can intuitively display the activity of various functional areas of the brain when evaluating mental workload, thereby improving the convenience of human-computer interaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Attachment Figure 1 It is a flow chart of the method of the present invention.

[0044] Attachment Figure 2 This is a diagram of the neural network structure used in the present invention to classify BEAM images.

[0045] Attachment Figure 3 This is a functional diagram of the GUI design used for visually analyzing mental workload according to the present invention. DETAILED DESCRIPTION

[0046] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0047] The present invention provides a method and system for evaluating mental workload based on BEAM. Figure 1 , including the following steps:

[0048] Step 1: Collect EEG datasets corresponding to high, medium, and low loads of the subjects and edit the labels.

[0049] A number of healthy subjects (balanced gender ratio) were selected to conduct experiments on three types of mental workload tasks: high, medium, and low, and the n-back paradigm was used for testing. The experimental task was to click on the target color block with the mouse. The task type and time were the same, and the difficulty index gradually increased. The difficulty index can be described as the movement speed of the target color block in each test: the number of target appearances in each test was fixed, and the next target appeared at a random position on the screen immediately after a valid click was performed. The time required to complete all target hits was recorded. Under a certain difficulty setting, this task scenario will fully test the subject's concentration and dynamic visual ability, thereby inducing changes in mental workload.

[0050] The test requires participants to compare their recently completed task with the nth task before it. By controlling the difference between the current task and the target task, the task difficulty is manipulated to increase mental workload. Participants are required to complete the NASA-TLX Mental Workload Subjective Rating Scale after each round of testing.

[0051] It should be noted that the task type and difficulty index can be changed according to actual needs.

[0052] During the experiment, a Boricon EEG acquisition helmet was used to collect EEG signals. This environment enables wireless, synchronous data acquisition of 8-64 channels. The recording bandwidth was 0-200Hz, the EEG sampling frequency was set to 1024Hz, and all electrode impedances were controlled below 5kΩ (with the use of conductive gel). All EEG signal channels were collected synchronously.

[0053] Step 2: Preprocess the collected data to reduce the amount of calculation and retain the effective frequency band information.

[0054] The original data were downsampled to 250 Hz and filtered using a 0.5–34 Hz ​​bandpass filter.

[0055] Since the sampled signal is mixed with unwanted signals such as electrooculogram and electrocardiogram, independent component analysis (ICA) is required to separate the mixed signal after filtering.

[0056] Considering that if the data segments are short, the changes in brain regions are not obvious, which is not conducive to the subsequent classification of mental workload; and if the data segments are too long, the database sample size may be too small. Therefore, the unit data length is set to 30s.

[0057] Step 3: Calculate the power spectral density using the Welch method. The segmented windowed average periodogram, also known as the Welch method, divides the data into overlapping segments, calculates the windowed periodogram for each segment, and then sums and averages the periodograms of all segments to obtain the final power spectral density.

[0058] Analyze the 0.5-34Hz frequency band in the EEG signal, mark N data observation points in a section of EEG signal as a finite signal x N (n), and divided into overlapping L segments. N The i-th segment signal in (n) is recorded as Then the power spectrum of the signal p i (ω) is calculated as:

[0059]

[0060] Where M is the amount of data contained in the signal segment, and d2(n) is the window function. As a normalization factor. Averaging the L segmented periodograms, we can get x N The power spectral density estimation of (n) is calculated as follows:

[0061]

[0062] Step 4: Use the obtained power spectrum density and electrode coordinates to construct a color BEAM image.

[0063] Brain Electrical Activity Mapping (BEAM) is an advanced method for studying brain function, offering the advantages of high sensitivity and non-invasiveness. BEAM technology uses color-coded EEG power spectrum distribution to transform brain waves into intuitive topographic maps, demonstrating the activity status of various brain functional areas.

[0064] The 3D spatial information of the electrode is projected onto a 2D plane using the equidistant projection method. Let r represent the spherical coordinate radius, φ the polar angle, and θ the azimuth angle. The 2D coordinates of the electrode are then determined from the spherical coordinate parameters using the following formula:

[0065] x=(rcosφ)·cosθ

[0066] y=(r cosφ)·sinθ

[0067] Use bicubic interpolation to fill the blank areas between electrodes based on the power spectrum density characteristics. For a certain pixel point (x, y) to be interpolated, the mathematical formula for calculating the value of the pixel point to be interpolated is:

[0068]

[0069] where f(x i ,y j ), A(x–x i ), A(y–y j ) represents the data values ​​and horizontal and vertical coordinate weights of the 16 pixels closest to (x, y). Weight A(x–x i ), A(y–y j ) is calculated according to the function A(s):

[0070]

[0071] Where a is a constant and its value is -0.5.

[0072] Alternatively, if the dataset is limited and the number of BEAM samples is small, dataset augmentation can be considered. The resulting BEAM images are randomly divided into training, validation, and test sets in a 3:1:1 ratio. The training set is then subjected to five operations: horizontal folding, vertical folding, 90° clockwise and 90° counterclockwise rotation, and random addition of Gaussian noise. This can increase the original training set by six times.

[0073] Step 5: Input the BEAM sequence into the convolutional neural network to extract deep spatial features.

[0074] Residual neural networks (ResNets) address the vanishing gradient problem in deep network training by introducing a residual learning mechanism. The core idea is to construct residual blocks using cross-layer skip connections, enabling the network to directly learn the residual mapping between input and output. Residual neural networks, or ResNets, are a type of convolutional neural network that incorporates a residual learning mechanism to address the vanishing gradient problem.

[0075] In step 5, the ResNet18 network is introduced. A residual block is constructed through cross-layer skip connections, enabling the network to directly learn the residual mapping between input and output. The BEAM sequence is first input into the ResNet module. The original expected mapping is defined as H(u), and the residual block reconstructs it as:

[0076] H(u)=F(u)+u

[0077] Where u is the input and F(u) is the residual function to be learned. By optimizing the residual term F(u) = H(u) - u, the network can more efficiently approximate complex mapping relationships. Taking the residual block structure of BasicBlock as an example, its feature transformation path is:

[0078] F(u)=ReLU(W2·ReLU(W1·u+b1)+b2)

[0079] It consists of two stacked 3×3 convolutional layers, with the input feature tensor u, weight matrices W1 and W2, biases b1 and b2, and ReLU(·) as the ReLU activation function.

[0080] When the input and output dimensions are consistent, x is directly transferred through the jump connection; if the dimensions do not match, 1×1 convolution is used for adjustment, and the mathematical expression is:

[0081] u proj =W s ·u

[0082] Where W s is the weight matrix of the 1×1 convolution kernel, u proj It is the result of 1×1 convolution on the input u.

[0083] Even if the deep web The gradient can still be effectively propagated back through the constant 1 of the skip connection, significantly alleviating the gradient disappearance.

[0084] The output of ResNet18 is directly globally pooled to extract deep spatial features to provide support for subsequent time series modeling.

[0085] Step 6: Pass the image sequence into the BiLSTM module to capture temporal features.

[0086] The Long Short-Term Memory (LSTM) network is a temporal recurrent neural network designed to address the long-term dependency issues inherent in typical RNNs. The Bidirectional Long Short-Term Memory (BiLSTM) network consists of two LSTM layers: one processes the sequence in a forward (sequential) direction, and the other processes it in a backward (reverse) direction. The outputs of the two LSTM layers are concatenated or weighted together to better capture contextual information within the sequence.

[0087] After spatial feature extraction, the BiLSTM module is further used for time series modeling. The spatial features extracted by ResNet18 are fed into the BiLSTM network for processing.

[0088] The BiLSTM gating mechanism formula is as follows:

[0089] Input gate: determines the current input u t For cell status c t Degree of impact:

[0090] i t =σ(W i ·u t +U i·h t-1 +b i )

[0091] Where σ is the Sigmoid activation function, W i is the input vector u acting at the current moment t The weight matrix of the input gate, h t-1 is the hidden state of the previous moment, U i is acting on h t-1 The weight matrix of the input gate, b i is the bias term of the input gate, i t is the output of the input gate.

[0092] Forget gate: determines the previous cell state c t-1 How much information should be forgotten:

[0093] f t =σ(W f ·u t +U f ·h t-1 +b f )

[0094] Where W f is applied to the input u t The weight matrix of the forget gate, U f is acting on h t-1 The weight matrix of the forget gate, b f is the bias term of the forget gate, f t is the output of the forget gate at the current moment.

[0095] Output gate: determines the current cell state c t Whether part of the content is output to the next time step:

[0096] o t =σ(W o ·u t +U o ·h t-1 +b o )

[0097] Where W o is acting on u t The weight matrix of the output gate, U o is acting on h t-1 The weight matrix of the output gate, b o is the bias term of the output gate, o t is the output of the output gate at the current moment.

[0098] Cell state update: The cell state combines the input of the input gate and the forgotten content of the forget gate to update the current cell state:

[0099] c t =f t ⊙c t-1 +i t ⊙tanh(W c ·u t +U c ·h t-1 +b c )

[0100] Where ⊙ is element-wise multiplication, c t-1 is the cell state at the previous moment, tanh(·) is the hyperbolic tangent activation function, W c Is the cell state update acting on u t The weight matrix, U c Is the cell state update acting on h t-1 The weight matrix, b c is the bias term in the cell state update, c t is the cell state at the current moment.

[0101] Hidden state update: The output gate determines part of the current cell state as the hidden state output of the current time step:

[0102] h t =o t ⊙tanh(c t )

[0103] The BiLSTM layer receives the spatial features extracted by ResNet18 and feeds them into a bidirectional LSTM layer. After processing by the LSTM layer, the output of the last time step (representing the features of the entire sequence) is selected and normalized using LayerNormalization. The final output is processed by a fully connected layer to output the classification result.

[0104] The spatial features extracted by ResNet18 are fed into a BiLSTM network for processing. The BiLSTM output is normalized using LayerNormalization to further stabilize model training, and regularized using Dropout to reduce the risk of overfitting.

[0105] Figure 2 Shows the structure of the entire deep learning network.

[0106] Step 7: Build a GUI to intuitively display the spatial distribution of EEG signals.

[0107] PyQt is one of the most powerful and widely used GUI frameworks in Python. We built the main interface framework using QMainWindow, based on PyQt5. By integrating EEG preprocessing algorithms with deep learning prediction models, we can intuitively display the spatial distribution of EEG signals. The graphical user interface integrates data loading and validation, asynchronous signal processing, visualization and interaction, and integrated functional modules.

[0108] See also Figure 3 , the GUI should have the following functions:

[0109] On the basis of the main interface UI_main inheriting the QMainWindow class, add components as shown in the figure.

[0110] When the user clicks the "Read File" button, the system executes the following: pops up the FileTypeDialog dialog box (inherited from QDialog), provides two modes to choose from: "Single File" and "Batch". Finally, the data information passed by lambda is passed to the main interface; in single file mode, QFileDialog.getOpenFileName() is called and the file filter is set to "GDF Files (*.gdf)"; in batch mode, QFileDialog.getExistingDirectory() is called to start a background thread to traverse the directory tree;

[0111] When the user clicks the "Preprocessing" button, asynchronous operation is implemented by inheriting the DataProcessThread of the QThread class, enabling modular code for data acquisition and feature extraction to generate the original BEAM image output.

[0112] When the user clicks the "Start Prediction" button, the system calls the deep learning modular code to predict the original BRAM and generate output.

[0113] When the user clicks the "<" and ">" buttons, the system reads each image in the output folder through the index and places it on the canvas of the main interface.

[0114] When the user clicks the "Zoom in" button, the system creates a QDialog child window to implement the proportional zoom display function.

[0115] A BEAM-based mental workload assessment system, comprising:

[0116] The data collection module is used to collect EEG data sets corresponding to high, medium and low loads of the subjects and edit labels;

[0117] The power spectrum density calculation module is used to calculate the power spectrum density using the Welch method;

[0118] Construct a color BEAM image module to construct a color BEAM image using power spectral density and electrode coordinates;

[0119] The spatial feature extraction module is used to input the BEAM sequence into the convolutional neural network to extract deep spatial features;

[0120] The module for capturing temporal features is used to pass the image sequence into the BiLSTM module to capture temporal features;

[0121] The display module is used to build a GUI to intuitively display the spatial distribution of EEG signals.

[0122] The preprocessing module is used to preprocess the collected data to reduce the amount of calculation and retain the effective frequency band information.

[0123] A computer medium stores a computer program, which, when executed by a processor, implements the above-mentioned BEAM-based mental workload assessment method.

[0124] The above embodiments are intended only to illustrate the technical solutions of the present disclosure, and are not intended to limit them. Although the present disclosure has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they may still modify the technical solutions described in the aforementioned embodiments, or replace some of the technical features therein with equivalents. Such modifications or replacements do not deviate from the spirit and scope of the technical solutions of the various embodiments of the present disclosure, and should be included within the scope of protection of the present disclosure. Furthermore, any content not described in detail in this specification belongs to the prior art known to those skilled in the art.

Claims

1. A BEAM-based mental workload assessment method, characterized in that: include: Collect EEG datasets corresponding to high, medium, and low loads of the subjects and edit the labels; The power spectral density was calculated using the Welch method. The obtained power spectrum density and electrode coordinates are used to construct a color BEAM image; Input the BEAM sequence into the convolutional neural network to extract deep spatial features; The image sequence is passed into the BiLSTM module to capture temporal features; Construct a GUI to intuitively display the spatial distribution of EEG signals.

2. The method for evaluating mental workload according to claim 1, wherein: The Welch method is used to calculate the power spectral density, specifically including: Analyze the 0.5-34Hz frequency band in the EEG signal, mark N data observation points in a signal segment, and record the i-th signal segment as The power spectrum calculation formula of this signal is: Where M is the total number of data; as a normalization factor; Average the L segmented periodograms to get x N The power spectral density estimation of (n) is calculated as follows:

3. The method for evaluating mental workload according to claim 1, wherein: The method of constructing a color BEAM image using the obtained power spectrum density and electrode coordinates specifically includes: The three-dimensional electrode information was projected onto a two-dimensional surface using the equidistant projection method; the blank areas between electrodes were filled using the bicubic interpolation method to construct a color topographic map.

4. The method for evaluating mental workload according to claim 1, wherein: The process of inputting the BEAM sequence into the convolutional neural network to extract deep spatial features specifically includes: inputting the image sequence into the ResNet module; if the original expected mapping is defined as H(u), the residual block reconstructs it as: H(u)=F(u)+u Where u is the input and F(u) is the residual function to be learned.

5. The method for evaluating mental workload according to claim 1, wherein: The image sequence is passed into the BiLSTM module to capture the temporal features, specifically including: the spatial features extracted by ResNet18 are sent to the BiLSTM network for processing; the cell state is combined with the input of the input gate and the forgotten content of the forget gate to update the current cell state, which is expressed as: c t =f t ☉c t-1 +i t ☉tanh(W c ·u t +U c ·f t-1 +b c ) Where c t and c t-1 are the cell states at the current moment and the previous moment, respectively, f t is the output of the forget gate at the current moment, with a value between 0 and 1, ☉ represents element-by-element multiplication, i t is the output of the input gate at the current moment, with a value between 0 and 1, tanh(·) is the hyperbolic tangent activation function, u t is the input vector at the current moment, W c Is the cell state update acting on u t The weight matrix, h t-1 is the hidden state of the previous moment, U c Is the cell state update acting on h t-1 The weight matrix, b c is the bias term in the cell state update.

6. The method for evaluating mental workload according to claim 1, wherein: The GUI is constructed to intuitively display the spatial distribution of the EEG signal, specifically by constructing a main interface framework using QMainWindow based on PyQt5.

7. The method for evaluating mental workload according to claim 1, wherein: Before using the Welch method to calculate the power spectral density, the collected data is preprocessed to reduce the amount of calculation and retain the effective frequency band information.

8. A BEAM-based mental workload assessment system, characterized in that: include: The data collection module is used to collect EEG data sets corresponding to high, medium and low loads of the subjects and edit labels; The power spectrum density calculation module is used to calculate the power spectrum density using the Welch method; Construct a color BEAM image module to construct a color BEAM image using power spectral density and electrode coordinates; The spatial feature extraction module is used to input the BEAM sequence into the convolutional neural network to extract deep spatial features; The module for capturing temporal features is used to pass the image sequence into the BiLSTM module to capture temporal features; The display module is used to build a GUI to intuitively display the spatial distribution of EEG signals.

9. The BEAM-based mental workload assessment system according to claim 8, characterized in that: Also includes: The preprocessing module is used to preprocess the collected data to reduce the amount of calculation and retain the effective frequency band information.

10. A computer medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images