Method for monitoring electroencephalogram cognitive load in real time

Abstract

The invention discloses an electroencephalogram cognitive load real-time monitoring method. The method comprises the following steps: firstly, constructing a sparse structure adjacent matrix based on a spatial distance between electrodes, and introducing GCN to aggregate local features; different brain mesoscopic areas are constructed, and the position and the feature of a virtual mesoscopic center in each area are calculated; original electrode nodes and virtual mesoscopic nodes are fused, a sparse global graph connectivity network is constructed by using an attention mechanism, and global features are aggregated by using graph convolution; and finally, inputting the fused features into a three-layer full-connection network, and carrying out final fatigue detection. According to the method, data preprocessing is carried out on original electroencephalogram data, artifact interference is removed, feature data is extracted, meanwhile, objective task expression data is used for defining the current cognitive load state, finally, a PGCN model is adopted for achieving training of a general electroencephalogram cognitive load real-time monitoring model, and the effect is better than that of other models.

Description

Technical Field

[0001] The present invention belongs to the technical field of electroencephalogram analysis, and particularly relates to a method for real-time monitoring of electroencephalogram cognitive load. Background Art

[0002] With the rapid development of information technology, the application scenarios of human-computer interaction (HCI) have become increasingly complex, and the real-time monitoring and understanding of users' cognitive load have become particularly important. Cognitive load refers to the mental effort exerted by an individual when processing information, which directly affects task execution efficiency, decision-making quality, and learning effects. In order to optimize the user experience, improve work efficiency, and ensure operation safety, especially in key fields such as aviation, medical care, and education, developing technical solutions that can accurately measure and evaluate cognitive load has become a research hotspot.

[0003] Electroencephalogram (EEG) is a non-invasive method for collecting physiological signals, which can record the weak electrical signals generated by the activities of brain neurons. EEG has the characteristic of high temporal resolution and can capture changes in brain states on a millisecond time scale, so it is very suitable for real-time monitoring of cognitive load. In recent years, with the progress of sensor technology and the popularization of portable EEG devices, it has become possible to collect long-term and continuous electroencephalogram data in natural environments. At the same time, the development of machine learning and deep learning algorithms provides strong support for extracting effective features from complex EEG signals, thus realizing more accurate quantitative analysis of cognitive load.

[0004] Traditional electroencephalogram cognitive load monitoring methods have some limitations. First, EEG signals are affected by various factors, such as artifacts interference caused by muscle movement, blinking, etc., which increases the difficulty of signal processing. Second, there are significant differences in electroencephalogram characteristics among different individuals, resulting in poor performance of general models. Third, most of the existing methods rely on subjective feelings to define the cognitive load level, lacking objectivity and being difficult to adapt to actual application scenarios. Summary of the Invention

[0005] The purpose of the present invention is to propose a method that can detect the objective cognitive load ability through electroencephalogram data. First, it is necessary to perform data preprocessing on the original electroencephalogram data, remove artifact interference and extract feature data. At the same time, objective task performance data is used to define the current cognitive load state. Finally, the PGCN model is used to train a general electroencephalogram cognitive load real-time monitoring model.

[0006] A method for real-time monitoring of electroencephalogram cognitive load is carried out according to the following steps:

[0007] (1) Considering the frequent local connections in the brain network, a sparse structural adjacency matrix is constructed based on the spatial distance between electrodes, and a GCN is introduced to aggregate local features;

[0008] (2) Different mesoscopic regions of the brain are constructed, and the positions and features of the virtual mesoscopic centers in each region are calculated;

[0009] (3) To balance the importance and economy of long-distance connections, the original electrode nodes are fused with virtual mesoscopic nodes, a sparse global graph connectivity network is constructed using the attention mechanism, and graph convolution is used to aggregate global features;

[0010] (4) Finally, the fused features are input into a three-layer fully connected network for final fatigue detection.

[0011] The sparse structural adjacency matrix described in step (1) is:

[0012]

[0013] where d ij is the three-dimensional distance between nodes i and j, and δ is the sparsity factor.

[0014] The method for aggregating local features described in step (1) is: First, a sparse graph relationship inference is constructed based on the relative spatial positions of the electrodes, so that its degree follows a power-law distribution with exponential truncation and the adjacency matrix remains sparse enough; then a two-layer GCN is constructed to aggregate the features of adjacent nodes; finally, the original features are fused with the layer-by-layer outputs of the graph convolution to suppress over-smoothing.

[0015] The different mesoscopic regions of the brain described in step (2) are the frontal lobe, parietal lobe, temporal lobe, and occipital lobe.

[0016] The method for constructing the virtual mesoscopic center described in step (2) is: Calculate the weight coefficient of each mesoscopic region by summing each row of e where Λ i represents the functional connection weight of each node with all other nodes; then start to calculate the features and positions of the virtual mesoscopic region center:

[0017] In the formula is the absolute position of the electrode;

[0018] p locate = softmax(Λ)P

[0019] m feature = softmax(Λ)h,

[0020] After calculating the regional center of each mesoscopic region, fuse the features and positions according to different partitions; the fused features and positions are respectively

[0021] Finally, the virtual mesoscopic center and the original electrode nodes are fused through virtual node interpolation to obtain a fused feature and position matrix that includes important local and mesoscopic attributes:

[0022] X meso =(concat((x local ), (M T ), (M (1) ))) T , (M (2)T )) T

[0023] P meso =(concat(P T , (P (1) ), (P T ), (P (2) ))) T )) T .

[0024] The specific construction of the sparse global graph connectivity network in step (3) is as follows: The complete node P meso is composed of the original electrode node P and the virtual mesoscopic node P virtual ; adding the feature and position encoding of all nodes, the position-enhanced node feature is obtained as follows:

[0025] X enhanced = X meso + embed(P meso );

[0028] After that, a multi-head self-attention with 6 heads is used to calculate the relationship matrix and a learnable weight vector is used to fuse G into the final attention relationship matrix :

[0029] A global = WG.

[0030] The graph convolution in step (3) is specifically as follows: Based on the attention relationship matrix A global , the corresponding Laplacian matrix is calculated for graph convolution:

[0031] where O (l) is the input node representation, O (l+1) is the output node representation, and the initial input representation O (0) is the original

[0032]

[0033] Input feature X meso ; W (l) is a learnable weight matrix, and σ is an activation function.

[0034] Step (4) is specifically: the input mesoscopic feature X meso is connected with the global GCN output O (1) to obtain features covering local, mesoscopic, and global receptive fields:

[0035] X global = conct(X meso , O (1) )

[0036] Set X global as the input of a three-layer fully connected network for final fatigue detection.

[0037] Advantages of the present invention: The present invention performs data preprocessing on the original EEG data, removes artifact interference and extracts feature data, simultaneously uses objective task performance data to define the current cognitive load state, and finally uses the PGCN model to implement the training of a general EEG cognitive load real-time monitoring model, establishing the EEG cognitive load real-time monitoring model PGCN, and the effect is better than other models. Brief Description of the Drawings

[0038] Figure 1 is a flowchart of the PGCN model. Specific Embodiments

[0039] To facilitate the understanding of the present invention, the present invention will be described more comprehensively below. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, these embodiments are provided to make the understanding of the disclosure of the present invention more thorough and comprehensive.

[0040] Embodiment 1

[0041] As Figure 1 shown, the technical model PGCN proposed in this embodiment includes the following steps: PGCN constructs a pyramid network to fuse EEG features at different scales layer by layer. The overall architecture of the model is as Figure 1 shown and can be divided into the following steps:

[0042] (1) Considering the frequent local connections in the brain network, a sparse structure adjacency matrix is constructed based on the spatial distance between electrodes, and GCN is introduced to aggregate local features.

[0043] Local feature aggregation focuses on the frequent local connections in the human brain, constructs an effective information transmission mechanism, and fuses the EEG information of adjacent nodes. For short-range adjacent features, this method first constructs a sparse graph relationship inference based on the relative spatial positions of electrodes, making its degree follow a power distribution with exponential truncation and keeping the adjacency matrix sufficiently sparse. Then a two-layer GCN is constructed to aggregate the features of adjacent nodes. Finally, the original features are fused with the layer-by-layer outputs of the graph convolution to suppress over-smoothing.

[0044] Sparse graph relationship inference: To better describe the connections between different nodes, the method of this embodiment constructs a sparse structure adjacency matrix based on the inverse square of the spatial distance between different nodes:

[0045]

[0046] where d ij is the three-dimensional distance between nodes i and j, and δ is the sparsity factor.

[0047] In the experiment, it is found that the effect is the best when δ is 9; to maintain sparsity, A ij is clipped to be less than 0.1; A ij is taken to be greater than 1 to reduce the weights of self-loops and extremely close neighbors; A is set as a learnable matrix and updated during the training process.

[0048] GCN for local representation aggregation: After constructing the appropriate adjacency matrix A, the corresponding Laplacian matrix is calculated, and GCN is used to transmit information between adjacent nodes:

[0049]

[0050] where H(l) and H(l + 1) are the input and output node representations of the l-th layer respectively; the initial input representation H(0) is the original input feature x, W(l) is the learnable weight matrix, and σ is the activation function.

[0051] To avoid over-smoothing of GCN, only a two-layer local GCN network is designed and cross-layer connections are introduced; the final output of local feature aggregation is:

[0052] X local = concat(X, H (1) , H (2) )

[0053] (2) Drawing on the prior knowledge of neuroscience research, different mesoscopic regions of the brain are constructed, and the positions and features of the virtual mesoscopic centers in each region are calculated; since the features of this region only converge to the virtual mesoscopic center, over-smoothing can be effectively avoided while increasing the receptive field.

[0054] Coarsening of Mesoscopic Regions: Design two mesoscopic regions with different receptive fields. The cerebral cortex is usually divided into four lobes, the frontal lobe, parietal lobe, temporal lobe, and occipital lobe, each responsible for different tasks; for example, the occipital lobe is related to visual processing and interpretation. First, divide the electrodes into five regions; second, since the electrodes located in the temporal lobe (FT7, T7, TP7, FT8, T8, TP8) are important for fatigue detection, a more detailed secondary division is made for the temporal lobe electrodes; finally, the regional division is adjusted to meet the requirements of structural and functional connectivity.

[0055] Mesoscopic Node Relationships: After constructing the brain mesoscopic regions, focus on the functional connectivity within the mesoscopic regions through self-attention. The input is a set of features within each mesoscopic region, where N is the number of nodes in the mesoscopic region and F is the number of features for each node; the attention-based connectivity matrix e can be expressed as:

[0056] e = LeakyReLU((hW)(hW) T )

[0057] Virtual Mesoscopic Center: After calculating the attention-based mesoscopic region connectivity matrix, start constructing the virtual mesoscopic center; calculate the weight coefficient for each mesoscopic region by summing each row of e where Λ i represents the functional connection weight of each node with all other nodes; then start calculating the virtual mesoscopic X enhanced = X meso + embed(P meso )

[0058] p locate = softmax(Λ)P

[0059] m feature = softmax(Λ)h,

[0060] Features and Positions of Regional Centers:

[0061] In the formula is the absolute position of the electrode.

[0062] After calculating the regional center of each mesoscopic region, fuse the features and positions according to different divisions; the fused features and positions are respectively

[0063] Finally, fuse the virtual mesoscopic center with the original electrode nodes through virtual node interpolation to obtain a fused feature and position matrix containing important local and mesoscopic attributes.

[0064] X meso = (concat((X local )) T,(M (1) ) T ,(M (2) ) T )) T

[0065] P meso = (concat(P T ,(P (1) )) T ,(P (2) )) T )) T

[0066] Although mesoscopic partitioning can effectively alleviate the problem that node information cannot be transmitted over long distances, with the increase in the number of nodes, a larger mesoscopic area is prone to information loss.

[0067] The attention mechanism is very effective in constructing global feature relationships. To aggregate long-distance node associations, the method of this embodiment constructs an attention-based absolute position-dependent feature association adjacency matrix to describe the possible connections between each node and eliminate the weak connection relationships between them. After constructing the global electrode connection relationship, this method uses graph convolution for global feature aggregation.

[0068] Global node relationship: To better describe the global node relationship, the method of this embodiment introduces node position encoding. The complete node P meso consists of the original electrode node P and the virtual mesoscopic node P virtual . Since the virtual node also has corresponding features, the method of this embodiment adds the features of all nodes to the position encoding to obtain the position-enhanced node features as follows:

[0069] X enhanced = X meso + embed(P meso )

[0070] After that, the method of this embodiment uses multi-head self-attention with 6 heads to calculate the relationship matrix and uses a learnable weight vector to fuse G into the final attention relationship matrix :

[0071] A global = wG

[0072] The dense attention matrix empirically exacerbates the similarity between node features, but this method deliberately retains the top 20% of the connections in the adjacency matrix to ensure its sparsity.

[0073] After obtaining the global attention-based adjacency matrix A global , the method of this embodiment calculates the corresponding Laplacian matrix and use it for graph convolution:

[0074]

[0075] where O(l) is the input node representation, O(l + 1) is the output node representation, and the initial input representation O(0) is the original input feature X meso ; W(l) is the learnable weight matrix, and σ is the activation function.

[0076] Finally, the method of this embodiment connects the input mesoscopic feature X meso with the global GCN output O (1) to obtain features covering local, mesoscopic, and global receptive fields.

[0077] X global = concat(X meso , O (1) )

[0078] After obtaining the feature representations of each domain in multiple layers, this method sets X global as the input of a three-layer fully connected network for final fatigue detection.

[0079] In the technical solution proposed by the present invention, a real-time monitoring model of EEG cognitive load, PGCN, is established, and its effect is better than other models. The specific effects are shown in Table 1-11 below.

[0080] Table 1 Performance Metrics of the Transformer Model

[0081]

[0082] Table 2 Performance Metrics of the DGCNN Model

[0083]

[0084]

[0085] Table 3 Performance Metrics of the Deep_Forest Model

[0086]

[0087] Table 4 Performance Metrics of the 3DCNN Model

[0088]

[0089]

[0090] Table 5 Performance Metrics of the 2DCNN + GRU Model

[0091]

[0092] Table 6 MLP Model Performance Metrics

[0093]

[0094]

[0095] Table 7 SVM Model Performance Metrics

[0096]

[0097]

[0098] Table 8 MAS-DGAT-Net Model Performance Metrics

[0099]

[0100] Table 9 BLS Model Performance Metrics

[0101]

[0102]

[0103] Table 10 DGAT Model Performance Metrics

[0104]

[0105] Table 11 PGCN Model Performance Metrics

[0106]

[0107]

[0108] It can be seen from the table that the performance of the PGCN model is better than that of other models.

[0109] The above-described embodiments merely represent several implementation manners of the present invention. Their descriptions are relatively specific and detailed, but should not be construed as limiting the scope of the invention patent. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present invention, several modifications and improvements can still be made, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the present invention patent shall be subject to the appended claims.

Claims

1. A method for real-time monitoring of EEG cognitive load, characterized in that: Follow these steps: (1) Considering the frequent local connections of the brain network, a sparse structured adjacency matrix is ​​constructed based on the spatial distance between electrodes, and GCN is introduced to aggregate local features; (2) construct different brain mesoscopic regions and calculate the location and characteristics of the virtual mesoscopic center in each region; (3) In order to balance the importance and economy of long-distance connections, the original electrode nodes are fused with virtual mesoscopic nodes, a sparse global graph connectivity network is constructed using an attention mechanism, and global features are aggregated using graph convolution; (4) Finally, the fused features are input into a 3-layer fully connected network for final fatigue detection.

2. The method for real-time monitoring of EEG cognitive load according to claim 1, characterized in that: The sparse structure adjacency matrix in step (1) is: where d ij is the three-dimensional distance between nodes i and j, and δ is the sparsity factor.

3. The method for real-time monitoring of EEG cognitive load according to claim 1, characterized in that: The method for aggregating local features in step (1) is as follows: first, a sparse graph relationship reasoning is constructed based on the relative spatial positions of the electrodes, so that its degree obeys an exponentially truncated power distribution and the adjacency matrix is ​​kept sufficiently sparse; Then a two-layer GCN is constructed to aggregate the features of adjacent nodes; finally, the original features are fused with the layer-by-layer output of graph convolution to suppress over-smoothing.

4. The method for real-time monitoring of EEG cognitive load according to claim 1, characterized in that: The different brain mesoscopic regions described in step (2) are the frontal lobe, parietal lobe, temporal lobe and occipital lobe.

5. The method for real-time monitoring of EEG cognitive load according to claim 1, characterized in that: The method for constructing the virtual mesoscopic center in step (2) is: calculating the weight coefficient of each mesoscopic region by summing e row by row where Λ i Represents the functional connection weight of each node with all other nodes; then begins to calculate the characteristics and position of the center of the virtual mesoscopic region: In the formula is the absolute position of the electrode; p locate =softmax(Λ)P m feature =softmax(Λ)h, After calculating the regional center of each mesoscopic region, the features and positions are fused according to different divisions; the fused features and positions are Finally, the virtual mesoscopic center is fused with the original electrode node through virtual node interpolation to obtain the fusion feature and position matrix containing important local attributes and mesoscopic attributes: X meso =(concat((x local ) T ,(M (1)T ,(M (2) ) T )) T P meso =(concat(P T ,(P (1) ) T ,(P (2) ) T )) T 。 6. The method for real-time monitoring of EEG cognitive load according to claim 1, characterized in that: Step (3) of constructing a sparse global graph connectivity network is as follows: meso The original electrode node P and the virtual mesoscopic node P virtual Composition; add the features of all nodes to the position code, and get the position-enhanced node features as follows: X enhanced =X mmeso +embed(P meso ); Then use the multi-head self-attention containing 6 heads to calculate the relationship matrix And use a learnable weight vector Fusion G into the final attention relation matrix middle: THE global =wG。 7. The method for real-time monitoring of EEG cognitive load according to claim 1, characterized in that: The graph convolution in step (3) is specifically as follows: based on the attention relationship matrix A global , calculate the corresponding Laplacian matrix For graph convolution: Among them, (l) is the input node representation, O (l+1) is the output node representation, and the initial input representation is O (0) is the original input feature X meso ; W (l) is the learnable weight matrix and σ is the activation function.

8. The method for real-time monitoring of EEG cognitive load according to claim 1, characterized in that: Step (4) is as follows: meso With the global GCN output O (1) Concatenate to obtain features covering local, mesoscopic, and global receptive fields: X global =concat(X meso ,O (1) ) X global It is set as the input of a three-layer fully connected network for final fatigue detection.

Images