A brain network representation learning method for self-attention dynamic graph neural network

Abstract

The present invention discloses a brain network representation learning method of a self-attention dynamic graph neural network, which belongs to the field of artificial intelligence and is applied to smart medical care. First, the EEG signal is subjected to a short-time Fourier transform to obtain the time-frequency characteristics of the EEG signal; then, the multi-channel EEG features are modeled, and the topological structure of the EEG signal is used to learn a more discriminative EEG feature representation; finally, the long short-term memory network is used to extract the temporal dynamic evolution characteristics of the brain network spatial feature representation. The above method realizes effective feature extraction, improves the recognition effect of EEG signals on emotions, and greatly improves the performance of the emotion recognition model.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence and is applied to smart medical treatment, and in particular relates to a brain network representation learning method of a self-attention dynamic graph neural network. Background Art

[0002] In recent years, due to the objectivity of the central nervous system in emotional expression, EEG signals have become one of the most robust clues for identifying and inferring emotions. For EEG signals, the most critical point in building an emotion recognizer is how to extract corresponding deep features from EEG signals.

[0003] The current popular EEG emotion recognition methods mainly extract the spatiotemporal feature representation of a single channel through deep learning models such as convolutional neural networks (CNN) and recurrent neural networks (RNN). However, each channel of EEG signals has complex spatiotemporal intrinsic connections in emotional expression. When extracting EEG signal features, the spatial distribution information of EEG channels and their mutual influence must be considered.

[0004] Therefore, considering the EEG signal feature representation of multi-channel spatiotemporal information fusion plays an important role in improving EEG signal emotion recognition. Since multi-channel EEG information plays an important role in emotion recognition, people often use multi-channel data fusion to model emotions, which also greatly improves the performance of emotion recognition models. Summary of the invention

[0005] Aiming at the problem that the dynamic spatiotemporal evolution of multi-channel EEG signals lacks effective feature extraction, which has a significant impact on the performance improvement of emotion recognition systems, the present invention provides a brain network representation learning method of a self-attention dynamic graph neural network.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is: a brain network representation learning method of a self-attention dynamic graph neural network, comprising the following steps:

[0007] Step 101, performing short-time Fourier transform (STFT) on the EEG signal to obtain the time-frequency characteristics of the EEG signal to reduce the influence of redundant information in the original EEG signal on emotion recognition;

[0008] Step 102, modeling multi-channel EEG features through a brain network, and using the topological structure of the EEG signal to learn a more discriminative EEG feature representation;

[0009] Step 103, defining whether the channels are adjacent to each other by the spatial distance between different channels;

[0010] Step 104, using a long short-term memory network (LSTM) to extract the temporal dynamic evolution characteristics of the brain network spatial feature representation;

[0011] The brain network graphs at different times are expanded, and the importance weight of the brain network graph at each time is obtained through LSTM. The weighted sum is taken according to the importance of each moment to focus on a more distinctive brain network structure.

[0012] In step 101, the window length of the Fourier transform is 128, the complement length is 128, and the number of Fourier transform operators in each transform is 128.

[0013] In step 102, in order to reflect the relationship between the nodes of the graph, the graph attention network (GAT) is used to aggregate the features of adjacent nodes. The specific calculation method is:

[0014]

[0015]

[0016] Among them, N v = {u∈V:(u,v)∈ε} is the set of neighboring nodes of node v; W s ∈R D×F is the shared transformation matrix applied to each node; a∈R 2D is a weight vector parameter; A uv is the adjacency matrix of the graph; σ is the nonlinear activation function;

[0017] For a single channel feature g v ∈G t Through the above calculations, the weights between adjacent channels are obtained, and the result after weighted average is used as the final representation of this channel; after passing through GAT, each channel aggregates spatial features.

[0018] In step 103, it is specifically defined as:

[0019]

[0020] where d ij Represents the spatial distance between nodes i and j. The mean distance E(L) between the node and all other nodes is used as the threshold. All nodes with a distance less than this distance are considered adjacent nodes. The reciprocal of the distance is used as the weight between the two points. The weight of the node and itself is the reciprocal of the mean distance of all adjacent nodes.

[0021] In step 104, the brain network structure includes both the spatial structural features and the temporal features of the EEG signal.

[0022] Due to the adoption of the above technical scheme, the present invention aggregates the spatial information of the brain network through GAT, and then uses LSTM to extract the temporal evolution information, so as to achieve effective feature extraction and predict the final emotional state, thereby improving the recognition effect of EEG signals on emotions and greatly improving the performance of the emotion recognition model. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The present invention will be described in detail below with reference to the accompanying drawings and in combination with examples, and the advantages and implementation methods of the present invention will become more obvious. The contents shown in the accompanying drawings are only used to explain the present invention and do not constitute any limitation to the present invention in any sense. In the accompanying drawings:

[0024] Figure 1 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0025] like Figure 1 As shown, a brain network representation learning method of a self-attention dynamic graph neural network includes the following steps:

[0026] Step 101, performing short-time Fourier transform (STFT) on the EEG signal to obtain the time-frequency characteristics of the EEG signal to reduce the influence of redundant information in the original EEG signal on emotion recognition; wherein the window length of the Fourier transform is 128, the padding length is 128, and the number of Fourier transform operators in each transformation is 128;

[0027] Step 102, modeling multi-channel EEG features through brain networks, using the topological structure of EEG signals to learn more discriminative EEG feature representations; in order to reflect the association between the nodes of the graph, the graph attention network (GAT, which proposes to use the attention mechanism to weighted sum the features of neighboring nodes in the graph, and the weights of the features of neighboring nodes completely depend on the node features and are independent of the graph structure) is used to aggregate the features of adjacent nodes. The specific calculation method is:

[0028]

[0029]

[0030] Among them, N v = {u∈V:(u,v)∈ε} is the set of neighboring nodes of node v; W s ∈R D×F is the shared transformation matrix applied to each node; a∈R 2D is a weight vector parameter; A uv is the adjacency matrix of the graph; σ is the nonlinear activation function;

[0031] For a single channel feature g v ∈G t, through the above calculation, the weights between adjacent channels are obtained, and the result after weighted average is used as the final representation of this channel; after passing through GAT, each channel aggregates spatial features;

[0032] Step 103, defining whether the channels are adjacent to each other by the spatial distance between different channels, which is specifically defined as:

[0033]

[0034] where d ij Represents the spatial distance between nodes i and j. The mean distance E(L) between the node and all other nodes is used as the threshold. Any node with a distance less than this distance is considered an adjacent node. The reciprocal of the distance is used as the weight between the two points. The weight of the node and itself is the reciprocal of the mean distance of all adjacent nodes.

[0035] Step 104, using a long short-term memory network (LSTM, a special RNN that can be used to solve the long-term dependency problem in a recurrent neural network. In many tasks, the recurrent neural network with an LSTM structure performs better than a standard recurrent neural network) to extract the temporal dynamic evolution characteristics of the brain network spatial feature representation;

[0036] Expand the brain network graphs at different times, and use LSTM to obtain the importance weight of the brain network graph at each moment. The weighted sum is calculated based on the importance of each moment to focus on a more differentiated brain network structure.

[0037] The brain network structure contains both the spatial structural characteristics and temporal characteristics of EEG signals.

[0038] It can be seen that the present invention first obtains time-frequency information by short-time Fourier transforming the original signal. Then the time-frequency information of each channel is divided according to the time window size of T to form multiple EEG signal frame sequences. After that, the spectrum diagram of each frame at each moment represents the characteristics of the brain network node at this moment, and the connection between two different nodes represents an edge of the brain network diagram. We initialize each edge with the inverse of the Euclidean distance between two different nodes. Therefore, we can get a brain network sequence framework based on spectrum diagram representation. Then, the graph representation of the brain network at each moment of each frame is learned using the graph-based self-attention mechanism. LSTM is then used to capture the long-term dynamic characteristics of the brain network graph representation sequence. Finally, the learned deep features are input into the deep neural network, and Softmax classification is used to predict the final emotional state.

[0039] The embodiments of the present invention are described in detail above, but the contents described are only preferred embodiments of the present invention and cannot be considered to limit the scope of implementation of the present invention. All equivalent changes and improvements made within the scope of the present invention should still fall within the scope of this patent.

Claims

1. A brain network representation learning method for a self-attention dynamic graph neural network, characterized in that: The steps include: Step 101, performing short-time Fourier transform on the EEG signal to obtain the time-frequency characteristics of the EEG signal to reduce the influence of redundant information in the original EEG signal on emotion recognition; Step 102, modeling multi-channel EEG features through a brain network, and using the topological structure of the EEG signal to learn a more discriminative EEG feature representation; Step 103, defining whether the channels are adjacent to each other by the spatial distance between different channels; Step 104, using the long short-term memory network to extract the temporal dynamic evolution characteristics of the brain network spatial feature representation; Expand the brain network graphs at different times, and obtain the importance weight of the brain network graph at each moment through the long short-term memory network. The weighted sum is calculated according to the importance of each moment to focus on a more distinctive brain network structure. In step 102, in order to reflect the relationship between the nodes of the graph, the graph attention network is used to aggregate the features of adjacent nodes. The specific calculation method is: in, is the set of adjacent nodes of node v; is the shared transformation matrix applied to each node; is a weight vector parameter; is the adjacency matrix of the graph; is a nonlinear activation function; For a single channel feature Through the above calculations, the weights between adjacent channels are obtained, and the result after weighted average is used as the final representation of this channel; after passing through the graph attention network, each channel aggregates spatial features.

2. The method for learning brain network representation of a self-attention dynamic graph neural network according to claim 1, characterized in that: In step 101, the window length of the Fourier transform is 128, the complement length is 128, and the number of Fourier transform operators in each transform is 128.

3. The method for learning brain network representation of a self-attention dynamic graph neural network according to claim 1, characterized in that: In step 103, it is specifically defined as: in Represents the spatial distance between nodes i and j, and the average of the distances between the node and all other nodes As a threshold, all nodes with a distance less than this distance are considered adjacent nodes, and the reciprocal of the distance is used as the weight between the two points. The weight of the node and itself is the reciprocal of the mean distance of all adjacent nodes.

4. The method for learning brain network representation of a self-attention dynamic graph neural network according to claim 1, characterized in that: In step 104, the brain network structure includes both the spatial structural features and the temporal features of the EEG signal.

Images