Brain cognitive state layered recognition method based on space-time attention

Abstract

The invention discloses a spatial-temporal attention-based brain cognitive state layered recognition method, which introduces a brain cognitive domain recognition module and a brain fine-grained state recognition module to be applied in sequence, performs accurate recognition on a brain fine-grained cognitive state for functional magnetic resonance imaging (fMRI), and can combine dynamic, static, temporal and spatial characteristics to achieve the recognition of the brain fine-grained cognitive state. The problem of low accuracy of brain fine-grained cognitive state recognition is innovatively solved, so that the design scheme is applied to recognition of the brain cognitive state, and a more accurate recognition result can be obtained for the brain fine-grained cognitive state which is difficult to recognize.

Description

Technical Field

[0001] The present invention designs a hierarchical recognition method of brain cognitive states based on spatiotemporal attention, which is used to realize the recognition of fine-grained cognitive states of the brain and belongs to the fields of cognitive neuroscience and computational neuroscience. Background Art

[0002] In the field of neuroscience research, functional magnetic resonance imaging (fMRI) technology has become an important tool for revealing the cognitive state of the brain. By analyzing the blood oxygen level-dependent (BOLD) signal of the brain at rest, researchers can observe the functional connectivity (FC) between brain regions and explore the cognitive state and behavior of the brain.

[0003] Functional connectivity (FC) provides key information for brain state recognition. Current brain state recognition models mainly focus on static and dynamic aspects. Static FC mainly extracts the correlation between brain regions and represents the global data in the form of a graph. Using graph neural networks to learn brain information can extract global information of brain networks. However, this method may not be able to capture the dynamic changes of brain states. To make up for this shortcoming, researchers have begun to pay attention to dynamic FC, which uses techniques such as sliding window correlation (SWC) to analyze changes in brain functional connectivity over a short period of time, thereby revealing the time-varying characteristics of brain activity.

[0004] A recent research trend is to combine static and dynamic brain information to extract more comprehensive brain information. For example, the BrainNetFormer model decodes the brain's cognitive state by combining the characteristics of static and dynamic functional connections and uses a spatial-temporal cross-attention mechanism to improve the accuracy of prediction.

[0005] In practical applications, the brain cognitive state of the subject changes dynamically when performing a task, and different cognitive states within a period of time can be summarized into the same brain cognitive state domain, namely the main task. Existing research mainly focuses on modeling and classifying the entire time series, while ignoring the brain activity state at each time point. In order to achieve the prediction of fine-grained brain cognitive state, it is necessary to extract brain state information moment by moment, which makes it impossible to directly apply the FC time-varying characteristics obtained by SWC to the recognition of fine-grained brain cognitive state.

[0006] The brain's cognitive domains provide important guidance information that can limit the prediction of fine-grained states to a certain range. This guidance information comes from the organization of cognitive tasks, that is, the type of cognitive state of the brain over a period of time, which can first be divided into several cognitive domains, and each moment of each cognitive domain can be divided into a specific state. Therefore, it is very important to use coarse-grained cognitive domain labels to guide the prediction of fine-grained states under each domain.

[0007] In summary, by combining static and dynamic functional connectivity information, and utilizing guidance information from cognitive domains, researchers can more accurately identify and predict the cognitive state of the brain, which is of great significance for understanding how the brain works and for diagnosing and treating neuropsychiatric diseases. Summary of the invention

[0008] The technical problem to be solved by the present invention is to provide a hierarchical recognition method for brain cognitive states based on spatiotemporal attention, which introduces a brain cognitive domain recognition module and a brain fine-grained state recognition module and applies them successively to realize the recognition of fine-grained cognitive states of the brain.

[0009] In order to solve the above technical problems, the present invention adopts the following technical solution: a hierarchical recognition method of brain cognitive state based on spatiotemporal attention, which is used to realize the recognition of fine-grained cognitive state of the brain, comprising the following steps:

[0010] Step A. Take the BOLD signal in the entire time period as input, encode the brain signal at each moment into a vector through the temporal attention layer, obtain the global temporal static representation of the entire brain, and then enter step B;

[0011] Step B. Taking the correlation matrix based on the BOLD signal in the entire time period as input, passing through the spatial encoding layer, obtaining the global spatial static representation, and then entering step C;

[0012] Step C. Using the dynamic spatiotemporal graph data extracted from the BOLD time series data through the sliding window as input, GIN is applied for feature aggregation and spatial fusion to obtain local dynamic features at different times, and then proceed to step D;

[0013] Step D. Taking the local dynamic features obtained in step C, the global temporal static representation vector and the global spatial static representation vector obtained in steps A and B respectively as input, entering the time transfer layer (TTL), using the dot product attention mechanism, obtaining the final graphic readout vector of each sliding window, and then entering step E;

[0014] Step E. Using the final graphic readout vector obtained in step D and the global time static representation vector obtained in step A as input, the local features and global features are fused through the spatiotemporal cross attention module to obtain the brain cognitive domain state recognition result, and then proceed to step F.

[0015] Step F. Taking the BOLD signal as input, according to the brain cognitive domain state recognition result obtained in step E, select the corresponding brain fine-grained cognitive state prediction submodule to obtain the final brain fine-grained cognitive state recognition result.

[0016] As a preferred technical solution of the present invention: the brain cognitive domain state recognition module is used to identify the coarse-grained brain cognitive domain state;

[0017] In step A, the BOLD signal in the entire time period is first used as input, and the long short-term memory (LSTM) model is used in the time attention layer to positionally encode the time series, and encode the brain signal at each moment into a vector:

[0018] X e =XW e +LSTM(XW e )

[0019] X∈R T×N is the original BOLD signal, W e ∈R N×C is a learnable matrix, T is the total length of the input signal, N is the number of brain regions, and C is the hidden dimension. After that, we use a single-head attention layer to extract global information:

[0020] Q=X e W Q ; K = X e W K ; V = X e W V

[0021]

[0022] Through calculation, we get X in step A. global ∈R T×C , which is a global static representation of the brain and contains the brain's temporal information.

[0023] Next, in step B, the correlation matrix based on the BOLD signal in the entire time period is flattened into a vector, and then the vector is sent to the neural network SC composed of a linear layer, a normalization layer and an activation layer:

[0024] g s =SC(Flatten(R s ))

[0025] Where R s ∈R N×N , g s ∈R C , N is the number of brain regions, C is the hidden dimension, and the global space static representation g is obtained in step B s .

[0026] Then, in step C, a sliding window is used to extract dynamic spatiotemporal graph data from the BOLD time series data, and the neighbor information of each node is aggregated by multiplying the adjacency matrix with the node feature matrix through a graph isomorphism network (GIN). At the same time, a self-loop is introduced and adjusted by a learnable parameter. Finally, a multilayer perceptron is used to perform nonlinear transformation on the aggregated features to obtain local dynamic features at different times.

[0027] Next, in step D, the brain global dynamic vector and global static vector obtained from the brain static representation module are introduced to calculate the weight of the brain region. global and the global space static representation vector g s Add them together to get the query vector, and then perform dot product attention calculation on the local dynamic features obtained in step C to get the brain map readout vector at each sampling moment:

[0028]

[0029]

[0030] Among them, SA t ∈R N represents the attention distribution of the brain area at sample point t, is the output of the last layer of the graph convolutional network, x dec,t ∈R C is the final brain map readout vector at sample point t.

[0031] Finally, in step E, a multi-head attention mechanism is used to effectively capture the dynamic interactions between brain regions:

[0032] Q T =X dec W QT ; K T =X global W KT ; V T =X global W VT

[0033]

[0034] H embed ∈R L×C is a fused feature matrix containing rich information. After attention calculation, the feature matrix passes through a fully connected network and a normalized residual connection network. At this point, the global information and local information are fused together, and then the embedding vector of the aggregated information is summed in the time dimension to obtain the brain cognitive domain state representation vector h dyn ∈R C .

[0035] And during the training process of the brain cognitive domain state recognition module, the corresponding cross entropy loss function value is calculated for the brain cognitive domain state representation vector as the training threshold to realize the training of the brain cognitive domain cognitive state module.

[0036] As a preferred technical solution of the present invention: in the graph convolution layer of step C in the brain cognitive domain state recognition module, in order to enhance the model's ability to learn the node's own features, we introduce self-loops and adjust the weight of the self-loops by a learnable parameter ∈. This step is done by converting the adjacency matrix A t It is implemented by adding a unit matrix multiplied by ∈. A multi-layer perceptron (MLP) is then used to perform a nonlinear transformation on the aggregated features, which consists of a series of hidden layers and nonlinear activation functions. At the kth layer, the MLP accepts the feature matrix adjusted by the adjacency matrix and self-loop weights, and passes it through the weight matrix W (k) and the node feature matrix of the k-1th layer The dot product result is linearly transformed, and then a nonlinear activation function is applied to output the local dynamic features of the k-th layer at different times. The calculation formula for the k-th layer is as follows:

[0037]

[0038] Among them, ∈ (k) ∈R is a learnable parameter, I∈R N×N is the identity matrix, A t ∈R N×N is the adjacency matrix at time t, W (k) is the weight matrix of the kth layer, is a learnable matrix from the original input F t , which is mapped from the feature matrix at time t.

[0039] As a preferred technical solution of the present invention: in the step F, the brain fine-grained cognitive state prediction module reuses the temporal attention layer in step A;

[0040] First, in the brain fine-grained cognitive state module, for the brain cognitive domain identification results in step E, a submodule was established for each predicted cognitive state domain, with a total of 7 submodules;

[0041] Next, a submodule is selected for use according to the brain cognitive domain identification results in step E. After the original BOLD signal corresponding to the state domain is input into the corresponding prediction submodule, the fine-grained state feature information of the cycle is extracted according to the steps described in step A;

[0042] Finally, the feature information matrix is ​​passed through a fully connected layer to obtain the probability vector of the fine-grained state, and the vector is dimensionally expanded to obtain the final fine-grained state probability vector.

[0043] And during the training process of the brain fine-grained cognitive state module, the corresponding cross entropy loss function value is calculated for the final fine-grained state probability vector as the training threshold to realize the training of the brain fine-grained cognitive state module.

[0044] Compared with the prior art, the advantages of the present invention are as follows:

[0045] Compared with the existing classification recognition models used in the field of brain state recognition, the brain cognitive domain state recognition accuracy of the present invention is 99.28%, which is not inferior to the best model currently investigated at 99.15%. At the same time, there are few models for identifying fine-grained brain states. The accuracy of the present invention for fine-grained brain states is 85.95%, which is higher than the current best model of 77.44%. In addition, the present invention integrates time and space attention, combines static and dynamic brain information, innovatively proposes the concept of hierarchy, and uses the results of brain cognitive domain recognition to guide and assist in the selection of the corresponding small model for brain fine-grained state recognition, which not only ensures the model's recognition accuracy for the brain cognitive domain, but also improves the model's recognition accuracy for the brain's fine-grained cognitive state. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a schematic diagram of the process of the present invention for identifying the fine-grained cognitive state of the brain;

[0047] Figure 2 It is a schematic diagram of the global static representation extraction module model architecture;

[0048] Figure 3 It is a schematic diagram of the architecture of the dynamic representation and static representation fusion module model;

[0049] Figure 4 It is a heat map and schematic diagram of the results of the fine-grained cognitive state of the brain designed to identify the MOTION domain of the present invention;

[0050] Figure 5 It is a heat map and schematic diagram of the results of the fine-grained cognitive state of the brain designed to identify the EMOTIONAL domain of the present invention. DETAILED DESCRIPTION

[0051] The specific implementation modes of the present invention will be further described in detail below in conjunction with the accompanying drawings.

[0052] Embodiment: The present invention designs a hierarchical recognition method based on spatiotemporal attention to realize the recognition of fine-grained cognitive states of the brain. In practical application, it specifically includes the following steps.

[0053] Step A. Take the BOLD signal in the entire time period as input, encode the brain signal at each moment into a vector through the temporal attention layer, obtain the global temporal static representation of the entire brain, and then enter step B;

[0054] Step B. Taking the correlation matrix based on the BOLD signal in the entire time period as input, passing through the spatial encoding layer, obtaining the global spatial static representation, and then entering step C;

[0055] Step C. Using the dynamic spatiotemporal graph data extracted from the BOLD time series data through the sliding window as input, GIN is applied for feature aggregation and spatial fusion to obtain local dynamic features at different times, and then proceed to step D;

[0056] Step D. Taking the local dynamic features obtained in step C, the global temporal static representation vector and the global spatial static representation vector obtained in steps A and B respectively as input, using the dot product attention mechanism, obtain the final graphic readout vector of each sliding window, and then proceed to step E;

[0057] Step E. Using the final graphic readout vector obtained in step D and the global time static representation vector obtained in step A as input, the local features and global features are fused through the spatiotemporal cross attention module to obtain the brain cognitive domain state recognition result, and then proceed to step F.

[0058] Step F. Taking the BOLD signal as input, according to the brain cognitive domain state recognition result obtained in step E, select the corresponding brain fine-grained cognitive state prediction submodule to obtain the final brain fine-grained cognitive state recognition result.

[0059] For the above-mentioned technical solution, if Figure 2 As shown, step A and step B are designed as follows:

[0060] For step A, the BOLD signal is first passed through a combination module consisting of a linear layer, a batch normalization layer, and an activation function layer for 5 times, and then the input time series is position-encoded through a long short-term memory network, and the dimension is changed from 128 to 512. Then, the position-encoded vector is processed using a single-head attention, and then the global temporal static representation of the brain is obtained through layer normalization and combination modules. global .

[0061] For step B, first obtain the correlation matrix of the brain area, then flatten the input time attention layer, convert it into a 128-dimensional vector through linear transformation, then process it through batch normalization and relu activation function, and then repeat the combined module consisting of the linear transformation layer, batch normalization layer and activation function layer that converts 128 dimensions to 128 dimensions 5 times to obtain the global space static representation g s.

[0062] g s =SC(Flatten(R s ))

[0063] Where R s ∈R N×N , g s ∈R C , N is the number of brain regions, C is the hidden dimension, and SC is a neural network consisting of the above linear layer, normalization layer, and activation layer.

[0064] Among them, Figure 3 As shown, steps C, D, and E are designed as follows:

[0065] For step C, the combined module consisting of graph convolution and MLP is repeated 4 times, and the graph structure information is extracted using graph convolution; for step D, the local dynamic features obtained in step C, the global time static representation vector and the global space static representation vector obtained in steps A and B are used as inputs, and the module consisting of matrix multiplication and accumulation is repeated 4 times to calculate the weights of different brain regions; for step E, the final graphic readout vector obtained in step D and the global time static representation vector obtained in step A are used as inputs, and first pass through the multi-head attention layer, the dimension changes from 128 to 3 times to 384 and then to 128, and then pass through two layer normalization processes, two MLP layers, and the above combination is repeated 4 times to fuse global information and local information, and finally pass through the linear layer to obtain the probability distribution vector of the seven cognitive domains of the brain, and obtain the brain cognitive domain state recognition result. The specific formula for using the multi-head attention mechanism in step E to effectively capture the dynamic interaction between brain regions is as follows:

[0066] Q T =X dec W QT ; K T =X global W KT ; V T =X global W VT

[0067]

[0068] H embed ∈R L×C is the fusion feature matrix, L is the sliding window size, and H embed Summing over this dimension gives h dyn ∈R C , then h dynIt is sent to the linear layer for processing, and then the regularization technology is used to randomly discard (set to zero) the output of some neurons during the training process. The processed vectors are accumulated to obtain the probability distribution vectors of the seven cognitive domains of the brain.

[0069] Among them, in step F, the BOLD signal is used as input, and according to the brain cognitive domain state recognition result obtained in step E, the operation in step B is reused to select the corresponding brain fine-grained cognitive state prediction submodule to obtain the final brain fine-grained cognitive state recognition result. Our design is inspired by the organization of cognitive methods. The brain cognitive state of the subject can be divided into 7 cognitive domains, and the cognitive state at each moment in each cognitive domain can be divided into specific fine-grained cognitive states. We first predict the cognitive domain state of the brain, and then input the BOLD signal into the corresponding fine-grained state prediction module.

[0070] The above-mentioned method designed to identify fine-grained brain states is used in actual application. The HCP data set data is taken as an example to verify the implementation of the design and application of the present invention for identifying fine-grained brain cognitive states.

[0071] Experimental conditions: A computer is selected for the experiment. The computer is configured with NVIDIA TITAN RTX GPU, 64-bit operating system, Python (version 3.10) as the programming language, and Pytorch as the deep learning framework.

[0072] The experimental data is the fMRI dataset of version S1200 in the HCP dataset. The HCP dataset includes brain imaging and behavioral data of more than 1,200 healthy young people. The fMRI data includes 7 cognitive state domains, each of which corresponds to several different experimental conditions. Under different experimental conditions, there are 25 different fine-grained cognitive states. In the preprocessing stage, the human brain neuroanatomy atlas (AAL) is used as a template to divide the brain into 90 brain regions. Then, according to the above design method, the training batch size is set to 16, the number of training rounds is set to 40, the learning rate is set to 0.0005, the hidden dimension is set to 128, the sliding window length is set to 50, the sliding window step size is set to 3, and the model is trained using 5-fold cross-validation.

[0073] In order to verify the recognition accuracy of the present invention for fine-grained brain cognitive states and the reliability of the model design, we conducted comparative experiments, ablation experiments, and attention analysis experiments.

[0074] In comparative experiments, we tested our model using the test set, compared the recognition accuracy with the labels, and compared it with several recently used models for identifying fine-grained cognitive states of the brain.

[0075]

[0076] The results in Table 1 show that the prediction results of the present invention have higher accuracy and are better than the previous models.

[0077] In the ablation experiment, in order to study the role of the spatiotemporal cross-attention model and hierarchical structure, we removed the spatiotemporal cross-attention module in step E and the fine-grained brain cognitive state recognition module in step F in the above method, and performed the recognition task again.

[0078]

[0079] The results in Table II show that the spatiotemporal attention module and the hierarchical recognition architecture guided by coarse-grained brain cognitive domain labels are both very important for recognizing fine-grained brain states.

[0080] In the attention analysis experiments, we visualized temporal attention on fine-grained cognitive state tasks in the cognitive domains of movement and emotion. Figure 4 , 5 As shown, each row represents a temporal pattern and each column represents a sampling time point of a fine-grained cognitive state. The visualized temporal attention clearly reflects the different patterns of different fine-grained cognitive states, and it can be seen that our invention can accurately identify the fine-grained cognitive states of the brain.

[0081] It should be noted that the above embodiments are not intended to limit the protection scope of the present invention, and equivalent changes or substitutions made on the basis of the above technical solutions all fall within the protection scope of the claims of the present invention.

Claims

1. A hierarchical recognition method of brain cognitive states based on spatiotemporal attention, used to achieve fine-grained recognition of brain cognitive states, characterized in that: The steps include: Step A. Take the BOLD signal in the entire time period as input, encode the brain signal at each moment into a vector through the temporal attention layer, obtain the global temporal static representation of the entire brain, and then enter step B; Step B. Taking the correlation matrix based on the BOLD signal in the entire time period as input, passing through the spatial coding layer (SC), obtaining a global spatial static representation, and then entering step C; Step C. Using the dynamic spatiotemporal graph data extracted from the BOLD time series data through the sliding window as input, GIN is applied for feature aggregation and spatial fusion to obtain local dynamic features at different times, and then proceed to step D; Step D. Taking the local dynamic features obtained in step C, the global temporal static representation vector and the global spatial static representation vector obtained in steps A and B respectively as input, the dot product attention mechanism is used to obtain the final graph readout vector of each sliding window, and then proceeding to step E; Step E. Taking the final graphic readout vector obtained in step D and the global time static representation vector obtained in step A as input, the local features and the global features are fused through the spatiotemporal cross attention module to obtain the brain cognitive domain state recognition result, and then proceed to step F; Step F. Taking the BOLD signal as input, according to the brain cognitive domain state recognition result obtained in step E, select the corresponding brain fine-grained cognitive state prediction submodule to obtain the final brain fine-grained cognitive state recognition result.

2. According to claim 1, a method for hierarchical recognition of brain cognitive states based on spatiotemporal attention is characterized in that: In step A, the temporal attention layer is designed based on the long short-term memory network LSTM and the attention mechanism, wherein the input BOLD signal first passes through a multi-layer perceptron (sequentially passes through a linear layer, a normalization layer and a ReLU nonlinear activation, repeated twice) and then enters the LSTM for position encoding, and then passes through a single-headed attention layer, that is, the position encoding matrix is ​​input into the attention layer as a query vector, a numerical vector and a key vector to obtain a global temporal static representation of the brain.

3. A method for hierarchical recognition of brain cognitive states based on spatiotemporal attention according to claim 2, characterized in that: In step A, the BOLD signal in the entire time period is first used as input, and the long short-term memory (LSTM) model is used in the time attention layer to position encode the time series, and the brain signal at each moment is encoded into a vector: X e =XW e +LSTM(XW e ) X∈R T×N is the original BOLD signal, W e ∈R N×C is a learnable matrix, T is the total length of the input signal, N is the number of brain regions, and C is the hidden dimension. After that, we use a single-head attention layer to extract global information: Q=X e W Q ;K=X e W K ;V=X e W V 4. The method for hierarchical recognition of brain cognitive states based on spatiotemporal attention according to claim 1, characterized in that: In step B, the correlation matrix of the brain region in the spatial encoding layer is first flattened into a vector, then sequentially passed through a linear layer, a normalization layer, and finally a ReLU nonlinear activation, and repeated twice, to finally obtain a global spatial static representation of the brain; specifically as follows: In step B, the correlation matrix based on the BOLD signal in the entire time period is flattened into a vector, and then the vector is sent to the neural network SC composed of a linear layer, a normalization layer and an activation layer: g s =SC(Flatten(R s )) Where R s ∈R N×N , g s ∈R C , N is the number of brain regions, C is the hidden dimension, and the global space static representation g is obtained in step B s .

5. The method for hierarchical recognition of brain cognitive states based on spatiotemporal attention according to claim 1, characterized in that: In step C, the graph convolution module is designed based on the graph isomorphism network GIN. The BOLD signal first obtains the dynamic node feature matrix and the brain adjacency matrix of each sampling point through a sliding window method, and then multiplies the adjacency matrix with the node feature matrix to aggregate the neighbor information of each node. At the same time, a learnable parameter is used to introduce and adjust the self-loop. Finally, the aggregated features are nonlinearly transformed through a multi-layer perceptron to learn more complex feature representations.

6. The method for hierarchical recognition of brain cognitive states based on spatiotemporal attention according to claim 1, characterized in that: In step D, the dot product attention mechanism introduces the global temporal static representation vector and the global spatial static representation vector of the brain obtained in steps A and B to calculate the weights of the brain regions. Specifically, the global temporal static representation vector and the global spatial static representation vector of the brain obtained in steps A and B are added to obtain a query vector, and then the local dynamic features obtained in step C are subjected to dot product attention calculation to obtain the brain map readout vector at each sampling moment. Finally, the map readout vectors at each sampling moment are superimposed to obtain the final map readout vector.

7. A method for hierarchical recognition of brain cognitive states based on spatiotemporal attention according to claim 6, characterized in that: In step D, the brain global dynamic vector and global static vector obtained from the brain static representation module are introduced to calculate the weight of the brain region. The brain's global time static representation vector X global and the global space static representation vector g s Add them together to get the query vector, and then perform dot product attention calculation on the local dynamic features obtained in step C to get the brain map readout vector at each sampling moment: Among them, SA t ∈R N represents the attention distribution of the brain area at sample point t, is the output of the last layer of the graph convolutional network, x dec,t ∈R C is the final brain map readout vector at sample point t.

8. The method for hierarchical recognition of brain cognitive states based on spatiotemporal attention according to claim 1, characterized in that: In the step E, the spatiotemporal cross-attention module is designed based on the multi-head self-attention mechanism. The global temporal static representation of the brain obtained in step A is used as the numerical vector and key vector of the attention layer, and the final graph readout vector obtained in step D is used as the query vector of the attention layer. It is input into the attention layer to obtain a fused feature matrix, and then passes through a fully connected network and a normalized residual connection network to obtain the final brain state representation vector. Finally, it passes through a linear feedforward layer to obtain the recognition result of the brain cognitive domain.

9. A method for hierarchical recognition of brain cognitive states based on spatiotemporal attention according to claim 8, characterized in that: In step E, a multi-head attention mechanism is used to effectively capture the dynamic interactions between brain regions: Q T =X dec W QT ;K T =X global W KT ;V T =X global W VT H embed ∈R L×C It is a fusion feature matrix, which contains rich information. After attention calculation, the feature matrix passes through the fully connected network and the normalized residual connection network. At this time, the global information and the local information are fused together, and then the embedding vector of the aggregated information is summed in the time dimension to obtain the brain cognitive domain state representation vector h dyn ∈R C .

10. The method for hierarchical recognition of brain cognitive states based on spatiotemporal attention according to claim 1, characterized in that: In step F, the brain fine-grained cognitive state prediction submodule reuses the temporal attention layer in step A; First, in the brain fine-grained cognitive state module, for the brain cognitive domain identification results in step E, a submodule was established for each predicted cognitive state domain, with a total of 7 submodules; Next, according to the brain cognitive domain recognition results in step E, a submodule is selected for use, and after the original BOLD signal corresponding to the state domain is input into the corresponding prediction submodule, the fine-grained state feature information of the cycle is extracted according to the steps described in step A; Finally, the feature information matrix is ​​passed through a fully connected layer to obtain the probability vector of the fine-grained state, and the vector is dimensionally expanded to obtain the final fine-grained state probability vector. And during the training process of the brain fine-grained cognitive state module, the corresponding cross entropy loss function value is calculated for the final fine-grained state probability vector as the training threshold to realize the training of the brain fine-grained cognitive state module.

Images