A hierarchical recognition method for brain cognitive states based on spatiotemporal attention

Abstract

The present invention discloses a hierarchical recognition method for brain cognitive states based on spatiotemporal attention. The method introduces a brain cognitive domain recognition module and a brain fine-grained state recognition module, which are applied successively. The method accurately recognizes the brain's fine-grained cognitive states based on functional magnetic resonance imaging (fMRI). The method can combine dynamic and static, temporal and spatial features to innovatively solve the problem of low accuracy in the recognition of the brain's fine-grained cognitive states. Therefore, the design scheme is applied to the recognition of brain cognitive states, and more accurate recognition results can be obtained for difficult-to-identify fine-grained cognitive states of the brain.

Description

Technical Field

[0001] The present invention designs a hierarchical recognition method for brain cognitive states based on spatiotemporal attention, which is used to realize the recognition of fine-grained cognitive states of the brain and belongs to the fields of cognitive neuroscience and computational neuroscience. Background Art

[0002] In the field of neuroscience research, functional magnetic resonance imaging (fMRI) has become an important tool for revealing the brain's cognitive state. By analyzing the blood oxygen level-dependent (BOLD) signal during resting brain activity, researchers can observe the functional connectivity (FC) between brain regions and explore the brain's cognitive state and behavior.

[0003] Functional connectivity (FC) provides key information for brain state recognition. Current brain state recognition models mainly focus on static and dynamic aspects. Static FC mainly extracts correlations between brain regions and represents global data in the form of a graph. Using graph neural networks to learn brain information can extract global information about brain networks. However, this method may not be able to capture the dynamic changes in brain state. To address this shortcoming, researchers have begun to focus on dynamic FC, which uses techniques such as sliding window correlation (SWC) to analyze changes in brain functional connectivity over short periods of time, thereby revealing the time-varying characteristics of brain activity.

[0004] A recent research trend is to combine static and dynamic brain information to extract more comprehensive brain information. For example, the BrainNetFormer model decodes brain cognitive states by combining the characteristics of static and dynamic functional connectivity and uses a spatial-temporal cross-attention mechanism to improve prediction accuracy.

[0005] In practical applications, a subject's brain cognitive state changes dynamically while performing a task. Different cognitive states over a period of time can be summarized into the same brain cognitive state domain, namely, the primary task. Existing research primarily focuses on modeling and classifying the entire time series, while neglecting the brain activity state at each time point. To achieve fine-grained prediction of brain cognitive states, it is necessary to extract brain state information moment by moment. This makes it impossible to directly apply the time-varying FC characteristics obtained by SWC to the identification of fine-grained brain cognitive states.

[0006] The brain's cognitive domains provide important guidance, limiting the prediction of fine-grained states to a certain range. This guidance comes from the organization of cognitive tasks, namely, the types of cognitive states the brain experiences over time. These can be divided into several cognitive domains, and each moment in each cognitive domain can be classified as a specific state. Therefore, it is crucial to use coarse-grained cognitive domain labels to guide the prediction of fine-grained states within each domain.

[0007] In summary, by combining static and dynamic functional connectivity information and utilizing guidance information from cognitive domains, researchers can more accurately identify and predict the cognitive state of the brain, which is of great significance for understanding how the brain works and for diagnosing and treating neuropsychiatric diseases. Summary of the Invention

[0008] The technical problem to be solved by the present invention is to provide a hierarchical recognition method for brain cognitive states based on spatiotemporal attention, which introduces two layers of brain cognitive domain recognition module and brain fine-grained state recognition module and applies them successively, so as to realize the recognition of fine-grained cognitive states of the brain.

[0009] In order to solve the above technical problems, the present invention adopts the following technical solution: a hierarchical recognition method of brain cognitive state based on spatiotemporal attention, which is used to realize the recognition of fine-grained cognitive state of the brain, comprising the following steps:

[0010] Step A: Taking the BOLD signal over the entire time period as input, the temporal attention layer encodes the brain signal at each moment into a vector, obtaining a global temporal static representation of the entire brain, and then proceeding to step B.

[0011] Step B. Taking the correlation matrix based on the BOLD signal over the entire time period as input, passing it through the spatial encoding layer to obtain a global spatial static representation, and then proceeding to step C;

[0012] Step C. Using the dynamic spatiotemporal graph data extracted from the BOLD time series data through the sliding window as input, GIN is applied for feature aggregation and spatial fusion to obtain local dynamic features at different moments, and then proceed to step D;

[0013] Step D: Take the local dynamic features obtained in step C, the global temporal static representation vector obtained in step A and the global spatial static representation vector obtained in step B respectively as input, enter the temporal transfer layer (TTL), use the dot product attention mechanism to obtain the final image readout vector of each sliding window, and then enter step E;

[0014] Step E. Using the final graphic readout vector obtained in step D and the global time static representation vector obtained in step A as input, the local features and global features are fused through the spatiotemporal cross attention module to obtain the brain cognitive domain state recognition result, and then proceed to step F.

[0015] Step F. Using the BOLD signal as input, according to the brain cognitive domain state recognition result obtained in step E, select the corresponding brain fine-grained cognitive state prediction submodule to obtain the final brain fine-grained cognitive state recognition result.

[0016] As a preferred technical solution of the present invention: the brain cognitive domain state recognition module is used to identify the coarse-grained brain cognitive domain state;

[0017] In step A, the BOLD signal over the entire time period is first used as input, and the long short-term memory (LSTM) model is used in the temporal attention layer to position-encode the time series, encoding the brain signal at each moment into a vector:

[0018] X e =XW e +LSTM(XW e )

[0019] X∈R T×N is the original BOLD signal, W e ∈R N×C is a learnable matrix, T is the total length of the input signal, N is the number of brain regions, and C is the hidden dimension. Afterwards, we use a single-head attention layer to extract global information:

[0020] Q=X e W Q ; K = X e W K ; V = X e W V

[0021]

[0022] Through calculation we get X in step A global ∈R T×C , which is a global static representation of the brain and contains the brain's temporal information.

[0023] Next, in step B, the correlation matrix based on the BOLD signal over the entire time period is flattened into a vector, which is then fed into a neural network SC consisting of a linear layer, a normalization layer, and an activation layer:

[0024] g s =SC(Flatten(R s ))

[0025] Where R s ∈R N×N , g s ∈R C , N is the number of brain regions, C is the hidden dimension, and the global space static representation g is obtained in step B s .

[0026] Then, in step C, a sliding window is used to extract dynamic spatiotemporal graph data from the BOLD time series data. A graph isomorphism network (GIN) is used to aggregate each node's neighbor information by multiplying the adjacency matrix with the node feature matrix. Self-loops are introduced and regulated by a learnable parameter. Finally, a multilayer perceptron is used to perform nonlinear transformations on the aggregated features to obtain local dynamic features at different moments.

[0027] Next, in step D, the brain global dynamic vector and global static vector obtained from the brain static representation module are introduced to calculate the weight of the brain region. global and the global space static representation vector g s Add them together to get the query vector, then perform dot product attention calculation on the local dynamic features obtained in step C to get the brain map readout vector at each sampling moment:

[0028]

[0029]

[0030] Among them SA t ∈R N represents the attention distribution of the brain area at sample point t, is the output of the last layer of the graph convolutional network, x dec,t ∈R C is the final brain map readout vector at sample point t.

[0031] Finally, in step E, a multi-head attention mechanism is used to effectively capture the dynamic interactions between brain regions:

[0032] Q T =X dec W QT ;K T =X global W KT ; V T =X global W VT

[0033]

[0034] H embed ∈R L×C It is a fusion feature matrix that contains rich information. After attention calculation, the feature matrix passes through the fully connected network and the normalized residual connection network. At this time, the global information and local information are fused together, and then the embedding vector of the aggregated information is summed in the time dimension to obtain the brain cognitive domain state representation vector h dyn ∈R C .

[0035] In the process of training the brain cognitive domain state recognition module, the corresponding cross entropy loss function value is calculated for the brain cognitive domain state representation vector and used as the training threshold to realize the training of the brain cognitive domain cognitive state module.

[0036] As a preferred technical solution of the present invention: in the graph convolution layer of step C in the brain cognitive domain state recognition module, in order to enhance the model's ability to learn the node's own characteristics, we introduce self-loops and adjust the weight of the self-loops by a learnable parameter ∈. This step is achieved by converting the adjacency matrix A t It is implemented by adding a unit matrix multiplied by ∈. A multi-layer perceptron (MLP) is then used to perform nonlinear transformation on the aggregated features, which consists of a series of hidden layers and nonlinear activation functions. At the kth layer, the MLP accepts the feature matrix adjusted by the adjacency matrix and self-loop weights, and transforms it through the weight matrix W (k) and the node feature matrix of the k-1th layer The dot product result is linearly transformed, and then a nonlinear activation function is applied to output the local dynamic features of the k-th layer at different times. The calculation formula for the k-th layer is as follows:

[0037]

[0038] Among them, ∈ (k) ∈R is a learnable parameter, I∈R N×N is the identity matrix, A t ∈R N×N is the adjacency matrix at time t, W (k) is the weight matrix of the kth layer, is a learnable matrix from the original input F t , which is mapped from the feature matrix at time t.

[0039] As a preferred technical solution of the present invention: in step F, the brain fine-grained cognitive state prediction module reuses the temporal attention layer in step A;

[0040] First, in the brain fine-grained cognitive state module, for the brain cognitive domain identification results in step E, a submodule was established for each predicted cognitive state domain, with a total of 7 submodules;

[0041] Next, a submodule is selected based on the brain cognitive domain identification results in step E. After the original BOLD signal corresponding to the state domain is input into the corresponding prediction submodule, the fine-grained state feature information of the cycle is extracted according to the steps described in step A;

[0042] Finally, the feature information matrix is ​​passed through a fully connected layer to obtain the probability vector of the fine-grained state, and the vector is dimensionally expanded to obtain the final fine-grained state probability vector.

[0043] In the process of training the brain's fine-grained cognitive state module, the corresponding cross-entropy loss function value is calculated for the final fine-grained state probability vector and used as the training threshold to realize the training of the brain's fine-grained cognitive state module.

[0044] Compared with the prior art, the advantages of the present invention are as follows:

[0045] Compared to existing classification and recognition models used in the field of brain state recognition, the present invention's brain cognitive domain state recognition accuracy of 99.28% is comparable to the best currently investigated model's 99.15%. Furthermore, while there are currently few models for recognizing fine-grained brain states, the present invention's fine-grained state recognition accuracy of 85.95% surpasses the current best model's 77.44%. Furthermore, the present invention integrates temporal and spatial attention, combines static and dynamic brain information, and innovatively proposes a hierarchical concept. Using brain cognitive domain recognition results to guide and assist in the selection of corresponding small models for fine-grained brain state recognition, this not only ensures the model's recognition accuracy for the brain cognitive domain but also improves its recognition accuracy for fine-grained brain states. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 This is a schematic diagram of the process of the present invention for identifying fine-grained cognitive states of the brain;

[0047] Figure 2 It is a schematic diagram of the global static representation extraction module model architecture;

[0048] Figure 3 This is a schematic diagram of the architecture of the dynamic representation and static representation fusion module model;

[0049] Figure 4 This is a heat map and schematic diagram of the results of the fine-grained cognitive state of the brain designed to identify the MOTION domain in the present invention;

[0050] Figure 5 It is a heat map and schematic diagram of the results of the fine-grained cognitive state of the brain designed by the present invention to identify the EMOTIONAL domain. DETAILED DESCRIPTION

[0051] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0052] Example: The present invention designs a hierarchical recognition method based on spatiotemporal attention to realize the recognition of fine-grained cognitive states of the brain. In practical application, it specifically includes the following steps.

[0053] Step A: Taking the BOLD signal over the entire time period as input, the temporal attention layer encodes the brain signal at each moment into a vector, obtaining a global temporal static representation of the entire brain, and then proceeding to step B.

[0054] Step B. Taking the correlation matrix based on the BOLD signal over the entire time period as input, passing it through the spatial encoding layer to obtain a global spatial static representation, and then proceeding to step C;

[0055] Step C. Using the dynamic spatiotemporal graph data extracted from the BOLD time series data through the sliding window as input, GIN is applied for feature aggregation and spatial fusion to obtain local dynamic features at different moments, and then proceed to step D;

[0056] Step D. Using the local dynamic features obtained in step C, the global temporal static representation vector obtained in step A and the global spatial static representation vector obtained in step B respectively as input, a dot product attention mechanism is used to obtain the final image readout vector of each sliding window, and then proceed to step E.

[0057] Step E. Using the final graphic readout vector obtained in step D and the global time static representation vector obtained in step A as input, the local features and global features are fused through the spatiotemporal cross attention module to obtain the brain cognitive domain state recognition result, and then proceed to step F.

[0058] Step F. Using the BOLD signal as input, according to the brain cognitive domain state recognition result obtained in step E, select the corresponding brain fine-grained cognitive state prediction submodule to obtain the final brain fine-grained cognitive state recognition result.

[0059] For the technical solution designed above, if Figure 2 As shown, step A and step B are designed as follows:

[0060] For step A, the BOLD signal is first passed through a combination module consisting of a linear layer, a batch normalization layer, and an activation function layer for 5 times. Then, the input time series is position-encoded through a long short-term memory network, and the dimension is changed from 128 to 512. Then, single-head attention is used to process the position-encoded vector, and then layer normalization and combination modules are used to obtain the global temporal static representation of the brain X global .

[0061] For step B, we first obtain the correlation matrix of the brain area, then flatten the input temporal attention layer, convert it into a 128-dimensional vector through linear transformation, and then process it through batch normalization and relu activation function. Then, the combined module consisting of the linear transformation layer, batch normalization layer and activation function layer that converts 128 dimensions to 128 dimensions is repeated 5 times to obtain the global spatial static representation g s.

[0062] g s =SC(Flatten(R s ))

[0063] Where R s ∈R N×N , g s ∈R C , N is the number of brain regions, C is the hidden dimension, and SC is a neural network consisting of the above linear layer, normalization layer, and activation layer.

[0064] Among them, such as Figure 3 As shown, steps C, D, and E are designed as follows:

[0065] For step C, the combined module consisting of graph convolution and MLP is repeated 4 times, and the graph structure information is extracted using graph convolution; for step D, the local dynamic features obtained in step C, the global time static representation vector and the global space static representation vector obtained in steps A and B respectively are used as input, and the module consisting of matrix multiplication and accumulation is repeated 4 times to calculate the weights of different brain regions; for step E, the final graphic readout vector obtained in step D and the global time static representation vector obtained in step A are used as input. First, the multi-head attention layer is used, and the dimension is expanded from 128 to 3 times to 384 and then to 128. Then, it is normalized twice and then MLP layers are used twice. The above combination is repeated 4 times to fuse global and local information. Finally, it is passed through a linear layer to obtain the probability distribution vectors of the seven cognitive domains of the brain and the brain cognitive domain state recognition results. The specific formula for using the multi-head attention mechanism in step E to effectively capture the dynamic interactions between brain regions is as follows:

[0066] Q T =X dec W QT ;K T =X global W KT ; V T =X global W VT

[0067]

[0068] H embed ∈R L×C is the fusion feature matrix, L is the sliding window size, and H embed Sum across this dimension to get h dyn ∈R C , then h dynIt is sent to the linear layer for processing, and then regularization technology is used to randomly discard (set to zero) the output of some neurons during the training process. The processed vectors are accumulated to obtain the probability distribution vectors of the seven cognitive domains of the brain.

[0069] In step F, the BOLD signal is used as input. Based on the brain cognitive domain state recognition result obtained in step E, the operations in step B are reused to select the corresponding fine-grained brain cognitive state prediction submodule to obtain the final fine-grained brain cognitive state recognition result. Our design is inspired by the organization of cognitive modes. The subject's brain cognitive state can be first divided into seven cognitive domains. The cognitive state at each moment in each cognitive domain can be further divided into specific fine-grained cognitive states. We first predict the brain cognitive domain state and then input the BOLD signal into the corresponding fine-grained state prediction module.

[0070] The above-mentioned method for identifying fine-grained brain states is used in actual applications. The HCP dataset data is used as an example to verify the implementation of the design and application of the present invention for identifying fine-grained cognitive states of the brain.

[0071] Experimental conditions: A computer is selected for the experiment. The computer is configured with NVIDIA TITAN RTX GPU, 64-bit operating system, Python (version 3.10) as the programming language, and Pytorch as the deep learning framework.

[0072] The experimental data is the fMRI dataset (version S1200) from the HCP dataset. The HCP dataset includes brain imaging and behavioral data from over 1,200 healthy young adults. The fMRI data includes seven cognitive state domains, each corresponding to several different experimental conditions. Under different experimental conditions, a total of 25 different fine-grained cognitive states can be observed. In the preprocessing stage, the human brain was divided into 90 regions using the Human Anatomy Atlas (AAL) as a template. Following the design described above, the training batch size was set to 16, the number of training epochs to 40, the learning rate to 0.0005, the hidden dimension to 128, the sliding window length to 50, the sliding window step size to 3, and the model was trained using 5-fold cross-validation.

[0073] In order to test the recognition accuracy of fine-grained brain cognitive states and the reliability of the model design of this invention, we conducted comparative experiments, ablation experiments, and attention analysis experiments.

[0074] In comparative experiments, we tested our model using the test set, compared the recognition accuracy with the labels, and compared it with several recently used models for identifying fine-grained cognitive states of the brain.

[0075]

[0076] The results in Table 1 show that the prediction results of the present invention have higher accuracy and are better than those of previous models.

[0077] In the ablation experiment, in order to study the role of the spatiotemporal cross-attention model and hierarchical structure, we removed the spatiotemporal cross-attention module in step E and the fine-grained brain cognitive state recognition module in step F in the above method, and performed the recognition task again.

[0078]

[0079] The results in Table II show that the spatiotemporal attention module and the hierarchical recognition architecture guided by coarse-grained brain cognitive domain labels are both very important for recognizing fine-grained brain states.

[0080] In the attention analysis experiments, we visualized the temporal attention of fine-grained cognitive state tasks in the cognitive domains of movement and emotion. Figure 4 、 5 As shown, each row represents a temporal pattern, and each column represents a sampling time point of a fine-grained cognitive state. The visualized temporal attention clearly reflects the different patterns of different fine-grained cognitive states. It can also be seen that our invention can accurately identify the fine-grained cognitive states of the brain.

[0081] It should be noted that the above embodiments are not intended to limit the scope of protection of the present invention, and equivalent changes or substitutions made on the basis of the above technical solutions fall within the scope of protection of the claims of the present invention.

Claims

1. A hierarchical recognition method for brain cognitive states based on spatiotemporal attention, used to achieve fine-grained recognition of brain cognitive states, characterized in that: The steps include: Step A: Using the BOLD signal over the entire time period as input, the temporal attention layer encodes the brain signal at each moment into a vector, obtaining a global temporal static representation of the entire brain, and then proceeding to step B. Step B. Take the correlation matrix based on the BOLD signal over the entire time period as input, pass it through the spatial encoding layer, obtain a global spatial static representation, and then proceed to step C; Step C. Using the dynamic spatiotemporal graph data extracted from the BOLD time series data using a sliding window as input, apply GIN to perform feature aggregation and spatial fusion to obtain local dynamic features at different moments, and then proceed to Step D. Step D. Using the local dynamic features obtained in step C, the global temporal static representation vector obtained in steps A and B, and the global spatial static representation vector obtained in steps B, respectively, as input, a dot-product attention mechanism is used to obtain the final image readout vector for each sliding window, and then proceed to step E. Step E. Using the final image readout vector obtained in Step D and the global temporal static representation vector obtained in Step A as input, the spatiotemporal cross-attention module fuses local and global features to obtain the brain cognitive domain state recognition result, and then proceeds to Step F. Step F. Using the BOLD signal as input, based on the brain cognitive domain state recognition result obtained in step E, select the corresponding brain fine-grained cognitive state prediction submodule to obtain the final brain fine-grained cognitive state recognition result.

2. The method for hierarchical recognition of brain cognitive states based on spatiotemporal attention according to claim 1, characterized in that: In step A, the temporal attention layer is designed based on the long short-term memory network LSTM and the attention mechanism, wherein the input BOLD signal first passes through the multi-layer perceptron and then enters the LSTM for position encoding, and then passes through the single-head attention layer, that is, the position encoding matrix is ​​input into the attention layer as the query vector, numerical vector, and key vector to obtain the global temporal static representation of the brain.

3. The method for hierarchical recognition of brain cognitive states based on spatiotemporal attention according to claim 2, characterized in that: In step A, the BOLD signal over the entire time period is first used as input, and the long short-term memory (LSTM) model is used in the temporal attention layer to position-encode the time series, encoding the brain signal at each moment into a vector: is the original BOLD signal, is a learnable matrix, T is the total length of the input signal, N is the number of brain regions, and C is the hidden dimension. After that, we use a single-head attention layer to extract global information: 。 4. The method for hierarchical recognition of brain cognitive states based on spatiotemporal attention according to claim 1, characterized in that: In step B, the correlation matrix of the brain region in the spatial encoding layer is first flattened into a vector, then sequentially passed through a linear layer, a normalization layer, and finally a ReLU nonlinear activation, and repeated twice to finally obtain a global spatial static representation of the brain; specifically, as follows: In step B, the correlation matrix based on the BOLD signal over the entire time period is flattened into a vector, which is then fed into a neural network SC consisting of a linear layer, a normalization layer, and an activation layer: In the formula is the number of brain regions, C is the hidden dimension, and the global space static representation is obtained in step B. .

5. The method for hierarchical recognition of brain cognitive states based on spatiotemporal attention according to claim 1, characterized in that: In step C, the graph convolution module is designed based on the graph isomorphism network (GIN). The BOLD signal first obtains the dynamic node feature matrix and the brain adjacency matrix of each sampling point through a sliding window method. The adjacency matrix is ​​then multiplied by the node feature matrix to aggregate the neighbor information of each node. At the same time, a learnable parameter is used to introduce and adjust self-loops. Finally, the aggregated features are nonlinearly transformed through a multilayer perceptron to learn more complex feature representations.

6. The method for hierarchical recognition of brain cognitive states based on spatiotemporal attention according to claim 1, characterized in that: In step D, the dot product attention mechanism introduces the global temporal static representation vector and the global spatial static representation vector of the brain obtained in steps A and B to calculate the weights of the brain regions. Specifically, the global temporal static representation vector and the global spatial static representation vector of the brain obtained in steps A and B are added together to obtain a query vector, and then the dot product attention calculation is performed on the local dynamic features obtained in step C to obtain the brain map readout vector at each sampling moment. Finally, the map readout vectors at each sampling moment are superimposed to obtain the final map readout vector.

7. The method for hierarchical recognition of brain cognitive states based on spatiotemporal attention according to claim 6, characterized in that: In step D, the brain global dynamic vector and global static vector obtained from the brain static representation module are introduced to calculate the weight of the brain region. The brain's global time static representation vector and global space static representation vector Add them together to get the query vector, then perform dot product attention calculation on the local dynamic features obtained in step C to get the brain map readout vector at each sampling moment: in represents the attention distribution of the brain area at sample point t, is the output of the last layer of the graph convolutional network, is the final brain map readout vector at sample point t.

8. The method for hierarchical recognition of brain cognitive states based on spatiotemporal attention according to claim 1, characterized in that: In step E, the spatiotemporal cross attention module is designed based on the multi-head self-attention mechanism. The global temporal static representation of the brain obtained in step A is used as the numerical vector and key vector of the attention layer, and the final graph readout vector obtained in step D is used as the query vector of the attention layer. The vector is input into the attention layer to obtain a fusion feature matrix. After that, the module passes through a fully connected network and a normalized residual connection network to obtain the final brain state representation vector. Finally, the module passes through a linear feedforward layer to obtain the recognition result of the brain cognitive domain.

9. The method for hierarchical recognition of brain cognitive states based on spatiotemporal attention according to claim 8, characterized in that: In step E, a multi-head attention mechanism is used to effectively capture the dynamic interactions between brain regions: It is a fusion feature matrix containing rich information. After attention calculation, the feature matrix passes through the fully connected network and the normalized residual connection network. At this time, the global information and local information are fused together, and then the embedding vector of the aggregated information is summed in the time dimension to obtain the brain cognitive domain state representation vector .

10. The method for hierarchical recognition of brain cognitive states based on spatiotemporal attention according to claim 1, characterized in that: In step F, the brain fine-grained cognitive state prediction submodule reuses the temporal attention layer in step A; First, in the brain fine-grained cognitive state module, for the brain cognitive domain identification results in step E, a submodule was established for each predicted cognitive state domain, with a total of 7 submodules; Next, based on the brain cognitive domain identification results in step E, a submodule is selected to use. After the original BOLD signal corresponding to the state domain is input into the corresponding prediction submodule, the fine-grained state feature information of the cycle is extracted according to the steps described in step A. Finally, the feature information matrix is ​​passed through a fully connected layer to obtain the probability vector of the fine-grained state, and the vector is dimensionally expanded to obtain the final fine-grained state probability vector. In the process of training the brain's fine-grained cognitive state module, the corresponding cross-entropy loss function value is calculated for the final fine-grained state probability vector and used as the training threshold to realize the training of the brain's fine-grained cognitive state module.

Images