FMRI brain decoding method and device based on self-supervised space-time diagram model

Abstract

The invention provides an fMRI brain decoding method and device based on a self-supervised space-time diagram model, and relates to the field of artificial intelligence, and the method comprises the steps: obtaining an fMRI time sequence, dividing the fMRI time sequence into a plurality of sub-segments, carrying out the brain space feature extraction of the plurality of sub-segments through space diagram convolution, and obtaining a first brain space representation; generating a first brain time characterization; generating a brain space-time comprehensive representation; obtaining a second brain space representation; based on the second brain space representation and the second brain time representation, generating an enhanced brain space-time comprehensive representation as an initial brain decoding model; determining the sum of the first CCM and the second CCM as a target function of an initial brain decoding model, training the initial brain decoding model according to the target function, and generating a brain decoding model; and performing brain decoding based on the brain decoding model. According to the method, the brain decoding model is constructed based on the space-time brain features, and the brain decoding performance is improved through brain space-time representation extraction and loss constraint.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an fMRI brain decoding method and device based on a self-supervised spatiotemporal graph model. Background Art

[0002] Brain decoding refers to recording the brain's activity responses and modeling the brain responses through machine learning or deep learning methods to predict the type of external stimuli. Brain decoding has become one of the key research topics in the fields of human-computer interaction, signal processing, and machine learning. Brain function is completed by the interaction of many neuronal circuits operating in the spatiotemporal dimensions. Brain decoding from the perspective of spatiotemporal covariation can better reflect the way the brain itself operates and can further improve the brain's process of processing and characterizing information.

[0003] With the rapid development of artificial intelligence, especially deep learning technology, a variety of deep learning technologies have been successfully applied to spatiotemporal brain decoding. A series of spatiotemporal hybrid models can automatically extract brain spatiotemporal representations in an end-to-end manner, laying the foundation for further improving decoding performance and exploring new brain markers. Among them, the spatiotemporal hybrid model based on graph neural network (GNN) can model the complex interactive relationship between brain regions and has strong interpretability, which has attracted widespread attention recently. However, the superior performance of these deep models often depends on large-scale functional magnetic resonance imaging (fMRI) annotated data, which limits their innovation and flexibility. Graph self-supervised learning (GSSL) aims to use auxiliary tasks to mine its own supervision information from unlabeled data and learn general representations for downstream tasks, providing a potential new approach for neural decoding. Some studies have applied the GSSL framework to fMRI analysis, which can be divided into two categories: 1) Generation-based methods, taking autoencoders as an example, the brain representation is refined by the encoder and then reconstructed by the decoder. However, this single reconstruction constraint based on the generative method is difficult to fully learn the complex dependencies of low signal-to-noise ratio brain signals; 2) Contrast-based methods, with the enhanced contrast framework as a typical example, focus on generating appropriate brain views through data enhancement. Contrast-based methods are easy to implement and understand, and can be conveniently used for neural decoding. However, whether it is a generative or contrast-based self-supervised fMRI analysis framework, most of them only focus on the connection analysis between brain regions, while ignoring the temporal dependency of brain signals. Previous studies have shown that the interactions between brain regions are dynamic, and modeling the brain's temporal dependency can further improve the performance of self-supervised decoding.

[0004] Current brain decoding methods based on graph self-supervised learning are mainly divided into two categories.

[0005] Based on the generation method, an auto-encoder (AE) is usually used to reconstruct brain signals or brain topology. AE usually consists of two parts: an encoder and a decoder. The encoder maps the input brain signal into a compact representation, and then the decoder restores the learned representation to the original signal. The goal of the generation-based GSSL is to minimize the error between the original signal and the reconstructed signal. Li et al. were the first to try to use the deep sparse recurrent auto-encoder (DSRAE) to learn the spatiotemporal covariation pattern of the brain. Specifically, DSRAE trains the entire network by reconstructing the activation values ​​of brain voxels, but DSRAE ignores the interaction between different regions of interest (ROI), which is crucial for brain decoding. With the emergence of graph neural networks (GNNs), more and more studies have introduced GNNs into brain network modeling. In a recent study, Wen et al. developed a brain ensemble masked graph self-supervised learning framework (BrainGSL) for brain disease diagnosis. Specifically, the framework consists of a local topology-aware encoder, a node-edge dual decoder, a signal representation learning module, and a classifier. BrainGSL masks the edges and feature embeddings of some brain regions with the goal of reconstructing the brain adjacency matrix (functional connectivity between brain regions). Autoencoder-based fMRI signal reconstruction can effectively handle noise and uncertainty, however, modeling the long-term dependencies of fMRI data using a single reconstruction constraint is challenging.

[0006] Contrast-based GSSL has strong learning ability and has been widely used in natural language processing and computer vision. The core of contrast-based methods is to select appropriate positive and negative samples to learn the potential representation of fMRI data. Existing studies usually combine brain prior knowledge or sampling to guide decoding model optimization. Wang et al. incorporated prior knowledge (individual brain differences) into sample construction and proposed a contrast-based functional connectivity graph learning (CFCGL) framework. The study first divided the entire fMRI time series (each subject) into multiple subsequences. The subsequences within the same subject are positive samples of each other, and the subsequences of different subjects are negative samples of each other. Its advantage is that positive and negative samples can be selected more intuitively and quickly, but it is always very difficult to obtain reliable prior knowledge, and incorrect prior knowledge will lead to biased brain embedding.

[0007] All of the above studies focus on modeling the spatial interactions between brain regions. Studies have shown that the adjacency of brain regions changes over time and has a strong time dependency. Only focusing on modeling the spatial interactions between brain regions may result in missing features of learned brain representations, thus affecting decoding performance. Summary of the invention

[0008] In order to solve the technical problem of poor brain decoding performance in the prior art, an embodiment of the present invention provides an fMRI brain decoding method and device based on a self-supervised spatiotemporal graph model. The technical solution is as follows:

[0009] In one aspect, a method for fMRI brain decoding based on a self-supervised spatiotemporal graph model is provided, the method comprising:

[0010] Acquire an fMRI time series, divide the fMRI time series into a plurality of sub-segments by means of a sliding window, extract brain spatial features from the plurality of sub-segments by means of spatial graph convolution, and acquire a first brain spatial representation;

[0011] Determine a normalized autocorrelation of a time dimension for the fMRI time series, determine a feature of each time stamp in the fMRI time series through a time attention map, determine a time attention weight based on the feature of the time stamp, and generate a first brain time representation through a residual module based on the time attention weight and the normalized autocorrelation;

[0012] Generate a brain spatiotemporal integrated representation based on the first brain spatial representation and the first brain temporal representation by a time-aware brain representation extractor;

[0013] performing SCM on the fMRI time series, and obtaining a second brain spatial representation based on the fMRI time series after SCM;

[0014] performing TAM on the fMRI time series, and generating a second brain time representation based on the fMRI time series after TAM;

[0015] Generate an enhanced brain spatiotemporal comprehensive representation as an initial brain decoding model based on the second brain spatial representation and the second brain temporal representation by a time-aware brain representation extractor;

[0016] determining a first CCM between the integrated brain spatiotemporal representation and the second brain spatial representation;

[0017] determining a second CCM between the integrated brain spatiotemporal representation and the second brain temporal representation;

[0018] determining a sum of the first CCM and the second CCM as an objective function of the initial brain decoding model, and training the initial brain decoding model according to the objective function to generate a brain decoding model;

[0019] Brain decoding is performed based on the brain decoding model.

[0020] Optionally, the determining a first CCM between the brain spatiotemporal integrated representation and the second brain spatial representation is performed by the following formula:

[0021] Wherein, C represents the cross-correlation matrix between the brain spatiotemporal comprehensive representation and the second brain spatial representation, and s, i represent the row or column number of C. α is used to balance ∑ S (1-C ss ′) 2 and The constant of the difference between .

[0022] Optionally, the determining a second CCM between the brain spatiotemporal integrated representation and the second brain temporal representation is performed by the following formula:

[0023] Wherein, C represents the cross-correlation matrix between the brain spatiotemporal comprehensive representation and the second brain temporal representation, and t, j represent the row or column number of C respectively. α is used to balance ∑ t (1-C tt ′) 2 and The constant of the difference between .

[0024] Optionally, performing SCM on the fMRI time series is performed in the following manner:

[0025] By formula Determine the spatial masking probability of each ROI corresponding to the fMRI time series, where SMP i represents the spatial masking probability of the i-th ROI, d i represents the degree of the i-th ROI;

[0026] Masking is performed on the ROI whose spatial masking probability is greater than a first set value.

[0027] Optionally, performing TAM on the fMRI time series comprises:

[0028] By formula Determine the importance of each timestamp corresponding to the fMRI time series, where TI j Indicates the importance of the jth timestamp, c j Indicates the displacement between timestamps;

[0029] The timestamps whose importance is greater than a second set value are masked.

[0030] Optionally, generating a first brain time representation based on the time attention weight and the normalized autocorrelation through a residual module includes:

[0031] Based on the temporal attention weight, CAM = soft max(Conv1([NACT,TAM]))∈R N×L×L Generate a CAM, wherein TAM is the result of normalizing the temporal attention weight, NACT is the normalized autocorrelation, N represents the number of brain regions, and L represents the length of the subsegment;

[0032] Through the residual module, using the formula H t =Conv2(V(X)·CAM)+H s ∈R N×L×c Generate the first time representation, R is a real number;

[0033] Divide the sub-segment into a plurality of first sub-segments of length 3 by a non-overlapping sliding window Wherein, L is the length of the sub-segment;

[0034] Learn the importance scores of each timestamp and its left and right neighboring timestamps through the self-attention mechanism Where SA represents the self-attention mechanism, and R is a real number;

[0035] Weight the aggregate information based on the importance score to obtain the sub-segment representation Where sum represents the aggregation function of the time dimension;

[0036] Based on H mContinuing to assign weights to each sub-segment, the refined representation of the entire fMRI time series is hierarchically aggregated to strengthen the first temporal representation from local to global.

[0037] Optionally, performing brain spatial feature extraction on the multiple sub-segments through spatial graph convolution to obtain a first brain spatial representation includes:

[0038] The partial correlation coefficients between the time series were averaged to obtain the static brain network at the group level;

[0039] Based on the static brain network and the time series, through GNN, through the formula Extracting static spatial representations of the brain c , θ is a nonlinear activation function, W is a learnable parameter representing weight, b is a learnable parameter representing bias, is the adjacency relationship of the static brain network, D is the degree matrix, the diagonal elements are the degrees of each brain region, I is the unit matrix, is the time series;

[0040] By formula Generating dynamic spatial representations of the brain d , where I is the identity matrix, E N is the feature embedding for each brain region, W N is a learnable parameter, b N is a learnable parameter;

[0041] According to the formula H S =β1H c +β2H d The first brain spatial representation is generated according to the static spatial representation and the dynamic spatial representation of the brain, wherein H c is the static space representation, H d is the dynamic space representation, Among them, α m Used for constraints β m Iterative updates during training.

[0042] The present invention also provides an fMRI brain decoding device based on a self-supervised spatiotemporal graph model, the device is used to implement the fMRI brain decoding method based on a self-supervised spatiotemporal graph model provided by the present invention, the device comprising:

[0043] A first acquisition module is used to acquire an fMRI time series, divide the fMRI time series into multiple sub-segments through a sliding window, extract brain spatial features from the multiple sub-segments through spatial graph convolution, and acquire a first brain spatial representation;

[0044] A first generation module is used to determine a normalized autocorrelation of a time dimension for the fMRI time series, determine a feature of each time stamp in the fMRI time series through a time attention map, determine a time attention weight based on the feature of the time stamp, and generate a first brain time representation through a residual module based on the time attention weight and the normalized autocorrelation;

[0045] A second generating module, configured to generate a brain spatiotemporal integrated representation based on the first brain spatial representation and the first brain temporal representation by means of a time-aware brain representation extractor;

[0046] A second acquisition module is used to perform SCM on the fMRI time series and acquire a second brain spatial representation based on the fMRI time series after SCM;

[0047] A third generating module is used to perform TAM on the fMRI time series and generate a second brain time representation based on the fMRI time series after TAM;

[0048] a fourth generating module, configured to generate an enhanced brain spatiotemporal integrated representation as an initial brain decoding model based on the second brain spatial representation and the second brain temporal representation through a time-aware brain representation extractor;

[0049] A first determination module, configured to determine a first CCM between the brain spatiotemporal integrated representation and the second brain spatial representation;

[0050] a second determination module, configured to determine a second CCM between the brain spatiotemporal integrated representation and the second brain temporal representation;

[0051] a third determining module, configured to determine a sum of the first CCM and the second CCM as an objective function of the initial brain decoding model, and train the initial brain decoding model according to the objective function to generate a brain decoding model;

[0052] A decoding module is used to perform brain decoding based on the brain decoding model.

[0053] The present invention also provides an fMRI brain decoding device based on a self-supervised spatiotemporal graph model, the brain decoding device comprising:

[0054] processor;

[0055] A memory having computer-readable instructions stored therein, wherein the method provided by the present invention is implemented when the computer-readable instructions are executed by the processor.

[0056] The present invention also provides a computer-readable storage medium, in which program codes are stored. The program codes can be called by a processor to execute the method provided by the present invention.

[0057] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0058] The present invention generates a brain decoding model by generating a brain spatiotemporal comprehensive representation and an enhanced brain spatiotemporal comprehensive representation, and based on the brain spatiotemporal comprehensive representation and the enhanced brain spatiotemporal comprehensive representation, trains the enhanced brain spatiotemporal comprehensive representation to generate a brain decoding model, thereby constructing a brain decoding model based on spatiotemporal brain features, and improving the performance of brain decoding through brain spatiotemporal representation extraction and loss constraints. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0060] Figure 1 It is a flow chart of an fMRI brain decoding method based on a self-supervised spatiotemporal graph model provided by an embodiment of the present invention;

[0061] Figure 2 is a flow chart of a method for enhancing spatial representation and temporal representation provided by an embodiment of the present invention;

[0062] Figure 3 It is a block diagram of an fMRI brain decoding device based on a self-supervised spatiotemporal graph model provided by an embodiment of the present invention;

[0063] Figure 4 It is a structural schematic diagram of an fMRI brain decoding device based on a self-supervised spatiotemporal graph model provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0064] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0065] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "example" is intended to present the concept in a specific way. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or it can be either of the two.

[0066] In the embodiments of the present invention, "image" and "picture" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same. "of", "corresponding, relevant" and "corresponding" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same.

[0067] In the embodiments of the present invention, sometimes a subscript such as W1 may be mistakenly written as a non-subscript such as W1. When the difference is not emphasized, the meanings to be expressed are consistent.

[0068] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0069] In order to solve the technical problem of poor brain decoding performance in the prior art, an embodiment of the present invention provides an fMRI brain decoding method and device based on a self-supervised spatiotemporal graph model. The technical solution is as follows:

[0070] On the one hand, a fMRI brain decoding method based on a self-supervised spatiotemporal graph model is provided, such as Figure 1 As shown, the method includes:

[0071] Step 1: obtain an fMRI time series, divide the fMRI time series into multiple sub-segments through a sliding window, extract brain spatial features from the multiple sub-segments through spatial graph convolution, and obtain a first brain spatial representation.

[0072] Input fMRI time series X∈R N×T , N and T represent the number of brain regions and the length of the entire fMRI time series, respectively, and R is a real number. First, the original time series X is divided into multiple sub-segments of length L through a sliding window, and each sub-segment is subjected to spatial graph convolution to extract brain spatial features.

[0073] In the spatial dimension, considering the unstructured nature of the brain network, GNN is used to capture the potential synergistic relationships between different brain regions.

[0074] The present invention adopts a combination of static and dynamic methods to analyze specific functional areas of the brain, thereby deeply and comprehensively exploring the functional differences and biomarkers of brain networks during the execution of brain functions, such as Figure 2 shown.

[0075] Optionally, performing brain spatial feature extraction on the multiple sub-segments through spatial graph convolution to obtain a first brain spatial representation includes:

[0076] The partial correlation coefficients between the time series were averaged to obtain the static brain network at the group level.

[0077] Based on the static brain network and the time series, through GNN, through the formula Extracting static spatial representations of the brain c , θ is a nonlinear activation function, W is a learnable parameter representing weight, b is a learnable parameter representing bias, is the adjacency relationship of the static brain network, D is the degree matrix, the diagonal elements are the degrees of each brain region, I is the unit matrix, is the time series.

[0078] By formula Generating dynamic spatial representations of the brain d , where I is the identity matrix, E N is the feature embedding for each brain region, W N is a learnable parameter, b N are learnable parameters.

[0079] According to the formula H S =β1H c +β2H d The first brain spatial representation is generated according to the static spatial representation and the dynamic spatial representation of the brain, wherein H c is the static space representation, H d is the dynamic space representation, Among them, α m Used for constraints β m Iterative updates during training.

[0080] For the static brain network, the brain network construction method at the group level was used to average the partial correlation coefficients between all the subjects' time series to obtain the brain network at the group level, which reflects the common significant areas of the brain network among the subjects. Based on the constructed static brain network and time series samples, GNN was used to extract the static spatial representation of the brain. c .

[0081] For the construction of dynamic brain networks, considering that the original GNN shares W and b between all brain regions, this can significantly reduce model parameters while learning common patterns between different brain regions. However, previous studies have shown that when different brain regions cooperate to complete brain tasks, the changes in brain regions not only have similarities, but also have their own unique patterns. Considering only the common features between brain regions may not accurately decode brain states. But learning the unique embedding of each brain region requires designing an independent W and b for each brain region, which increases the parameter dimension. The present invention decomposes the learnable parameter W into two sub-parameter sets W=EN ·W N , E N Represents the feature embedding of each brain region, W N is a learnable parameter. The same operation is performed for the bias b, b = E N b N , E N Represents the feature embedding of each brain region, b N is a learnable parameter. This can significantly reduce the number of parameters while learning specific patterns in brain regions, thereby preventing overfitting. The present invention further embeds it into the weights and biases of graph convolution to dynamically update brain adjacency relationships. replace The softmax function is used to normalize the adaptive adjacency matrix, and ReLU is a nonlinear activation function. This method can make the adjacency matrix adaptively updated as the brain region is embedded, thereby adaptively mining the dynamic representation of the brain changing over time. d .

[0082] Step 2: determine the normalized autocorrelation of the time dimension of the fMRI time series, determine the characteristics of each timestamp in the fMRI time series through the time attention map, determine the time attention weight based on the characteristics of the timestamp, and generate the first brain time representation through the residual module based on the time attention weight and the normalized autocorrelation.

[0083] Alternatively, if Figure 2 As shown, the first brain time representation is generated through a residual module based on the time attention weight and the normalized autocorrelation, including:

[0084] Based on the temporal attention weight, CAM = softmax(Conv1([NACT,TAM]))∈R N×L×L Generate a CAM, where TAM is the result of normalizing the temporal attention weight, NACT is the normalized autocorrelation, N represents the number of brain regions, and L represents the length of the sub-segment.

[0085] Through the residual module, using the formula H t =Conv2(V(X)·CAM)+H s ∈R N×L×c Generate the first time representation, R is a real number.

[0086] The sub-segment is divided into a plurality of first sub-segments of length 3 by a non-overlapping sliding window. Wherein, L is the length of the sub-segment.

[0087] Learn the importance scores of each timestamp and its left and right neighboring timestamps through the self-attention mechanism Where SA represents the self-attention mechanism and R is a real number.

[0088] Weight the aggregate information based on the importance score to obtain the sub-segment representation Where sum represents the aggregation function of the time dimension.

[0089] Based on H m Continuing to assign weights to each sub-segment, the refined representation of the entire fMRI time series is hierarchically aggregated to strengthen the first temporal representation from local to global.

[0090] For each fMRI time series, the normalized autocorrelation for the temporal dimension (NACT) is calculated as follows:

[0091]

[0092] Then, the importance of each timestamp is calculated through the temporal attention map, and the continuous dependency of brain state transition is captured by weighting the features of all timestamps. The query, key and value are calculated through convolution operation to obtain the temporal attention weight (TAW).

[0093] Q(X)=Conv Q (H s )∈R N××L×C ,

[0094] K(X)=Conv K (H s )∈R N×L×c ,

[0095] V(X)=Conv V (H s )∈R N×L×C ,

[0096] Conv is the convolutional layer, c is the number of channels, and F is the affinity function, which is used to calculate the pairwise correlation between brain regions and realize the similarity encoding of different timestamps. The Softmax function is used here to normalize it.

[0097] TAM=F(Q(X),K(X))=Softmax(Q(X)K(X) T )∈R N×L×L ,

[0098] Although such a temporal attention map can effectively capture potential long-term temporal dependencies, it may sometimes focus too much on global similarities and ignore short-term information that should be paid attention to. Therefore, NACT and temporal attention mapping are introduced to guide the appropriate attention scope. In particular, NACT and the attention matrix are concatenated, a learnable transformation is applied, and the transformation result is normalized by a Softmax function to obtain a Calibration Attention Map (CAM), which can reveal non-local contextual brain temporal relationships in a more appropriate scope.

[0099] CAM=softmax(Conv1(INACT,TAM]))∈R N×L×L ,

[0100] Among them, [:, :] represents the connection operation. Based on CAM, the residual module is used to generate the temporal representation. The residual module can further learn the hidden temporal dependencies without destroying the original temporal relationship and obtain richer information.

[0101] H t =Conv2(V(X)·CAM)+H s ∈R N×L×c ,

[0102] Further, the sub-segment is divided into a plurality of first sub-segments of length 3 by a non-overlapping sliding window. Wherein, L is the length of the sub-segment, and the importance scores of each timestamp and its left and right adjacent timestamps are learned through the self-attention mechanism, and the weighted aggregation information is obtained based on the importance scores to obtain the sub-segment representation.

[0103] Where SA represents the self-attention mechanism and sum represents the aggregation function of the time dimension. Based on the importance score, the weight of each sub-segment is continuously assigned, and the fine representation of the entire time series is hierarchically aggregated. Through this gradual information integration, the time representation H t It can be strengthened from local to global.

[0104] Step 3: Generate a comprehensive brain spatiotemporal representation based on the first brain spatial representation and the first brain temporal representation through a time-aware brain representation extractor.

[0105] Step 4: Perform SCM on the fMRI time series, and obtain a second brain spatial representation based on the fMRI time series after SCM.

[0106] Optionally, performing SCM on the fMRI time series is performed in the following manner:

[0107] By formula Determine the spatial masking probability of each ROI corresponding to the fMRI time series, where SMP i represents the spatial masking probability of the i-th ROI, d i Represents the degree of the i-th ROI.

[0108] Masking is performed on the ROI whose spatial masking probability is greater than a first set value.

[0109] In the spatial dimension, considering the irregularity of brain area connections, spatial centrality masking (SCM) is proposed to generate alternative spatial views. The spatial information flow inspires the spatially masked fMRI data to generate features similar to the entire fMRI data, so that the proposed spatiotemporal feature extractor can capture the relationship between masked and unmasked brain areas, which is crucial for the subsequent decoding task.

[0110] In graph theory, centrality is usually regarded as the importance of a node in the entire network information flow, and nodes with higher centrality are considered to be hub nodes. Therefore, spatial masking emphasizes important brain topological structures rather than random damage. According to the centrality of each brain region, its masking probability is adaptively allocated, and the hub brain region is set with a higher probability to be masked, which helps the spatiotemporal feature extractor capture the global and high-level semantic representation of the brain state. In the present invention, degree centrality is selected as the centrality function because it is simple and effective. The masking probability of each brain region is calculated as follows:

[0111]

[0112] Among them SMP i represents the spatial masking probability of the i-th region of interest (ROI), d i In particular, the present invention discards the adjacency relationship and features of the shielded nodes, and only the unshielded nodes provide the learned brain representation.

[0113] Step 5: Perform TAM on the fMRI time series, and generate a second brain time representation based on the fMRI time series after TAM.

[0114] Optionally, performing TAM on the fMRI time series comprises:

[0115] By formula Determine the importance of each timestamp corresponding to the fMRI time series, where TI j Indicates the importance of the jth timestamp, c j Indicates the displacement between timestamps.

[0116] The timestamps whose importance is greater than a second set value are masked.

[0117] Temporal View Generation - Temporal Attentive Masking (TAM)

[0118] In the temporal dimension, considering the temporal continuity of changes in brain area connectivity, temporal attention masking (TAM) is developed to generate alternative temporal views. TAM encourages temporal masked fMRI data to generate features similar to the entire fMRI data. Since the brain area connections of adjacent timestamps tend to be consistent, the semantic information is more concentrated, and there may be more temporal redundant information. That is, some key timestamps contain most of the semantic information. Therefore, similar to SCM, the goal of TAM is to select key timestamps in the entire fMRI time series and mask the key timestamps to generate a temporal stream. And by encouraging similarity constraints, the spatiotemporal extractor can effectively learn the relationship between key timestamps and redundant timestamps. Specifically, the temporal rate of change of the entire fMRI time series can be defined as the displacement c between timestamps. j , its importance can be measured using the overall rate of change at each timestamp:

[0119]

[0120] Among them, TI j Indicates the importance of the jth timestamp. Here, more important timestamps are shielded to force the model to obtain global time information.

[0121] Step 6: Generate an enhanced brain spatiotemporal comprehensive representation as an initial brain decoding model based on the second brain spatial representation and the second brain temporal representation through a time-aware brain representation extractor.

[0122] Step 7: Determine a first CCM between the brain spatiotemporal integrated representation and the second brain spatial representation.

[0123] Optionally, the determining a first CCM between the brain spatiotemporal integrated representation and the second brain spatial representation is performed by the following formula:

[0124] Wherein, C represents the cross-correlation matrix between the brain spatiotemporal comprehensive representation and the second brain spatial representation, and s, i represent the row or column number of C. α is used to balance ∑ S (1-C ss ′) 2 and The constant of the difference between .

[0125] Step 8: Determine a second CCM between the brain spatiotemporal integrated representation and the second brain temporal representation.

[0126] Optionally, the determining a second CCM between the brain spatiotemporal integrated representation and the second brain temporal representation is performed by the following formula:

[0127] Wherein, C represents the cross-correlation matrix between the brain spatiotemporal comprehensive representation and the second brain temporal representation, and t, j represent the row or column number of C respectively. α is used to balance ∑ t (1-C tt ′) 2 and The constant of the difference between .

[0128] Step 9: Determine the sum of the first CCM and the second CCM as the objective function of the initial brain decoding model, train the initial brain decoding model according to the objective function, and generate a brain decoding model.

[0129] The objective function of the entire pre-training model: L total =L S +L T .

[0130] Step 10: Perform brain decoding based on the brain decoding model.

[0131] Among them, brain decoding is performed based on the brain decoding model, including:

[0132] Fine-tuning: Mining the representation of the brain in different states and adding linear layers to fine-tune the entire brain decoding to enhance the decoding ability of the brain decoding model.

[0133] Decoding: Based on the fine-tuned brain decoding model, the brain representations in different states are classified and decoded.

[0134] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0135] The present invention generates a brain decoding model by generating a brain spatiotemporal comprehensive representation and an enhanced brain spatiotemporal comprehensive representation, and based on the brain spatiotemporal comprehensive representation and the enhanced brain spatiotemporal comprehensive representation, trains the enhanced brain spatiotemporal comprehensive representation to generate a brain decoding model, thereby constructing a brain decoding model based on spatiotemporal brain features, and improving the performance of brain decoding through brain spatiotemporal representation extraction and loss constraints.

[0136] like Figure 3 As shown, the present invention also provides an fMRI brain decoding device based on a self-supervised spatiotemporal graph model, the device is used to implement the fMRI brain decoding method based on a self-supervised spatiotemporal graph model provided by the present invention, and the device includes:

[0137] The first acquisition module 301 is used to acquire an fMRI time series, divide the fMRI time series into multiple sub-segments through a sliding window, extract brain spatial features from the multiple sub-segments through spatial graph convolution, and acquire a first brain spatial representation.

[0138] The first generation module 302 is used to determine the normalized autocorrelation of the time dimension of the fMRI time series, determine the characteristics of each timestamp in the fMRI time series through the time attention map, determine the time attention weight based on the characteristics of the timestamp, and generate a first brain time representation through a residual module based on the time attention weight and the normalized autocorrelation.

[0139] The second generating module 303 is used to generate a brain spatiotemporal integrated representation based on the first brain spatial representation and the first brain temporal representation through a time-aware brain representation extractor.

[0140] The second acquisition module 304 is used to perform SCM on the fMRI time series, and acquire a second brain spatial representation based on the fMRI time series after SCM.

[0141] The third generating module 305 is used to perform TAM on the fMRI time series and generate a second brain time representation based on the fMRI time series after TAM.

[0142] The fourth generating module 306 is used to generate an enhanced brain spatiotemporal integrated representation as an initial brain decoding model based on the second brain spatial representation and the second brain temporal representation through a time-aware brain representation extractor.

[0143] The first determination module 307 is used to determine a first CCM between the brain spatiotemporal integrated representation and the second brain spatial representation.

[0144] The second determination module 308 is used to determine a second CCM between the brain spatiotemporal integrated representation and the second brain temporal representation.

[0145] The third determination module 309 is used to determine the sum of the first CCM and the second CCM as the objective function of the initial brain decoding model, and train the initial brain decoding model according to the objective function to generate a brain decoding model.

[0146] The decoding module 310 is configured to perform brain decoding based on the brain decoding model.

[0147] Figure 4 is a schematic diagram of the structure of an fMRI brain decoding device based on a self-supervised spatiotemporal graph model provided by an embodiment of the present invention, such as Figure 4 As shown, the fMRI brain decoding device based on the self-supervised spatiotemporal graph model may include the above Figure 3The fMRI brain decoding device based on the self-supervised spatiotemporal graph model is shown. Optionally, the fMRI brain decoding device based on the self-supervised spatiotemporal graph model 410 may include a first processor 2001 .

[0148] Optionally, the fMRI brain decoding device 410 based on the self-supervised spatiotemporal graph model may further include a memory 2002 and a transceiver 2003 .

[0149] The first processor 2001, the memory 2002 and the transceiver 2003 may be connected via a communication bus.

[0150] Combine the following Figure 4 The components of the fMRI brain decoding device 410 based on the self-supervised spatiotemporal graph model are specifically introduced:

[0151] The first processor 2001 is the control center of the fMRI brain decoding device 410 based on the self-supervised spatiotemporal graph model, and can be a processor or a general term for multiple processing elements. For example, the first processor 2001 is one or more central processing units (CPUs), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention, such as one or more microprocessors (digital signal processors, DSPs), or one or more field programmable gate arrays (field programmable gate arrays, FPGAs).

[0152] Optionally, the first processor 2001 can perform various functions of the fMRI brain decoding device 410 based on the self-supervised spatiotemporal graph model by running or executing a software program stored in the memory 2002 and calling data stored in the memory 2002.

[0153] In a specific implementation, as an embodiment, the first processor 2001 may include one or more CPUs, such as Figure 4 CPU0 and CPU1 are shown in FIG.

[0154] In a specific implementation, as an embodiment, the fMRI brain decoding device 410 based on the self-supervised spatiotemporal graph model may also include multiple processors, such as Figure 4The first processor 2001 and the second processor 2004 are shown in FIG. Each of these processors can be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here can refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0155] The memory 2002 is used to store the software program for executing the solution of the present invention, and is controlled to be executed by the first processor 2001. The specific implementation method can refer to the above method embodiment, which will not be repeated here.

[0156] Optionally, the memory 2002 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 2002 may be integrated with the first processor 2001, or may exist independently, and may be accessed through the interface circuit ( Figure 4 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0157] The transceiver 2003 is used to communicate with a network device or a terminal device.

[0158] Optionally, the transceiver 2003 may include a receiver and a transmitter ( Figure 4 The receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.

[0159] Optionally, the transceiver 2003 may be integrated with the first processor 2001, or may exist independently, and may be connected to the first processor 2001 through the interface circuit ( Figure 4 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0160] It should be noted that Figure 4 The structure of the fMRI brain decoding device 410 based on the self-supervised spatiotemporal graph model shown in the figure does not constitute a limitation on the router. The actual knowledge structure recognition device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0161] In addition, the technical effects of the fMRI brain decoding device 410 based on the self-supervised spatiotemporal graph model can refer to the technical effects of the XXX method described in the above method embodiment, which will not be repeated here.

[0162] It should be understood that the first processor 2001 in the embodiment of the present invention may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0163] It should also be understood that the memory in the embodiments of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0164] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0165] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0166] In the present invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can be represented by: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0167] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0168] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0169] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0170] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0171] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0172] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0173] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0174] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. An fMRI brain decoding method based on a self-supervised spatiotemporal graph model, characterized in that: The method comprises: Acquire an fMRI time series, divide the fMRI time series into a plurality of sub-segments by means of a sliding window, extract brain spatial features from the plurality of sub-segments by means of spatial graph convolution, and acquire a first brain spatial representation; Determine a normalized autocorrelation of a time dimension for the fMRI time series, determine a feature of each time stamp in the fMRI time series through a time attention map, determine a time attention weight based on the feature of the time stamp, and generate a first brain time representation through a residual module based on the time attention weight and the normalized autocorrelation; Generate a brain spatiotemporal integrated representation based on the first brain spatial representation and the first brain temporal representation by a time-aware brain representation extractor; performing SCM on the fMRI time series, and obtaining a second brain spatial representation based on the fMRI time series after SCM; performing TAM on the fMRI time series, and generating a second brain time representation based on the fMRI time series after TAM; Generate an enhanced brain spatiotemporal comprehensive representation as an initial brain decoding model based on the second brain spatial representation and the second brain temporal representation by a time-aware brain representation extractor; determining a first CCM between the integrated brain spatiotemporal representation and the second brain spatial representation; determining a second CCM between the integrated brain spatiotemporal representation and the second brain temporal representation; determining a sum of the first CCM and the second CCM as an objective function of the initial brain decoding model, and training the initial brain decoding model according to the objective function to generate a brain decoding model; Brain decoding is performed based on the brain decoding model.

2. The method according to claim 1, characterized in that The determining of the first CCM between the brain spatiotemporal integrated representation and the second brain spatial representation is performed by the following formula: Wherein, C represents the cross-correlation matrix between the brain spatiotemporal comprehensive representation and the second brain spatial representation, and s, i represent the row or column number of C. α is used to balance ∑ S (1-C ss ′) 2 and The constant of the difference between .

3. The method according to claim 1, characterized in that The determining of the second CCM between the brain spatiotemporal integrated representation and the second brain temporal representation is performed by the following formula: Wherein, C represents the cross-correlation matrix between the brain spatiotemporal comprehensive representation and the second brain temporal representation, and t, j represent the row or column number of C respectively. α is used to balance ∑ t (1-C tt ′) 2 and The constant of the difference between .

4. The method according to claim 1, characterized in that: The SCM of the fMRI time series is performed in the following manner: By formula Determine the spatial masking probability of each ROI corresponding to the fMRI time series, where SMP i represents the spatial masking probability of the i-th ROI, d i represents the degree of the i-th ROI; Masking is performed on the ROI whose spatial masking probability is greater than a first set value.

5. The method according to claim 1, characterized in that The performing TAM on the fMRI time series comprises: By formula Determine the importance of each timestamp corresponding to the fMRI time series, where TI j Indicates the importance of the jth timestamp, c j Indicates the displacement between timestamps; The timestamps whose importance is greater than a second set value are masked.

6. The method according to claim 1, characterized in that The step of generating a first brain time representation based on the time attention weight and the normalized autocorrelation through a residual module includes: Based on the temporal attention weight, CAM = softmax(Conv1([NACT,TAM]))∈R N×L×L Generate a CAM, wherein TAM is the result of normalizing the temporal attention weight, NACT is the normalized autocorrelation, N represents the number of brain regions, and L represents the length of the subsegment; Through the residual module, using the formula H t =Conv2(V(X)·CAM)+H s ∈R N×L×c Generate the first time representation, R is a real number; Divide the sub-segment into a plurality of first sub-segments of length 3 by a non-overlapping sliding window Wherein, L is the length of the sub-segment; Learn the importance scores of each timestamp and its left and right neighboring timestamps through the self-attention mechanism Where SA represents the self-attention mechanism, and R is a real number; Weight the aggregate information based on the importance score to obtain the sub-segment representation Where sum represents the aggregation function of the time dimension; Based on H m Continuing to assign weights to each sub-segment, the refined representation of the entire fMRI time series is hierarchically aggregated to strengthen the first temporal representation from local to global.

7. The method according to claim 1, characterized in that The extracting brain spatial features from the multiple sub-segments through spatial graph convolution to obtain a first brain spatial representation includes: The partial correlation coefficients between the time series were averaged to obtain the static brain network at the group level; Based on the static brain network and the time series, through GNN, through the formula Extracting static spatial representations of the brain c , θ is a nonlinear activation function, W is a learnable parameter representing weight, b is a learnable parameter representing bias, is the adjacency relationship of the static brain network, D is the degree matrix, the diagonal elements are the degrees of each brain region, I is the unit matrix, is the time series; By formula Generating dynamic spatial representations of the brain d , where I is the identity matrix, E N is the feature embedding for each brain region, W N is a learnable parameter, b N is a learnable parameter; According to the formula H S =β1H c +β2H d The first brain spatial representation is generated according to the static spatial representation and the dynamic spatial representation of the brain, wherein H c is the static space representation, H d is the dynamic space representation, Among them, α m Used for constraints β m Iterative updates during training.

8. An fMRI brain decoding device based on a self-supervised spatiotemporal graph model, the device being used to implement the fMRI brain decoding method based on a self-supervised spatiotemporal graph model as claimed in any one of claims 1 to 7, characterized in that: The device comprises: A first acquisition module is used to acquire an fMRI time series, divide the fMRI time series into multiple sub-segments through a sliding window, extract brain spatial features from the multiple sub-segments through spatial graph convolution, and acquire a first brain spatial representation; A first generation module is used to determine a normalized autocorrelation of a time dimension for the fMRI time series, determine a feature of each time stamp in the fMRI time series through a time attention map, determine a time attention weight based on the feature of the time stamp, and generate a first brain time representation through a residual module based on the time attention weight and the normalized autocorrelation; A second generating module, configured to generate a brain spatiotemporal integrated representation based on the first brain spatial representation and the first brain temporal representation by means of a time-aware brain representation extractor; A second acquisition module is used to perform SCM on the fMRI time series and acquire a second brain spatial representation based on the fMRI time series after SCM; A third generating module is used to perform TAM on the fMRI time series and generate a second brain time representation based on the fMRI time series after TAM; a fourth generating module, configured to generate an enhanced brain spatiotemporal integrated representation as an initial brain decoding model based on the second brain spatial representation and the second brain temporal representation through a time-aware brain representation extractor; A first determination module, configured to determine a first CCM between the brain spatiotemporal integrated representation and the second brain spatial representation; a second determination module, configured to determine a second CCM between the brain spatiotemporal integrated representation and the second brain temporal representation; a third determining module, configured to determine a sum of the first CCM and the second CCM as an objective function of the initial brain decoding model, and train the initial brain decoding model according to the objective function to generate a brain decoding model; A decoding module is used to perform brain decoding based on the brain decoding model.

9. An fMRI brain decoding device based on a self-supervised spatiotemporal graph model, characterized in that: The device comprises: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to any one of claims 1 to 7.

Images