An fMRI brain decoding method and device based on a self-supervised spatio-temporal graph model

Abstract

The application provides an fMRI brain decoding method and device based on a self-supervised spatiotemporal graph model, and relates to the field of artificial intelligence.The method comprises the following steps: acquiring an fMRI time sequence, dividing the fMRI time sequence into multiple subsegments, extracting brain spatial features through spatial graph convolution on the multiple subsegments to obtain a first brain spatial representation, generating a first brain temporal representation, generating a brain spatiotemporal comprehensive representation, acquiring a second brain spatial representation, generating an enhanced brain spatiotemporal comprehensive representation based on the second brain spatial representation and the second brain temporal representation as an initial brain decoding model, determining the sum of the first CCM and the second CCM as a target function of the initial brain decoding model, training the initial brain decoding model according to the target function to generate a brain decoding model, and performing brain decoding based on the brain decoding model.The application constructs a brain decoding model based on spatiotemporal brain features, and improves the performance of brain decoding through brain spatiotemporal representation extraction and loss constraint.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to an fMRI brain decoding method and device based on a self-supervised spatio-temporal graph model. BACKGROUND

[0002] Brain decoding refers to recording the activity response of the brain, and modeling the brain response through machine learning or deep learning methods to realize the prediction of the type of external stimulation. Brain decoding has become one of the key research topics in the fields of human-computer interaction, signal processing, machine learning, etc. Brain function is completed by the interaction of numerous neuron circuits running in the time-space dimension. From the perspective of spatio-temporal covariance, brain decoding can better reflect the operation mode of the brain itself and further improve the process of brain information processing and characterization.

[0003] With the rapid development of artificial intelligence, especially deep learning technology, various deep learning technologies have been successfully applied to spatio-temporal brain decoding. A series of spatio-temporal hybrid models can automatically extract brain spatio-temporal representations in an end-to-end manner, laying a foundation for further improving decoding performance and exploring new brain markers. Among them, the spatio-temporal hybrid model based on graph neural network (GNN) can model the complex interaction between brain regions and has strong interpretability, and has recently received extensive attention. However, the superior performance of these deep models often depends on large-scale functional magnetic resonance imaging (fMRI) labeled data, which limits their innovation and flexibility. Graph self-supervised learning (GSSL) aims to use auxiliary tasks to mine its own supervision information from unlabeled data and learn a general representation for downstream tasks, providing a potential new way for neural decoding. Some researches apply the GSSL framework to fMRI analysis, which can be mainly divided into two categories: 1) generation-based methods, for example, using an encoder to extract brain representations, and then using a decoder to reconstruct them. However, the single reconstruction constraint of this generation-based method is difficult to comprehensively learn the complex dependence of low signal-to-noise ratio brain signals; 2) contrast-based methods, for example, using an enhanced contrast framework, which focuses on generating appropriate brain views through data augmentation. Contrast-based methods are easy to implement and understand, and can be easily used for neural decoding. However, whether it is a generation-based or a contrast-based self-supervised fMRI analysis framework, they mostly only focus on the connection analysis between brain regions, ignoring the time dependence of brain signals. Existing researches have shown that the interaction between brain regions is dynamic, and modeling the time dependence of the brain can further improve the performance of self-supervised decoding.

[0004] Current brain decoding methods based on graph self-supervised learning are mainly divided into two categories.

[0005] Generative methods typically use autoencoders (AEs) to reconstruct brain signals or brain topology. An AE usually consists of an encoder and a decoder. The encoder maps the input brain signal into a compact representation, and the decoder then restores the learned representation to the original signal. The goal of generative GSSLs is to minimize the error between the original and reconstructed signals. Li et al. were among the first to attempt to use deep sparse recurrent autoencoders (DSRAEs) to learn spatiotemporal covariance patterns in the brain. Specifically, DSRAE trains the entire network by reconstructing the activation values ​​of brain voxels, but it neglects the interactions between different regions of interest (ROIs), which are crucial for brain decoding. With the advent of graph neural networks (GNNs), more and more research is introducing GNNs into brain network modeling. In a recent study, Wen et al. developed a brainensemble masked graph self-supervised learning framework (BrainGSL) for the diagnosis of brain diseases. Specifically, the framework includes a local topology-aware encoder, a node-edge dual decoder, a signal representation learning module, and a classifier. BrainGSL masks the edges and feature embeddings of partial brain regions with the aim of reconstructing the brain adjacency matrix (functional connectivity between brain regions). Autoencoder-based fMRI signal reconstruction can effectively handle noise and uncertainty; however, modeling long-term dependencies of fMRI data using a single reconstruction constraint is challenging.

[0006] Contrastive Functional Connectivity Graph Learning (CFCGL) possesses strong learning capabilities and has been widely applied in natural language processing and computer vision. The core of contrastive methods is selecting appropriate positive and negative samples to learn the latent representations of fMRI data. Existing research typically incorporates prior knowledge of the brain or sampling to guide decoding model optimization. Wang et al. incorporated prior knowledge (individual brain differences) into sample construction, proposing a Contrastive Functional Connectivity Graph Learning (CFCGL) framework. This study first divides the entire fMRI time series (for each subject) into multiple subsequences. Subsequences within the same subject are considered positive samples to each other, while subsequences from different subjects are considered negative samples to each other. Its advantage is that it allows for more intuitive and rapid selection of positive and negative samples; however, obtaining reliable prior knowledge is always very difficult, and incorrect prior knowledge can lead to biased brain embeddings.

[0007] All of these studies focus on modeling spatial interactions between brain regions. Research indicates that the adjacency relationships between brain regions change over time and are highly time-dependent. Focusing solely on modeling spatial interactions between brain regions may result in the loss of features in the learned brain representations, thus affecting decoding performance. Summary of the Invention

[0008] To address the poor brain decoding performance of existing technologies, this invention provides an fMRI brain decoding method and apparatus based on a self-supervised spatiotemporal graph model. The technical solution is as follows:

[0009] On the one hand, a method for fMRI brain decoding based on a self-supervised spatiotemporal graph model is provided, the method comprising:

[0010] The fMRI time series is acquired, and the fMRI time series is divided into multiple segments by sliding window. The multiple segments are then subjected to spatial graph convolution to extract brain spatial features and obtain the first brain spatial representation.

[0011] The normalized autocorrelation of the time dimension of the fMRI time series is determined, the features of each time point in the fMRI time series are determined by the time attention map, the time attention weight is determined based on the features of the time point, and the first brain time representation is generated by the residual module based on the time attention weight and the normalized autocorrelation.

[0012] Using a time-aware brain representation extractor, a comprehensive spatiotemporal brain representation is generated based on the first brain spatial representation and the first brain temporal representation.

[0013] SCM is performed on the fMRI time series, and a second brain spatial representation is obtained based on the fMRI time series after SCM.

[0014] The fMRI time series was subjected to TAM, and a second brain time representation was generated based on the TAM-enhanced fMRI time series.

[0015] Using a time-aware brain representation extractor, an enhanced spatiotemporal comprehensive brain representation is generated based on the second brain spatial representation and the second brain temporal representation, serving as the initial brain decoding model.

[0016] Determine the first CCM between the aforementioned spatiotemporal integrated brain representation and the second brain spatial representation;

[0017] Determine the second CCM between the aforementioned spatiotemporal integrated brain representation and the second brain temporal representation;

[0018] The sum of the first CCM and the second CCM is determined as the objective function of the initial brain decoding model. The initial brain decoding model is trained according to the objective function to generate a brain decoding model.

[0019] Brain decoding is performed based on the aforementioned brain decoding model;

[0020] The SCM process on the fMRI time series is performed as follows:

[0021] Through formula Determine the spatial masking probability of each ROI corresponding to the fMRI time series, where SMP i Let d represent the spatial masking probability of the i-th ROI. i This represents the degree of the i-th ROI;

[0022] The ROIs with a spatial masking probability greater than a first set value are masked;

[0023] The step of performing TAM on the fMRI time series includes:

[0024] Through formula Determine the importance of each timestamp corresponding to the fMRI time series, where TI j c represents the importance of the j-th timestamp. j Indicates the offset between timestamps;

[0025] Timestamps whose importance is greater than the second set value are masked.

[0026] Optionally, the determination of the first CCM between the spatiotemporal integrated brain representation and the second brain spatial representation is performed by the following formula:

[0027] Wherein, C represents the cross-correlation matrix between the spatiotemporal integrated brain representation and the second brain spatial representation, s,i represents the row or column number of C, and α is used for balancing and The constant representing the difference between them.

[0028] Optionally, the determination of the second CCM between the brain spatiotemporal integrated representation and the second brain temporal representation is performed by the following formula:

[0029] Wherein, C represents the cross-correlation matrix between the aforementioned spatiotemporal integrated brain representation and the second brain temporal representation, t and j represent the row or column numbers of C, and α is used for balancing. and The constant representing the difference between them.

[0030] Optionally, the SCM of the fMRI time series is performed in the following manner:

[0031] Through formula Determine the spatial masking probability of each ROI corresponding to the fMRI time series, where SMP i Let d represent the spatial masking probability of the i-th ROI. i This represents the degree of the i-th ROI;

[0032] The ROIs with a spatial masking probability greater than a first set value are masked.

[0033] Optionally, performing TAM on the fMRI time series includes: using the formula Determine the importance of each timestamp corresponding to the fMRI time series, where TI j c represents the importance of the j-th timestamp. j Indicates the offset between timestamps;

[0034] Timestamps whose importance is greater than the second set value are masked.

[0035] Optionally, the step of generating a first brain temporal representation based on the temporal attention weights and the normalized autocorrelation through a residual module includes:

[0036] Based on the aforementioned time attention weights, the formula CAM = softmax(Conv1([NACT, TAM])) ∈ R is used. N×L×L Generate CAM, where TAM is the result after normalizing the temporal attention weights, NACT is the normalized autocorrelation, N represents the number of brain regions, and L represents the length of the sub-segment;

[0037] Using the residual module, formula H t =Conv2(V(X)·CAM)+H s ∈R N×L×c Generate a first-time representation, where R is a real number; divide the sub-segment into multiple first sub-segments of length 3 using a non-overlapping sliding window. Where L is the length of the sub-segment;

[0038] The importance score of each timestamp and its left and right adjacent timestamps is learned through a self-attention mechanism. Where SA represents the self-attention mechanism, and R is a real number;

[0039] Information is weighted and aggregated based on importance scores to obtain segment representations. Where sum represents the aggregation function in the time dimension;

[0040] Based on H m The weights of each sub-segment are further assigned, and the fine representation of the entire fMRI time series is aggregated hierarchically to enhance the first temporal representation from local to global.

[0041] Optionally, the step of extracting brain spatial features from the plurality of sub-segments through spatial graph convolution to obtain a first brain spatial representation includes:

[0042] The static brain network at the group level is obtained by averaging the partial correlation coefficients between the time series.

[0043] Based on the aforementioned static brain network and the aforementioned time series, using GNN, and through the formula...

[0044] Extracting the static spatial representation H of the brain c θ is a non-linear activation function, W is a learnable parameter representing the weights, and b is a learnable parameter representing the bias. Let D be the adjacency relation of the static brain network, where D is the degree matrix, the diagonal elements are the degrees of each brain region, and I is the identity matrix. It is the time series;

[0045] Through formula Generate dynamic spatial representations of the brain H d Where I is the identity matrix, E N Feature embedding for each brain region, W N b is a learnable parameter N These are learnable parameters;

[0046] According to formula H S =β1H c +β2H dBased on the static spatial representation and the dynamic spatial representation of the brain, a first brain spatial representation is generated, wherein H c For the static spatial representation, H d This is the representation of the dynamic space. Where, α m Used for constraints β m Iterative updates are performed during the training process.

[0047] This invention also provides an fMRI brain decoding device based on a self-supervised spatiotemporal graph model. The device is used to implement the fMRI brain decoding method based on a self-supervised spatiotemporal graph model provided by this invention. The device includes:

[0048] The first acquisition module is used to acquire fMRI time series, divide the fMRI time series into multiple segments through a sliding window, extract brain spatial features from the multiple segments through spatial graph convolution, and acquire the first brain spatial representation.

[0049] The first generation module is used to determine the normalized autocorrelation of the time dimension of the fMRI time series, determine the features of each time stamp in the fMRI time series through the time attention map, determine the time attention weight based on the features of the time stamp, and generate the first brain time representation through the residual module based on the time attention weight and the normalized autocorrelation.

[0050] The second generation module is used to generate a spatiotemporal comprehensive brain representation based on the first brain spatial representation and the first brain temporal representation using a time-aware brain representation extractor.

[0051] The second acquisition module is used to perform SCM on the fMRI time series and acquire a second brain spatial representation based on the fMRI time series after SCM.

[0052] The third generation module is used to perform TAM on the fMRI time series and generate a second brain time representation based on the TAM-enhanced fMRI time series.

[0053] The fourth generation module is used to generate an enhanced spatiotemporal comprehensive brain representation based on the second brain spatial representation and the second brain temporal representation through a time-aware brain representation extractor, as an initial brain decoding model.

[0054] The first determining module is used to determine the first CCM between the brain spatiotemporal integrated representation and the second brain spatial representation;

[0055] The second determining module is used to determine the second CCM between the brain spatiotemporal integrated representation and the second brain temporal representation;

[0056] The third determining module is used to determine that the sum of the first CCM and the second CCM is the objective function of the initial brain decoding model, and to train the initial brain decoding model according to the objective function to generate a brain decoding model.

[0057] A decoding module is used to perform brain decoding based on the brain decoding model;

[0058] The second acquisition module performs SCM on the fMRI time series in the following manner:

[0059] Through formula Determine the spatial masking probability of each ROI corresponding to the fMRI time series, where SMP i Let d represent the spatial masking probability of the i-th ROI. i This represents the degree of the i-th ROI;

[0060] The ROIs with a spatial masking probability greater than a first set value are masked;

[0061] The third generation module performs TAM on the fMRI time series, including:

[0062] Through formula Determine the importance of each timestamp corresponding to the fMRI time series, where TI j c represents the importance of the j-th timestamp. j Indicates the offset between timestamps;

[0063] Timestamps whose importance is greater than the second set value are masked.

[0064] The present invention also provides an fMRI brain decoding device based on a self-supervised spatiotemporal graph model, the brain decoding device comprising:

[0065] processor;

[0066] A memory storing computer-readable instructions, which, when executed by the processor, implement the method provided by the present invention.

[0067] The present invention also provides a computer-readable storage medium storing program code, which can be invoked by a processor to execute the method provided by the present invention.

[0068] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following:

[0069] This invention generates a comprehensive spatiotemporal representation of the brain, as well as an enhanced comprehensive spatiotemporal representation of the brain. Based on the comprehensive spatiotemporal representation of the brain and the enhanced comprehensive spatiotemporal representation of the brain, the enhanced comprehensive spatiotemporal representation of the brain is trained to generate a brain decoding model. Thus, a brain decoding model is constructed based on spatiotemporal brain features. Through brain spatiotemporal representation extraction and loss constraints, the performance of brain decoding is improved. Attached Figure Description

[0070] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0071] Figure 1 This is a flowchart of an fMRI brain decoding method based on a self-supervised spatiotemporal graph model provided in an embodiment of the present invention;

[0072] Figure 2 This is a flowchart of a method for enhancing spatial and temporal representations provided in an embodiment of the present invention;

[0073] Figure 3 This is a block diagram of an fMRI brain decoding device based on a self-supervised spatiotemporal graph model provided in an embodiment of the present invention;

[0074] Figure 4 This is a schematic diagram of the structure of an fMRI brain decoding device based on a self-supervised spatiotemporal graph model provided in an embodiment of the present invention. Detailed Implementation

[0075] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0076] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0077] In the embodiments of this invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, their intended meanings are consistent. Similarly, the terms "of," "corresponding (relevant)," and "corresponding" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, their intended meanings are consistent.

[0078] In this embodiment of the invention, sometimes a subscript such as W1 may be mistakenly written as a non-subscript form such as W1. When the difference is not emphasized, the meaning they express is the same.

[0079] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0080] To address the poor brain decoding performance of existing technologies, this invention provides an fMRI brain decoding method and apparatus based on a self-supervised spatiotemporal graph model. The technical solution is as follows:

[0081] On the one hand, a method for fMRI brain decoding based on a self-supervised spatiotemporal graph model is provided, such as... Figure 1 As shown, the method includes:

[0082] Step 1: Obtain the fMRI time series, divide the fMRI time series into multiple segments using a sliding window, and extract brain spatial features from the multiple segments using spatial graph convolution to obtain the first brain spatial representation.

[0083] Input fMRI time series X∈R N×T N and T represent the number of brain regions and the length of the entire fMRI time series, respectively, and R is a real number. First, the original time series X is divided into multiple segments of length L using a sliding window, and each segment is used to extract brain spatial features through spatial graph convolution.

[0084] In the spatial dimension, considering the unstructured nature of brain networks, GNNs are used to capture potential synergistic relationships between different brain regions.

[0085] This invention employs a combination of static and dynamic methods to analyze specific functional areas of the brain, thereby deeply and comprehensively exploring the functional differences and biomarkers of brain networks during brain function execution, such as... Figure 2 As shown.

[0086] Optionally, the step of extracting brain spatial features from the plurality of sub-segments through spatial graph convolution to obtain a first brain spatial representation includes:

[0087] The static brain network at the group level is obtained by averaging the partial correlation coefficients between the time series.

[0088] Based on the aforementioned static brain network and the aforementioned time series, using GNN, and through the formula... Extracting the static spatial representation H of the brain c θ is a non-linear activation function, W is a learnable parameter representing the weights, and b is a learnable parameter representing the bias. Let D be the adjacency relation of the static brain network, where D is the degree matrix, the diagonal elements are the degrees of each brain region, and I is the identity matrix. It is the time series.

[0089] Through formula Generate dynamic spatial representations of the brain H d Where I is the identity matrix, E N For each brain region, feature embeddings are defined, where WN is a learnable parameter and bN is a learnable parameter.

[0090] According to formula H S =β1H c +β2H d Based on the static spatial representation and the dynamic spatial representation of the brain, a first brain spatial representation is generated, wherein H c For the static spatial representation, H d This is the representation of the dynamic space. Where, α m Used for constraints β m Iterative updates are performed during the training process.

[0091] For static brain networks, a group-level brain network construction method was adopted. The partial correlation coefficients among all participants' time series were averaged to obtain the group-level brain network, reflecting the common salient regions of the brain networks among participants. Based on the constructed static brain network and time series samples, a Genetic Neural Network (GNN) was used to extract the static spatial representation H of the brain. c .

[0092] For constructing dynamic brain networks, considering that the original GNN shares W and b across all brain regions, this can significantly reduce model parameters while learning common patterns among different brain regions. However, previous studies have shown that when different brain regions cooperate to complete brain tasks, the changes in brain regions not only have similarities but also their own unique patterns. Considering only the common features between brain regions may not accurately decode brain states. But learning the unique embedding of each brain region requires designing an independent W and b for each brain region, which increases the parameter dimensionality. This invention decomposes the learnable parameter W into two sub-parameter sets W = E. N ·W N E N Feature embeddings representing each brain region, W N These are learnable parameters. The same operation was performed for the bias b, b = E. N ·b N E N Feature embeddings representing each brain region, b NThese are learnable parameters. This allows for a significant reduction in the number of parameters while learning specific patterns in brain regions, thus preventing overfitting. Furthermore, this invention embeds them into the weights and biases of graph convolution to dynamically update brain adjacency relationships, utilizing… replace The softmax function is used to normalize the adaptive adjacency matrix, and ReLU is a non-linear activation function. This approach allows the adjacency matrix to be adaptively updated with brain region embeddings, thereby adaptively mining the dynamic representations of the brain over time. d .

[0093] Step 2: Determine the normalized autocorrelation of the time dimension of the fMRI time series, determine the features of each timestamp in the fMRI time series through the time attention map, determine the time attention weight based on the features of the timestamp, and generate the first brain time representation based on the time attention weight and the normalized autocorrelation through the residual module.

[0094] Optionally, such as Figure 2 As shown, the generation of the first brain temporal representation based on the temporal attention weights and the normalized autocorrelation through the residual module includes: based on the temporal attention weights, generating the first brain temporal representation through the formula CAM=soft max(Conv1([NACT,TAM]))∈R N×L×L Generate CAM, where TAM is the result after normalizing the temporal attention weights, NACT is the normalized autocorrelation, N represents the number of brain regions, and L represents the length of the sub-segment.

[0095] Using the residual module, formula H t =Conv2(V(X)·CAM)+H s ∈R N×L×c Generate the first-time representation, where R is a real number.

[0096] The sub-segment is divided into multiple first sub-segments of length 3 using a non-overlapping sliding window. Where L is the length of the sub-segment.

[0097] The importance score of each timestamp and its left and right adjacent timestamps is learned through a self-attention mechanism. Where SA represents self-attention mechanism, and R is a real number.

[0098] Information is weighted and aggregated based on importance scores to obtain segment representations. Where sum represents the aggregation function for the time dimension.

[0099] Based on H mThe weights of each sub-segment are further assigned, and the fine representation of the entire fMRI time series is aggregated hierarchically to enhance the first temporal representation from local to global.

[0100] For each fMRI time series, the normalized autocorrelation for the temporal dimension (NACT) is calculated as follows: Then, the importance of each timestamp is calculated using a temporal attention map, and the continuous dependencies of brain state transitions are captured by weighting the features of all timestamps. The Temporal Attention Weight (TAW) is obtained by calculating the Query, Key, and Value through convolution operations.

[0101] Q(X) = Conv Q (H s )∈R N×L×c ,

[0102] K(X) = Conv K (H s )∈R N×L×c ,

[0103] V(X) = Conv V (H s )∈R N×L×c ,

[0104] Where Conv is a convolutional layer, c is the number of channels, and F is the affinity function, used to calculate the pairwise correlation between brain regions to encode the similarity of different timestamps. Here, the Softmax function is used to normalize it.

[0105] TAM=F(Q(X),K(X))=Softmax(Q(X)K(X) T )∈R N×L×L While such temporal attention maps can effectively capture potential long-term temporal dependencies, they can sometimes focus too much on global similarity and neglect short-term information that should be noted. Therefore, a NACT-T temporal attention map is introduced to guide the appropriate attention scope. Specifically, by concatenating the NACT and the attention matrix, applying a learnable transformation, and normalizing the transformation result using a softmax function, a Calibration Attention Map (CAM) is obtained, which can reveal non-local contextual brain temporal relationships within a more appropriate scope.

[0106] CAM=softmax(Conv1([NACT,TAM]))∈R N×L×LIn this context, [:,:] represents a join operation. Based on CAM, a residual module is used to generate the time representation. The residual module can further learn the hidden time dependencies without destroying the original time relationships, thus obtaining richer information.

[0107] H t =Conv2(V(X)·CAM)+H s ∈R N×L×c Furthermore, the sub-segment is divided into multiple first sub-segments of length 3 using a non-overlapping sliding window. Where L is the length of the sub-segment, the importance score of each timestamp and its left and right adjacent timestamps is learned through a self-attention mechanism, and the information is weighted and aggregated based on the importance score to obtain the sub-segment representation.

[0108] Here, SA represents self-attention, and sum represents the aggregation function for the time dimension. Weights are further assigned to each segment based on importance scores, hierarchically aggregating a refined representation of the entire time series. Through this gradual information integration, the time representation H... t It can be enhanced from the local to the global level.

[0109] Step 3: Using a time-aware brain representation extractor, generate a comprehensive spatiotemporal brain representation based on the first brain spatial representation and the first brain temporal representation.

[0110] Step 4: Perform SCM on the fMRI time series and obtain the second brain spatial representation based on the fMRI time series after SCM.

[0111] Optionally, the SCM of the fMRI time series is performed in the following manner: using the formula Determine the spatial masking probability of each ROI corresponding to the fMRI time series, where SMP i Let d represent the spatial masking probability of the i-th ROI. i This represents the degree of the i-th ROI.

[0112] The ROIs with a spatial masking probability greater than a first set value are masked.

[0113] In the spatial dimension, considering the irregularities in brain region connectivity, a spatial centrality masking (SCM) is proposed to generate alternative spatial views. The spatial information flow stimulates the generation of features in the spatially masked fMRI data that are similar to those in the entire fMRI data. This enables the proposed spatiotemporal feature extractor to capture the relationships between masked and unmasked brain regions, which is crucial for subsequent decoding tasks.

[0114] In graph theory, centrality is often considered as the importance of a node in the overall information flow of a network, with nodes having higher centrality being considered hub nodes. Therefore, spatial masking emphasizes important brain topology rather than random corruption. Based on the centrality of each brain region, its masking probability is adaptively assigned, with hub regions assigned a higher probability of being masked. This helps the spatiotemporal feature extractor capture global and high-level semantic representations of brain states. In this invention, degree centrality is chosen as the centrality function because it is simple and effective. The masking probability of each brain region is calculated as follows: SMP i d represents the spatial masking probability of the i-th region of interest (ROI). i This represents the degree of the i-th ROI. In particular, this invention discards the adjacency relationships and features of the masked nodes, and only provides the learned brain representations from the unmasked nodes.

[0115] Step 5: Perform TAM on the fMRI time series, and generate a second brain time representation based on the TAM-enhanced fMRI time series.

[0116] Optionally, performing TAM on the fMRI time series includes: using the formula Determine the importance of each timestamp corresponding to the fMRI time series, where TI j c represents the importance of the j-th timestamp. j Indicates the offset between timestamps.

[0117] Timestamps whose importance is greater than the second set value are masked.

[0118] Temporal Attention Masking (TAM) considers the temporal continuity of brain region connectivity changes in the temporal dimension and has developed a method to generate alternative temporal views. TAM encourages the generation of features similar to the entire fMRI data through temporal masking. Since brain region connectivity tends to be consistent between adjacent time stamps, semantic information is more concentrated, potentially leading to more temporal redundancy. That is, some key time stamps contain most of the semantic information. Therefore, similar to SCM, TAM aims to select key time stamps throughout the fMRI time series and mask them to generate a temporal flow. By encouraging similarity constraints, it enables the spatiotemporal extractor to effectively learn the relationship between key and redundant time stamps. Specifically, the temporal rate of change of the entire fMRI time series can be defined as the displacement c between time stamps. j Its importance can be measured using the overall rate of change for each timestamp: Among them TIj This indicates the importance of the j-th timestamp. Here, more important timestamps are masked, forcing the model to obtain global time information.

[0119] Step 6: Using the time-aware brain representation extractor, based on the second brain spatial representation and the second brain temporal representation, generate an enhanced brain spatiotemporal integrated representation as the initial brain decoding model.

[0120] Step 7: Determine the first CCM between the brain spatiotemporal integrated representation and the second brain spatial representation.

[0121] Optionally, the determination of the first CCM between the spatiotemporal integrated brain representation and the second brain spatial representation is performed by the following formula: Where C represents the cross-correlation matrix between the aforementioned spatiotemporal integrated brain representation and the second brain spatial representation, and s, i represent the row or column number of C. α is used for balancing. and The constant representing the difference between them.

[0122] Step 8: Determine the second CCM between the brain spatiotemporal integrated representation and the second brain time representation.

[0123] Optionally, the determination of the second CCM between the brain spatiotemporal integrated representation and the second brain temporal representation is performed by the following formula: Where C represents the cross-correlation matrix between the aforementioned spatiotemporal integrated brain representation and the second brain temporal representation, and t and j represent the row or column numbers of C, respectively. α is used for balancing. and The constant representing the difference between them.

[0124] Step 9: Determine the sum of the first CCM and the second CCM as the objective function of the initial brain decoding model, and train the initial brain decoding model according to the objective function to generate a brain decoding model.

[0125] The objective function of the entire pre-trained model: L total =L S +L T .

[0126] Step 10: Perform brain decoding based on the brain decoding model.

[0127] Among them, brain decoding based on brain decoding models includes:

[0128] Fine-tuning: Explore the brain's representations under different states and add linear layers to fine-tune the entire brain decoding process to enhance the decoding capabilities of the brain decoding model.

[0129] Decoding: Based on a finely tuned brain decoding model, the brain's representations under different states are classified and decoded.

[0130] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following:

[0131] This invention generates a comprehensive spatiotemporal representation of the brain, as well as an enhanced comprehensive spatiotemporal representation of the brain. Based on the comprehensive spatiotemporal representation of the brain and the enhanced comprehensive spatiotemporal representation of the brain, the enhanced comprehensive spatiotemporal representation of the brain is trained to generate a brain decoding model. Thus, a brain decoding model is constructed based on spatiotemporal brain features. Through brain spatiotemporal representation extraction and loss constraints, the performance of brain decoding is improved.

[0132] like Figure 3 As shown, the present invention also provides an fMRI brain decoding device based on a self-supervised spatiotemporal graph model. The device is used to implement the fMRI brain decoding method based on a self-supervised spatiotemporal graph model provided by the present invention. The device includes:

[0133] The first acquisition module 301 is used to acquire fMRI time series, divide the fMRI time series into multiple segments through a sliding window, and extract brain spatial features from the multiple segments through spatial graph convolution to obtain a first brain spatial representation.

[0134] The first generation module 302 is used to determine the normalized autocorrelation of the time dimension of the fMRI time series, determine the features of each timestamp in the fMRI time series through a time attention map, determine the time attention weight based on the features of the timestamp, and generate a first brain time representation based on the time attention weight and the normalized autocorrelation through a residual module.

[0135] The second generation module 303 is used to generate a comprehensive spatiotemporal representation of the brain based on the first brain spatial representation and the first brain temporal representation through a time-aware brain representation extractor.

[0136] The second acquisition module 304 is used to perform SCM on the fMRI time series and acquire a second brain spatial representation based on the fMRI time series after SCM.

[0137] The third generation module 305 is used to perform TAM on the fMRI time series and generate a second brain time representation based on the fMRI time series after TAM.

[0138] The fourth generation module 306 is used to generate an enhanced spatiotemporal comprehensive brain representation based on the second brain spatial representation and the second brain temporal representation through a time-aware brain representation extractor, as an initial brain decoding model.

[0139] The first determining module 307 is used to determine the first CCM between the brain spatiotemporal integrated representation and the second brain spatial representation.

[0140] The second determining module 308 is used to determine the second CCM between the brain spatiotemporal integrated representation and the second brain time representation.

[0141] The third determining module 309 is used to determine that the sum of the first CCM and the second CCM is the objective function of the initial brain decoding model, and to train the initial brain decoding model according to the objective function to generate a brain decoding model.

[0142] Decoding module 310 is used to perform brain decoding based on the brain decoding model.

[0143] Figure 4 This is a schematic diagram of the structure of an fMRI brain decoding device based on a self-supervised spatiotemporal graph model provided in an embodiment of the present invention, as shown below. Figure 4 As shown, the fMRI brain decoding device based on a self-supervised spatiotemporal graph model can include the above-mentioned... Figure 3 The fMRI brain decoding device based on a self-supervised spatiotemporal graph model is shown. Optionally, the fMRI brain decoding device 410 based on the self-supervised spatiotemporal graph model may include a first processor 2001.

[0144] Optionally, the fMRI brain decoding device 410 based on a self-supervised spatiotemporal graph model may also include a memory 2002 and a transceiver 2003.

[0145] The first processor 2001, memory 2002, and transceiver 2003 can be connected via a communication bus.

[0146] The following is combined Figure 4 The components of the fMRI brain decoding device 410 based on a self-supervised spatiotemporal graph model are described in detail below:

[0147] The first processor 2001 is the control center of the fMRI brain decoding device 410 based on a self-supervised spatiotemporal graph model. It can be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 can be one or more central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0148] Optionally, the first processor 2001 can perform various functions of the fMRI brain decoding device 410 based on a self-supervised spatiotemporal graph model by running or executing software programs stored in the memory 2002 and calling data stored in the memory 2002.

[0149] In a specific implementation, as one example, the first processor 2001 may include one or more CPUs, for example... Figure 4 CPU0 and CPU1 are shown in the diagram.

[0150] In a specific implementation, as one example, the fMRI brain decoding device 410 based on a self-supervised spatiotemporal graph model may also include multiple processors, for example... Figure 4 The first processor 2001 and the second processor 2004 are shown in the diagram. Each of these processors can be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). Here, "processor" can refer to one or more devices, circuits, and / or processing cores used to process data (e.g., computer program instructions).

[0151] The memory 2002 is used to store the software program that executes the present invention, and is controlled by the first processor 2001 to execute it. The specific implementation method can be referred to the above method embodiment, and will not be repeated here.

[0152] Optionally, the memory 2002 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed discs, laser discs, optical discs, digital universal discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory 2002 may be integrated with the first processor 2001 or may exist independently, and may be connected via the interface circuit of the fMRI brain decoding device 410 based on a self-supervised spatiotemporal graph model. Figure 4 (Not shown in the figure) is coupled to the first processor 2001, and the embodiments of the present invention do not specifically limit this.

[0153] The transceiver 2003 is used to communicate with network devices or with terminal devices.

[0154] Alternatively, transceiver 2003 may include a receiver and a transmitter. Figure 4 (Not shown separately). The receiver is used to implement the receiving function, and the transmitter is used to implement the sending function.

[0155] Optionally, the transceiver 2003 can be integrated with the first processor 2001 or exist independently, and can be connected to the interface circuit of the fMRI brain decoding device 410 based on a self-supervised spatiotemporal graph model. Figure 4 (Not shown in the figure) is coupled to the first processor 2001, and the embodiments of the present invention do not specifically limit this.

[0156] It should be noted that, ​ The structure of the fMRI brain decoding device 410 based on the self-supervised spatiotemporal graph model shown in the figure does not constitute a limitation on the router. Actual knowledge structure recognition devices may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0157] Furthermore, the technical effects of the fMRI brain decoding device 410 based on the self-supervised spatiotemporal graph model can be referred to the technical effects of the fMRI brain decoding method based on the self-supervised spatiotemporal graph model described in the above method embodiments, and will not be repeated here.

[0158] It should be understood that the first processor 2001 in this embodiment of the invention may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0159] It should also be understood that the memory in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0160] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0161] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0162] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.

[0163] It should be understood that, in various embodiments of the present invention, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0164] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0165] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0166] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0167] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0168] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0169] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0170] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for fMRI brain decoding based on a self-supervised spatiotemporal graph model, characterized in that, The method includes: The fMRI time series is acquired, and the fMRI time series is divided into multiple segments by sliding window. The multiple segments are then subjected to spatial graph convolution to extract brain spatial features and obtain the first brain spatial representation. The normalized autocorrelation of the time dimension of the fMRI time series is determined, the features of each time point in the fMRI time series are determined by the time attention map, the time attention weight is determined based on the features of the time point, and the first brain time representation is generated by the residual module based on the time attention weight and the normalized autocorrelation. Using a time-aware brain representation extractor, a comprehensive spatiotemporal brain representation is generated based on the first brain spatial representation and the first brain temporal representation. SCM is performed on the fMRI time series, and a second brain spatial representation is obtained based on the fMRI time series after SCM. The fMRI time series was subjected to TAM, and a second brain time representation was generated based on the TAM-enhanced fMRI time series. Using a time-aware brain representation extractor, an enhanced spatiotemporal comprehensive brain representation is generated based on the second brain spatial representation and the second brain temporal representation, serving as the initial brain decoding model. Determine the first CCM between the aforementioned spatiotemporal integrated brain representation and the second brain spatial representation; Determine the second CCM between the aforementioned spatiotemporal integrated brain representation and the second brain temporal representation; The sum of the first CCM and the second CCM is determined as the objective function of the initial brain decoding model. The initial brain decoding model is trained according to the objective function to generate a brain decoding model. Brain decoding is performed based on the aforementioned brain decoding model; The SCM process on the fMRI time series is performed as follows: Through formula Determine the spatial masking probability of each ROI corresponding to the fMRI time series, where SMP i Let d represent the spatial masking probability of the i-th ROI. i This represents the degree of the i-th ROI; The ROIs with a spatial masking probability greater than a first set value are masked; The step of performing TAM on the fMRI time series includes: Through formula Determine the importance of each timestamp corresponding to the fMRI time series, where TI j c represents the importance of the j-th timestamp. j Indicates the displacement between timestamps; Timestamps whose importance is greater than the second set value are masked.

2. The method according to claim 1, characterized in that, The determination of the first CCM between the brain spatiotemporal integrated representation and the second brain spatial representation is performed by the following formula: Wherein, C represents the cross-correlation matrix between the spatiotemporal integrated brain representation and the second brain spatial representation, s,i represents the row or column number of C, and α is used for balancing and The constant representing the difference between them.

3. The method according to claim 1, characterized in that, The determination of the second CCM between the brain spatiotemporal integrated representation and the second brain temporal representation is performed using the following formula: Wherein, C represents the cross-correlation matrix between the aforementioned spatiotemporal integrated brain representation and the second brain temporal representation, t and j represent the row or column numbers of C, and α is used for balancing. and The constant representing the difference between them.

4. The method according to claim 1, characterized in that, The process of generating a first brain temporal representation based on the temporal attention weights and the normalized autocorrelation, through a residual module, includes: Based on the aforementioned time attention weights, the formula CAM = softmax(Conv1([NACT,TAM]))∈R is used. N×L×L Generate CAM, where TAM is the result after normalizing the temporal attention weights, NACT is the normalized autocorrelation, N represents the number of brain regions, and L represents the length of the sub-segment; Using the residual module, formula H t =Conv2(V(X)·CAM)+H s ∈R N×L×c Generate the first-time representation, where R is a real number; The sub-segment is divided into multiple first sub-segments of length 3 using a non-overlapping sliding window. Where L is the length of the sub-segment; The importance score of each timestamp and its left and right adjacent timestamps is learned through a self-attention mechanism. Where SA represents the self-attention mechanism, and R is a real number; Information is weighted and aggregated based on importance scores to obtain segment representations. Where sum represents the aggregation function in the time dimension; Based on H m The weights of each sub-segment are further assigned, and the fine representation of the entire fMRI time series is aggregated hierarchically to enhance the first temporal representation from local to global.

5. The method according to claim 1, characterized in that, The step of extracting brain spatial features from the multiple sub-segments through spatial graph convolution to obtain a first brain spatial representation includes: The static brain network at the group level is obtained by averaging the partial correlation coefficients between the time series. Based on the aforementioned static brain network and the aforementioned time series, using GNN, and through the formula... Extracting the static spatial representation H of the brain c θ is a non-linear activation function, W is a learnable parameter representing the weights, and b is a learnable parameter representing the bias. Let D be the adjacency relation of the static brain network, where D is the degree matrix, the diagonal elements are the degrees of each brain region, and I is the identity matrix. It is the time series; Through formula Generate dynamic spatial representations of the brain H d Where I is the identity matrix, E N Feature embedding for each brain region, W N b is a learnable parameter N These are learnable parameters; According to formula H S =β1H c +β2H d Based on the static spatial representation and the dynamic spatial representation of the brain, a first brain spatial representation is generated, wherein H c For the static spatial representation, H d This is the representation of the dynamic space. Where, α m Used for constraints β m Iterative updates are performed during the training process.

6. An fMRI brain decoding device based on a self-supervised spatiotemporal graph model, the device being used to implement the fMRI brain decoding method based on a self-supervised spatiotemporal graph model as described in any one of claims 1-5, characterized in that, The device includes: The first acquisition module is used to acquire fMRI time series, divide the fMRI time series into multiple segments through a sliding window, extract brain spatial features from the multiple segments through spatial graph convolution, and acquire the first brain spatial representation. The first generation module is used to determine the normalized autocorrelation of the time dimension of the fMRI time series, determine the features of each time stamp in the fMRI time series through the time attention map, determine the time attention weight based on the features of the time stamp, and generate the first brain time representation through the residual module based on the time attention weight and the normalized autocorrelation. The second generation module is used to generate a spatiotemporal comprehensive brain representation based on the first brain spatial representation and the first brain temporal representation using a time-aware brain representation extractor. The second acquisition module is used to perform SCM on the fMRI time series and acquire a second brain spatial representation based on the fMRI time series after SCM. The third generation module is used to perform TAM on the fMRI time series and generate a second brain time representation based on the TAM-enhanced fMRI time series. The fourth generation module is used to generate an enhanced spatiotemporal comprehensive brain representation based on the second brain spatial representation and the second brain temporal representation through a time-aware brain representation extractor, as an initial brain decoding model. The first determining module is used to determine the first CCM between the brain spatiotemporal integrated representation and the second brain spatial representation; The second determining module is used to determine the second CCM between the brain spatiotemporal integrated representation and the second brain temporal representation; The third determining module is used to determine that the sum of the first CCM and the second CCM is the objective function of the initial brain decoding model, and to train the initial brain decoding model according to the objective function to generate a brain decoding model. A decoding module is used to perform brain decoding based on the brain decoding model; The second acquisition module performs SCM on the fMRI time series in the following manner: Through formula Determine the spatial masking probability of each ROI corresponding to the fMRI time series, where SMP i Let d represent the spatial masking probability of the i-th ROI. i This represents the degree of the i-th ROI; The ROIs with a spatial masking probability greater than a first set value are masked; The third generation module performs TAM on the fMRI time series, including: Through formula Determine the importance of each timestamp corresponding to the fMRI time series, where TI j c represents the importance of the j-th timestamp. j Indicates the displacement between timestamps; Timestamps whose importance is greater than the second set value are masked.

7. An fMRI brain decoding device based on a self-supervised spatiotemporal graph model, characterized in that, The device includes: processor; A memory storing computer-readable instructions that, when executed by the processor, implement the method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains program code that can be invoked by a processor to execute the method as described in any one of claims 1 to 5.

Images