A method of spatiotemporal attention deep learning model using brain function image

Abstract

The application discloses a method of a spatiotemporal attention deep learning classification model of brain function images, and belongs to the technical field of medical image processing. In view of the problem that existing deep learning models cannot effectively extract the time domain change condition and spatial brain region change characteristics related to brain activity, and simultaneously improve the classification and prediction effect of brain image data, the method of the spatiotemporal attention deep learning model of brain function images is provided. The method combines the time domain attention and space domain attention modules, simultaneously extracts the spatiotemporal characteristics of brain function images, can not only obtain the change condition of the brain over time and the importance of different brain regions, but also improve the classification and prediction effect of the model.

Description

Technical Field

[0001] This invention belongs to the field of medical image processing technology, specifically relating to a method for using a spatiotemporal attention deep learning model based on brain functional images. This method can classify and predict samples using brain functional images, and can obtain the spatiotemporal features of the samples based on the trained model for further analysis. This method is expected to provide new ideas for neuroscience research, medical diagnosis of brain diseases, and other fields. Background Technology

[0002] With the continuous development of brain imaging technology, an increasing number of studies are dedicated to extracting useful information from large amounts of functional brain imaging data. Since brain activity changes dynamically over time, using functional brain imaging data to decode and analyze the brain is a crucial research method. Currently, a common approach is to use machine learning algorithms to classify and predict brain functional imaging data. However, due to the limited sample size of brain functional imaging data, existing methods generally suffer from low classification accuracy and poor interpretability.

[0003] With the remarkable performance of deep learning in computer vision and natural language processing, more and more researchers are using it to analyze brain functional imaging data. However, due to the black-box nature and poor interpretability of deep learning models, currently trained deep learning classification models cannot explicitly interpret the features in brain functional imaging data, limiting their analysis of brain structure and function. To address this, researchers have proposed some deep learning classification models that can indirectly analyze brain functional imaging data. They enhance the learning and analytical expression capabilities of deep learning models by adding attention mechanisms. However, existing attention mechanisms mainly target the channels and self-attention mechanisms of convolutional neural networks, extracting attention features only spatially without considering the temporal characteristics of time-series data, thus resulting in less than ideal performance on brain functional imaging data. Summary of the Invention

[0004] To address the problem that existing deep learning models cannot effectively extract temporal and spatial brain region changes related to brain activity, and to improve the classification and prediction performance using brain imaging data, this invention proposes a method for a spatiotemporal attention deep learning model using brain functional images. This method combines temporal and spatial attention modules to extract the spatiotemporal characteristics of brain functional images, which not only allows for the acquisition of brain changes over time and the importance of different brain regions, but also improves the model's classification and prediction performance.

[0005] A method for a spatiotemporal attention deep learning model utilizing brain functional imaging, the method comprising:

[0006] Step 1: Obtain time-series matrix data of all brain regions for each subject.

[0007] Furthermore, the specific process of step 1 is as follows:

[0008] Step 1.1: Based on the brain function imaging data of each subject, the brain is divided into different brain regions according to the brain region template, and all brain regions are divided into different functional domains;

[0009] Step 1.2: Based on the brain functional imaging data corresponding to each brain region, the time series signal of the brain region is calculated by averaging the time series signals of all voxels in that brain region;

[0010] Step 1.2.1: Assume X is the time-series matrix data of all brain regions of a subject, and y is the class label of the subject, where X is a two-dimensional matrix. Each element in the matrix represents the signal of the corresponding brain region at the corresponding time point, where N is the number of brain regions and T is the number of time points;

[0011] Step 1.3: Obtain the time series of each brain region.

[0012] Step 2: Input the time-series matrix data of all brain regions of each subject into a spatiotemporal attention deep learning model using brain functional imaging. This model consists of a temporal attention module, a spatial attention module, and a classification module. Calculate the loss using the squared loss function based on the model output and train the model using gradient descent until the trained model achieves its highest accuracy on an independent dataset.

[0013] Furthermore, the specific process of step 2 is as follows:

[0014] Step 2.1: Using the Temporal Attention Module (TAM) described in formula (1), the temporal attention weights at each time point are learned from the time-series matrix data of all brain regions of each subject. The model is as follows:

[0015]

[0016] Among them, X t =X T , X is the input data for the temporal attention module. t,j The signals of all brain regions at time point j, j∈{1,…,T}; LSTM TAM and MLP TAM These are the two basic modules that make up the temporal attention module; LSTM TAMIt consists of a basic long short-term memory (LSTM) network layer, used to extract features at different time points; MLP TAM It consists of a basic multilayer perceptron (MLP) layer for further feature extraction from features at different time points; Softmax and Average are the normalization exponential function and the averaging operation, respectively; α j (j=1,…,T) represents the output of the temporal attention module, which represents the temporal attention weights at all time points;

[0017] Step 2.2: Using the Spatial Attention Module (SAM) described in Formula (2), the spatial attention weights of each brain functional domain are learned from the time-series matrix data of all brain regions of each subject. The model is as follows:

[0018]

[0019] Among them, X s =X, This serves as the input data for the spatial attention module; the spatial attention module employs C LSTMs. SAM Modules process information from different brain functional domains, LSTM SAM,c Used to process the c-th brain functional domain; the c-th LSTM SAM The module's input is X s,[c] This input represents X s Time-series matrix data of all brain regions in the c-th functional domain of the brain; MLP SAM It consists of a basic multilayer perceptron layer, with different functional domains using the same MLP. SAM Feature processing is performed; Softmax and Average are the normalization exponential function and the averaging operation, respectively; β c (c=1,…,C) represents the output of the spatial attention module, which represents the spatial attention weights of all brain functional domains;

[0020] Step 2.3: Calculate the time-series matrix data of all brain regions with enhanced spatiotemporal attention for each subject using formula (3). The calculation formula is as follows:

[0021]

[0022] Where α is the temporal attention weight vector, α=[α1,α2,…,α T ], X t =X T , β represents the input data for the temporal attention module; β is the spatial attention weight vector, β = [β1, β2, ..., β2]. c ], X s =X, For the spatial attention module, ⊙ represents the Hadamard product; X z Time-series matrix data for all brain regions with enhanced spatiotemporal attention;

[0023] Step 2.4: Using formula (4), calculate the probability of the model predicting each label based on the time-series matrix data of all brain regions of each subject enhanced by spatiotemporal attention. The calculation formula is as follows:

[0024] cls = Sigmoid(LSTM) cls (MLP cls (X z (4)

[0025] Among them, MSTM cls and MLP cls The two basic modules that make up the classification module; LSTM cls It consists of a basic LSTM, used to extract features from the time-series matrix data of all brain regions for each subject; MLP cls It consists of a basic MLP that further extracts and classifies features from brain functional imaging data; Sigmoid is the activation function; cls is the predicted label result;

[0026] Step 2.5: Calculate the loss using the squared loss function based on the model output cls and update and train the weights using gradient descent until the trained model achieves its highest accuracy when tested on an independent dataset.

[0027] Step 3: Input the time-series matrix data samples of all brain regions of the new subjects into the trained model to obtain the temporal attention, spatial attention and label prediction results output by the model for subsequent analysis.

[0028] Furthermore, in step 3, the time-series matrix data of all brain regions of the new subject are input into the trained model for analysis, and the temporal attention, spatial attention and label prediction results of the sample are saved. The results of temporal attention and spatial attention are further applied to the analysis of brain functional images.

[0029] Compared with the prior art, the present invention has the following advantages:

[0030] Compared to existing deep learning classification models applied to brain functional imaging data, the model proposed in this invention achieves improved classification accuracy (approximately 6% improvement compared to the current best model). Furthermore, the model integrates temporal and spatial attention mechanisms developed specifically for brain functional imaging data. This not only enhances classification accuracy but also allows for the analysis and study of brain biomarkers and changes in brain activity over time using learned attention patterns across different brain regions and at different time points. This will provide new research ideas for neuroscience research and the medical diagnosis of brain diseases. Additionally, the spatiotemporal attention mechanism proposed in this invention can be applied to image analysis in the medical field for other brain imaging data with time-series characteristics. Attached Figure Description

[0031] Figure 1 This is a schematic diagram of a deep learning model and analysis process for brain image classification based on spatiotemporal attention mechanisms;

[0032] Figure 2 This is a schematic diagram of attention in different brain regions obtained based on spatial attention.

[0033] Figure 3 This is a schematic diagram illustrating the changes over time in temporal attention, classification confidence, and spatial attention in each brain functional domain. Detailed Implementation

[0034] The objectives, advantages, and features of this invention will be illustrated and explained through the following non-limiting description of preferred embodiments. These embodiments are merely typical examples of applying the technical solutions of this invention, and all technical solutions formed by equivalent substitutions or equivalent transformations fall within the scope of protection claimed by this invention.

[0035] This invention discloses a method for a spatiotemporal attention deep learning model utilizing brain functional imaging, such as... Figure 1 As shown, the method includes the following steps:

[0036] Step 1: Obtain the time-series matrix data of all brain regions for each subject. Assume X represents the time-series matrix data of all brain regions for a given subject, and y represents the subject's class label. Here, X is a two-dimensional matrix, with... Each element in the matrix represents the signal of the corresponding brain region (the row corresponding to the element) at the corresponding time point (the column corresponding to the element), where N is the number of brain regions and T is the number of time points.

[0037] Step 2: Input the time-series matrix data of all brain regions of each subject into a spatiotemporal attention deep learning model using brain functional imaging. This model consists of a temporal attention module, a spatial attention module, and a classification module. Calculate the loss using the squared loss function based on the model output and train the model using gradient descent until the accuracy of the trained model reaches its maximum on an independent dataset.

[0038] Step 3: During testing and analysis, save the temporal attention, spatial attention, and classification results of the tested samples. The results of temporal attention and spatial attention can be further applied to the analysis of brain functional images.

[0039] Performance comparison:

[0040] The spatiotemporal attention deep learning model of the present invention utilizing brain functional imaging was tested on the HumanConnectome Project (HCP) dataset. The method of the present invention was evaluated using classification accuracy metrics tested on independent test data, and a comparative analysis was conducted.

[0041] Specifically, the experiment used 41,796 samples from the task-related fMRI dataset of the Human Connectome Project (HCP). These samples were collected under different cognitive states and included data from seven different task categories: emotion, gambling, language, motor, relational, social, and working memory tasks. Each task also included several sub-tasks, totaling 23 sub-tasks across all seven categories. The length of the fMRI data for different sub-tasks varied, ranging from 15 to 39 time points.

[0042] Under the same experimental settings, including data, experimental parameters, and training parameters, five representative classification methods were selected as comparison methods (Support Vector Machine (SVM), Multilayer Perceptron (MLP), Long Short-Term Memory (LSTM), Gate Recurrent Unit (GRU), and Graph Convolutional Network (GCN)) and compared with the method of this invention, Spatio-Temporal Attention Long Short-Term Memory (STA-LSTM). The experimental results are as follows:

[0043] Table 1: Experimental Results of Various Classification Methods

[0044] method accuracy STA-LSTM 0.9259 SVM 0.7520 MLP 0.7841 LSTM 0.8445 GRU 0.8225 GCN 0.8644

[0045] This invention proposes a deep learning classification model integrating spatiotemporal attention. By fusing temporal and spatial attention mechanisms developed specifically for brain functional imaging data, the model's classification accuracy is improved. In the experiments, accuracy was used to measure the model's performance. The results in Table 1 show that the classification accuracy of the proposed method is superior to other compared classification methods, indicating that the proposed method can significantly improve the classification performance of brain functional images.

[0046] This invention proposes a deep learning classification model integrating spatiotemporal attention. By fusing temporal and spatial attention mechanisms developed for brain functional imaging data, it can analyze and study the changes in brain biomarkers and brain activity over time using the learned attention from different brain regions and at different time points. The spatial and temporal attention data for all samples were preserved in the experiment. Figure 2 and Figure 3 These are visualizations of spatial attention and temporal attention, respectively. These results can be used to analyze and study changes in the brain and biomarkers over time for different tasks.

[0047] This invention has many other embodiments, and all technical solutions formed by equivalent transformation or equivalent transformation fall within the protection scope of this invention.

[0048] Contents not described in detail in this specification are prior art known to those skilled in the art. Although illustrative specific embodiments of the invention have been described above to facilitate understanding by those skilled in the art, it should be understood that the invention is not limited to the scope of the specific embodiments. Various modifications are readily apparent to those skilled in the art as long as they fall within the spirit and scope of the invention as defined and determined by the appended claims, and all inventions utilizing the concept of this invention are protected.

Claims

1. A method for a spatiotemporal attention deep learning model utilizing brain functional imaging, characterized in that: The method includes: Step 1: Obtain time-series matrix data for all brain regions of each subject; Step 2: Input the time-series matrix data of all brain regions of each subject into the spatiotemporal attention deep learning model using brain functional imaging. The model consists of a temporal attention module, a spatial attention module, and a classification module. Calculate the loss using the squared loss function based on the model output and train the model using gradient descent until the accuracy of the trained model on independent datasets reaches its maximum. Step 2 includes: Step 2.1: Using the temporal attention module described in formula (1), the temporal attention weights at each time point are learned from the time-series matrix data of all brain regions of each subject. The model is as follows: ； in, This is the input data for the temporal attention module. For the first j Signals from all brain regions at a given time point ; and These are the two basic modules that make up the temporal attention module; It consists of a basic long short-term memory network layer, used to extract features at different time points; It consists of a basic multilayer perceptron layer, which is used to further extract features from features at different time points; Softmax and Average These are the normalized exponential function and the average calculation operation, respectively. The output of the temporal attention module represents the temporal attention weights at all time points; Step 2.2: Use the spatial attention module described in formula (2) Spatial Attention Module, SAM The spatial attention weights for each brain functional domain were learned from the time-series matrix data of all brain regions for each subject. The model is as follows: ； in, This serves as the input data for the spatial attention module; the spatial attention module uses... C indivual The module processes information from different functional domains of the brain. Used to process the c The first brain functional domain, c indivual The module's input is This input represents The Middle c Time-series matrix data of all brain regions in each brain functional domain; It consists of a basic multilayer perceptron, with different functional domains using the same... Perform feature processing. Softmax and Average These are the normalized exponential function and the average calculation operation, respectively. The output of the spatial attention module represents the spatial attention weights of all brain functional domains; Step 3: Input the time-series matrix data samples of all brain regions of the new subjects into the trained model to obtain the temporal attention, spatial attention and label prediction results output by the model for subsequent analysis.

2. The method for a spatiotemporal attention deep learning model utilizing brain functional imaging according to claim 1, characterized in that: The specific process of step 1 is as follows: Step 1.1: Based on the brain function imaging data of each subject, the brain is divided into different brain regions according to the brain region template, and all brain regions are divided into different functional domains; Step 1.2: Based on the brain functional imaging data corresponding to each brain region, the time series signal of the brain region is calculated by averaging the time series signals of all voxels in that brain region; Step 1.2.1: Assumption X This is a time-series matrix of brain regions for a given subject. y The category label for this subject, where, X It is a two-dimensional matrix, with Each element in the matrix represents the signal of the corresponding brain region at the corresponding time point. N The number of brain regions. T The number of time points; Step 1.3: Obtain the time series of each brain region.

3. The method for a spatiotemporal attention deep learning model utilizing brain functional imaging according to claim 1, characterized in that: The specific process of step 2 also includes: Step 2.3: Calculate the time-series matrix data of all brain regions with enhanced spatiotemporal attention for each subject using formula (3). The calculation formula is as follows: ， in, This is the temporal attention weight vector. , which is the input data for the temporal attention module; This is the spatial attention weight vector. , This is the input data for the spatial attention module; It represents the Hadamardi (or Hadama) stack; Time-series matrix data for all brain regions with enhanced spatiotemporal attention; Step 2.4: Using formula (4), calculate the probability of the model predicting each label based on the time-series matrix data of all brain regions of each subject enhanced by spatiotemporal attention. The calculation formula is as follows: ， in, and These are the two basic modules that make up the classification module; From a basic LSTM The composition is used to extract features from the time-series matrix data of all brain regions for each subject; From a basic MLP The composition involves further feature extraction and classification of brain functional imaging data; Sigmoid For activation functions; cls The prediction results for the labels; Step 2.5: Based on the results output by the model cls The loss is calculated using the squared loss function, and the weights are updated and trained using gradient descent until the trained model achieves its highest accuracy when tested on an independent dataset.

4. The method for a spatiotemporal attention deep learning model utilizing brain functional imaging according to claim 1, characterized in that: The specific process of step 3 is as follows: Step 3.1: Input the time-series matrix data of all brain regions of the new subject into the trained model for analysis, and save the temporal attention, spatial attention and label prediction results of the sample. The results of temporal attention and spatial attention are further applied to the analysis of brain functional images.

Images