Space-time fusion neural network construction method and system

Abstract

The application relates to a spatio-temporal fusion neural network construction method, which comprises the following steps: constructing a DFC network; calculating a functional connection matrix to form a dynamic functional connection network; modeling the functional connection matrix corresponding to each time window into a graph structure to obtain a series of graph structures, and realizing dynamic functional connection construction; constructing a multi-granularity brain network; realizing spatio-temporal feature extraction; obtaining a dynamic functional connection analysis result; performing feature extraction, averaging the features of all time steps to obtain a global feature representation, obtaining a final output feature, and obtaining an fMRI time sequence analysis result; splicing the dynamic functional connection analysis result and the fMRI time sequence analysis result together to form a final feature representation; and outputting a prediction label to realize multi-scale feature fusion. The application also relates to a spatio-temporal fusion neural network construction system. The application can synchronously extract and deeply fuse spatio-temporal features, and comprehensively improve the accuracy and robustness of mental disease recognition.

Description

Technical Field

[0001] This invention relates to a method and system for constructing a spatiotemporal fusion neural network. Background Technology

[0002] With the development of neuroimaging, the research value of functional connectivity networks (FCNs) constructed based on functional magnetic resonance imaging (fMRI) data in the auxiliary diagnosis of mental illnesses has become increasingly significant. In recent years, dynamic functional connectivity (DFC), as a time-varying extension of the traditional static FCN, has gradually become a research focus because it can capture the dynamic co-activation patterns of the brain at different time scales.

[0003] Unlike traditional static functional networks (FCNs), digital functional networks (DFCs) more realistically reflect the spontaneous activity changes of the brain during rest. They not only contain rich temporal dynamic features but also preserve the spatial structural information of brain networks. Therefore, modeling strategies based on DFCs must not only capture the changing trends of time series but also take into account the ability to express the topological relationships between brain regions, which also places higher demands on the spatiotemporal coupling of modeling methods.

[0004] To achieve more effective identification of brain diseases, recent research has gradually shifted towards simultaneously modeling the temporal characteristics and spatial structural information of fMRI data. How to fuse these two types of heterogeneous features within a unified framework has become one of the core issues in the application of deep learning in the assisted diagnosis of mental illnesses.

[0005] Although various deep learning methods based on fMRI data have been applied to the identification of mental illnesses, there are still many shortcomings in model design and feature modeling. Existing techniques often use only single-level functional connectivity features for spatial modeling, failing to systematically integrate local and global brain connectivity information, resulting in limited spatial structural representation capabilities. Simultaneously, most studies focus only on one type of time-varying information in fMRI time series or dynamic functional connectivity (DFC) for temporal modeling, ignoring the complementary characteristics of the two over time, thus limiting the model's comprehensive characterization of brain dynamics. Furthermore, existing techniques generally employ the separation and shallow fusion of temporal and spatial features, failing to fully explore their deep-seated interactions; the few network structures that attempt spatiotemporal fusion also lack systematic optimization in heterogeneous feature collaborative modeling, thus limiting model performance and generalization ability.

[0006] In summary, while existing technologies have made some progress in improving model performance, they still face the following common problems:

[0007] Most network architectures have not yet made structural-level optimizations for the spatiotemporal feature fusion of DFC; insufficient modeling of multi-scale spatial information (such as local and global connectivity features) affects the expressive power of the model; spatiotemporal interaction layers usually adopt simple splicing or post-fusion strategies, which limit the ability to model complex interaction relationships.

[0008] Therefore, how to design a neural network with structural fusion capabilities to support multi-scale spatial modeling and synchronous analysis of multi-source temporal information is a key problem that has not yet been fully solved. Summary of the Invention

[0009] In view of this, it is necessary to provide a method and system for constructing a spatiotemporal fusion neural network.

[0010] This invention provides a method for constructing a spatiotemporal fusion neural network, comprising the following steps: S1, constructing a DFC network based on a preprocessed fMRI time series matrix; calculating a set of functional connectivity matrices for data within each time window to form a dynamic functional connectivity network; modeling the functional connectivity matrix corresponding to each time window as a graph structure, thereby obtaining a series of graph structures to achieve dynamic functional connectivity construction; S2, performing multi-granularity analysis on the functional connectivity network corresponding to each time window to construct a multi-granularity brain network; the spatiotemporal fusion neural network STFNN extracts spatiotemporal features through a multi-layer network structure; thus obtaining the result Z of the dynamic functional connectivity analysis. DFC S3. The Transformer model is introduced to extract features from the time series. An average pooling mechanism is used to average the features from all time steps, thus obtaining a global feature representation. A linear transformation is then applied to obtain the final output features, yielding the results of the fMRI time series analysis. S4, the results of the dynamic function link analysis Z DFC Results of fMRI time series analysis The features are directly concatenated to form the final feature representation; then processed through a fully connected layer to output the predicted label, thus achieving multi-scale feature fusion.

[0011] Preferably, step S1 includes:

[0012] First, a DFC network is constructed using the sliding window method; given a preprocessed fMRI time series matrix H∈R T×N Where T is the number of time points and N is the number of brain regions; setting the window size to W and the sliding step size to S, the data corresponding to each time window m is the time series matrix H. m ∈R W×N The number of sliding windows, i.e. the number of subsequences, is calculated as shown in equation (1):

[0013]

[0014] in, Indicates rounding down;

[0015] Next, for the data within each time window m, a functional connectivity matrix FC is obtained by calculating the Pearson correlation coefficient of each brain region within that time window. m This process is iterated over the entire time series, with the sliding window moving S time steps each time, repeating the above calculation until all time windows have been processed. This yields a set of functional connectivity matrices, which together constitute the dynamic functional connectivity network DFC = {FC1, FC2, ..., FC...}. M};

[0016] Subsequently, the function connection matrix FC corresponding to each time window m is... m Model as a graph structure G m This results in a series of graph structures {G1,G2,…,G}. M}

[0017] Preferably, the spatiotemporal fusion neural network mainly consists of three modules: a spatial feature extraction module, a temporal feature extraction module, and a spatiotemporal synchronization module; the spatial feature extraction module is used to capture the static spatial characteristics of the brain network; the temporal feature extraction module focuses on capturing the dynamic characteristics of functional connections changing over time; and the spatiotemporal synchronization module is used to fuse spatial and temporal features to achieve joint modeling of the two.

[0018] Preferably, step S2 includes:

[0019] This application introduces a multi-granularity analysis method: for each time window m, the functional connectivity network (FC) m Multi-granularity analysis is performed to construct a multi-granularity brain network; subsequently, STFNN extracts spatiotemporal features through a multi-layer network structure; each layer of the network consists of a spatial feature extraction module and a temporal feature extraction module.

[0020] Preferably, the multi-granularity brain network includes: a fine-grained brain network, a coarse-grained brain network, and an intra-module brain network; the multi-granularity brain network can respectively reflect the fine structure, global characteristics, and modular characteristics of brain functional connections, thereby comprehensively characterizing the multi-level spatial information of brain functional connections.

[0021] Preferably, step S2 further includes:

[0022] First, a linear transformation is used to align the features to the same dimension to ensure data consistency. Then, the aligned features are concatenated to obtain the final feature representation Z. DFC .

[0023] Preferably, step S3 includes:

[0024] An average pooling mechanism is used to average the features across all time steps, thus obtaining a global feature representation; let Z... Encoder =Z (1) Z (2) ,…,Z (T) The features after averaging are integrated. As shown in equation (13):

[0025]

[0026] Through this process, the high-dimensional features corresponding to each time step of the original time series are compressed into a global feature vector with the same dimension as the number of time points in the original time series. Next, a fully connected layer is introduced to align the features to a uniform dimension through linear transformation, adapting to the requirements of subsequent tasks. The final output feature vector... As shown in equation (14):

[0027]

[0028] Among them, FC l (·) represents the linear transformation operation of the l-th layer.

[0029] Preferably, step S4 includes:

[0030] The feature representation Z obtained through multi-scale analysis DFC and The results of dynamic functional connectivity analysis and fMRI time series analysis are directly concatenated to form the final feature representation; finally, the feature representation is processed through a fully connected layer to output the predicted label.

[0031] This invention also provides a spatiotemporal fusion neural network construction system, which includes a dynamic functional connectivity construction unit, a dynamic functional connectivity analysis unit, an fMRI time series analysis unit, and a multi-scale feature fusion unit, wherein:

[0032] The dynamic functional connectivity building unit is used to construct a DFC network based on the preprocessed fMRI time series matrix. For the data in each time window, a set of functional connectivity matrices is calculated to form a dynamic functional connectivity network. The functional connectivity matrix corresponding to each time window is modeled as a graph structure, and then a series of graph structures are obtained, thereby realizing the construction of dynamic functional connectivity.

[0033] The dynamic functional connectivity analysis unit is used to perform multi-granularity analysis on the functional connectivity network corresponding to each time window, constructing a multi-granularity brain network; the spatiotemporal fusion neural network STFNN extracts spatiotemporal features through a multi-layer network structure; thus obtaining the result Z of the dynamic functional connectivity analysis. DFC ;

[0034] The fMRI time series analysis unit is used to introduce the Transformer model to extract features from the time series. It employs an average pooling mechanism to average the features from all time steps, thereby obtaining a global feature representation. A linear transformation is then performed to obtain the final output features, yielding the results of the fMRI time series analysis.

[0035] The multi-scale feature fusion unit is used to integrate the results of dynamic functional connectivity analysis Z... DFC Results of fMRI time series analysis The features are directly concatenated to form the final feature representation; then processed through a fully connected layer to output the predicted label, thus achieving multi-scale feature fusion.

[0036] This application enables multi-scale spatial feature modeling of functional connectivity networks, integrated analysis of dual-source time-varying information from fMRI and DFC, and simultaneous extraction and deep fusion of spatiotemporal features through a structure-optimized spatiotemporal fusion mechanism, thereby comprehensively improving the accuracy and robustness of mental illness identification.

[0037] Furthermore, compared with currently widely used baseline models, such as BrainNetCNN, GCN, GIN (spatial feature extraction), LSTM, GRU, TGCN (temporal feature modeling), and ST-GCN, STAGCN, MSSTAN (spatiotemporal fusion), the multi-scale spatiotemporal fusion neural network model (MS-STFNN) proposed in this invention shows significant advantages in structural design, modeling ability, and classification performance, mainly in the following aspects:

[0038] (1) It has strong structural innovation and solves the three core bottlenecks of traditional models.

[0039] This invention overcomes the limitations of existing "separate modeling of spatial and temporal features" in its modeling strategy. It integrates GraphSAGE, Gated Recurrent Units (GRUs), and Transformer mechanisms in a structured manner to model the dynamic functional connectivity (DFC) and the multi-granular spatial structure and multi-source temporal features in the original fMRI time series, respectively. Compared to traditional single-modality, single-scale methods, MS-STFNN achieves more comprehensive feature representation in both spatial and temporal dimensions through its multi-level modeling structure.

[0040] (2) The accuracy of multi-task classification is consistently leading, and the performance is significantly improved.

[0041] MS-STFNN achieved the best results in all four classification tasks related to MDD and its subtype identification:

[0042] The classification accuracy of major depressive disorder (MDD) compared to the healthy control group (NC) was 70.83%; the classification accuracy of recurrent major depressive disorder (rMDD) compared to the healthy control group (NC) was 81.12%; the classification accuracy of first-episode depressive disorder (FEDN) compared to the healthy control group (NC) was 75.53%; and the classification accuracy of recurrent major depressive disorder (rMDD) compared to first-episode depressive disorder (FEDN) was 75.67%.

[0043] Compared to other spatiotemporal modeling methods (such as ST-GCN, STAGCN, and MSSTAN), MS-STFNN outperforms other methods in all evaluation metrics (accuracy, sensitivity, specificity, and AUC), demonstrating its high adaptability and stability for brain disease classification tasks.

[0044] (3) The fusion strategy is efficient and significantly improves the ability of spatiotemporal feature collaborative modeling.

[0045] This invention designs a feature alignment mechanism to uniformly process the outputs from the DFC analysis module and the time series analysis module, achieving multi-source, multi-scale spatiotemporal feature fusion. Compared with simple splicing fusion models, MS-STFNN can simultaneously capture temporal dependencies and spatial structural changes within the same network layer, significantly enhancing the model's ability to capture complex brain activity patterns.

[0046] (4) The model training process is complete, with strong scalability and engineering deployment capabilities.

[0047] The proposed model is an end-to-end structure with good module decoupling and scalability. Each sub-module (such as the multi-granularity GraphSAGE sub-network or the Transformer temporal modeling module) can be independently deployed in other medical imaging tasks as needed. Furthermore, this structure has strong compatibility with input data types and classification targets, enabling further applications in the auxiliary diagnosis of other mental illnesses or neurodegenerative diseases.

[0048] In summary, MS-STFNN surpasses existing best-in-class technologies in terms of structural innovation, cross-task performance, spatiotemporal coordination capabilities, and practicality, fully demonstrating its technological advancement and application prospects in multi-scale fMRI analysis and brain disease identification tasks. Attached Figure Description

[0049] Figure 1 This is a flowchart of the spatiotemporal fusion neural network construction method of the present invention;

[0050] Figure 2 This is a schematic diagram of the overall method provided in an embodiment of the present invention;

[0051] Figure 3 A schematic diagram of a time feature extraction module provided in an embodiment of the present invention;

[0052] Figure 4 This is a hardware architecture diagram of the spatiotemporal fusion neural network construction system of the present invention. Detailed Implementation

[0053] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0054] See Figure 1 The diagram shown is a flowchart of a preferred embodiment of the spatiotemporal fusion neural network construction method of the present invention. Please refer to it as well. Figure 2 :

[0055] Step S1, Dynamic Functional Connectivity Construction. This involves: constructing a DFC network based on the preprocessed fMRI time-series matrix; calculating a set of functional connectivity matrices for the data within each time window, forming a dynamic functional connectivity network; and modeling the functional connectivity matrix corresponding to each time window as a graph structure, thereby obtaining a series of graph structures and achieving dynamic functional connectivity construction. Specifically, this includes:

[0056] First, a DFC network is constructed using the sliding window method. Given a preprocessed fMRI time series matrix H∈R... T×N Where T is the number of time points and N is the number of brain regions. Setting the window size to W and the sliding step size to S, the data corresponding to each time window m is a time series matrix H. m ∈R W×N The number of sliding windows, i.e. the number of subsequences, is calculated as shown in equation (1):

[0057]

[0058] in, This indicates rounding down to the nearest integer.

[0059] Next, for the data within each time window m, a functional connectivity matrix FC is obtained by calculating the Pearson correlation coefficient of each brain region within that time window. mThis functional connectivity matrix reflects the strength of functional connections between different brain regions within a given time period. This process is iterated over the entire time series, with the sliding window moving S time steps each time, repeating the calculation until all time windows have been processed. The final result is a set of functional connectivity matrices, which together constitute the dynamic functional connectivity network DFC = {FC1, FC2, ..., FC...}. M}

[0060] Subsequently, the function connection matrix FC corresponding to each time window m is... m Model as a graph structure G m This results in a series of graph structures {G1,G2,…,G}. M In the graph structure described above, the nodes represent different brain regions or ROIs (Regions of Interest), and the edges between nodes correspond to the strength of functional connectivity.

[0061] Step S2, Dynamic Functional Connectivity Analysis. This involves performing multi-granularity analysis on the functional connectivity network corresponding to each time window, constructing a multi-granularity brain network; the spatiotemporal fusion neural network STFNN extracts spatiotemporal features through a multi-layer network structure; thus obtaining the result Z of the dynamic functional connectivity analysis. DFC Specifically:

[0062] Since the Distance-Focused Connection (DFC) can reflect not only the time-varying characteristics of functional connections but also the topological relationships of brain networks, this application proposes a Spatio-Temporal Fusion Neural Network (STFNN) framework. This framework aims to simultaneously extract spatio-temporal features from the DFC and learn the multi-level connections between these features. The STFNN framework mainly consists of three modules: a spatial feature extraction module, a temporal feature extraction module, and a spatio-temporal synchronization module. Each module performs a specific function: the spatial feature extraction module captures the static spatial characteristics of the brain network; the temporal feature extraction module focuses on capturing the dynamic characteristics of functional connections over time; and the spatio-temporal synchronization module fuses spatial and temporal features to achieve joint modeling and explore the complex interaction between spatial structure and temporal dynamics.

[0063] To fully explore the spatial characteristics of dynamic functional connections, this application introduces a multi-granularity analysis method. Specifically:

[0064] First, for each time window m, the corresponding functional connection network FC mMulti-granularity analysis is performed to construct a multi-granularity brain network, which includes fine-grained brain networks, coarse-grained brain networks, and intra-module brain networks. This multi-granularity brain network can reflect the fine structure, global characteristics, and modular characteristics of brain functional connections, thereby comprehensively characterizing the multi-level spatial information of brain functional connections.

[0065] Subsequently, STFNN extracts spatiotemporal features through a multi-layer network structure. Each layer consists of a spatial feature extraction module and a temporal feature extraction module. In the spatial feature extraction module, the GraphSAGE model is used to learn representations of the graph structure to capture the spatial characteristics of the DFC. Specifically:

[0066] An architecture consisting of a single-layer GraphSAGE layer was designed, combined with Dropout and normalization layers to improve the robustness and generalization ability of STFNN. After passing through this spatial feature extraction module, the model can learn the preliminary feature set of the functional connectivity network. in, and These represent the feature embeddings corresponding to the fine-grained brain network, coarse-grained brain network, and intra-module brain network output by the first-layer spatial feature extraction module, respectively.

[0067] After the first-layer spatial feature extraction module, the feature set corresponding to all time windows is: The aforementioned feature set is passed as input to the next layer of the network to continue the feature extraction process.

[0068] Simultaneously, the feature set output by the first-layer spatial feature extraction module is input into the temporal feature extraction module to capture the temporal dependencies within it. The temporal feature extraction module is as follows: Figure 3 As shown. Since the feature matrix at this time is a node embedding matrix, it is first transformed into a graph embedding matrix through average pooling. Then, the GRU network is used for feature extraction. The feature extraction process of GRU is shown in equation (2):

[0069]

[0070] in, This represents the node embedding matrix after average pooling. GRU(·) indicates the feature extraction process using GRU, and Output represents the output of GRU, which generates a vector at each time step. To obtain a global representation, Output is averaged to obtain the output of the temporal feature extraction module, i.e., the output of the first layer of the network.

[0071] A three-layer network is used to synchronously extract spatiotemporal features, ultimately yielding three output results. and These represent spatiotemporal features at different levels. To obtain a final feature representation containing multi-level information, the features are first aligned to the same dimension using a linear transformation to ensure data consistency. Then, the aligned features are concatenated to obtain the final feature representation Z. DFC .

[0072] Step S3, fMRI time series analysis. This involves introducing a Transformer model to extract features from the time series, using average pooling to average the features across all time steps to obtain a global feature representation, and then performing a linear transformation to obtain the final output features, thus yielding the results of the fMRI time series analysis. Specifically:

[0073] Original fMRI time series H∈R T×N This application introduces the Transformer model to effectively model the time-varying characteristics of brain regions, reflecting the time-varying patterns of various brain regions at different times. Through its powerful global dependency capture capability, it extracts features from time series data.

[0074] In this embodiment, the Transformer's Encoder module is used as the primary modeling tool. The basic building block of the Encoder is the Encoder Layer, and multiple Encoder Layers are stacked to form a complete Encoder. Within the EncoderLayer, the original fMRI time series H∈R... T×N The input embedding X is transformed through an embedding layer and positional encoding. Then, Q (query), K (key), and V (value) are calculated using the formulas shown in equations (3), (4), and (5):

[0075] Q = XW Q (3)

[0076] K = XW K (4)

[0077] V = XW V (5)

[0078] Among them, W Q W K W V It is a learnable weight matrix. Next, the attention weights are calculated and summed using weighted averages, as shown in equation (6):

[0079]

[0080] Where, d k is the dimension of the key vector. The outputs of multiple attention heads are concatenated and linearly transformed, as shown in equation (7):

[0081] MultiHead(Q,K,V)=Concat(head1,head2,…,head h W O (7)

[0082] Each attention head i The attention mechanism is obtained through independent computation, and the computation process is shown in equation (8). O It is a weight matrix that undergoes linear transformation after concatenation, and h represents the number of attention heads in the multi-head attention mechanism.

[0083] head i =Attention(QWi i Q ,KW i K VW i V (8)

[0084] Subsequently, the multi-head attention output and input are residually connected and normalized to obtain Z1. Then, a position-by-position nonlinear transformation is performed on Z1. Combining the residual and normalization, the final output Z2 of this layer is obtained. The calculation process is shown in Equations (9), (10), and (11):

[0085] Z1=LayerNorm(X+MultiHead(Q,K,V))(9)

[0086] FFN(x)=ReLU(xW1+b1)W2+b2 (10)

[0087] Where W1 and W2 are learnable weights, and b1 and b2 are biases.

[0088] Z2=LayerNorm(Z1+FFN(Z1))(11)

[0089] In this model, two encoder layers are stacked, with the output of each layer serving as the input to the next layer. The encoder outputs the feature Z. Encoder As shown in equation (12):

[0090] Z Encoder =EncoderLayer2(EncoderLayer1(X))(12)

[0091] The final output feature Z of the encoder EncoderThis model captures the complex dependencies between each time step and other time steps in the original time series. To aggregate these features and obtain more representative time series information, this model employs an average pooling mechanism to average the features of all time steps, thus obtaining a global feature representation. Let Z... Encoder =Z (1) Z (2) ,…,Z (T) The features after averaging are integrated. As shown in equation (13):

[0092]

[0093] Through this process, the high-dimensional features corresponding to each time step of the original time series are compressed into a global feature vector with the same dimension as the number of time points in the original time series. Next, a fully connected layer is introduced to align the features to a uniform dimension through linear transformation, adapting to the requirements of subsequent tasks. The final output feature vector... As shown in equation (14):

[0094]

[0095] Among them, FC l (·) represents the linear transformation operation of the l-th layer.

[0096] Step S4, multi-scale feature fusion. That is: fusing the results Z from the dynamic functional connectivity analysis. dFC Results of fMRI time series analysis The features are directly concatenated to form the final feature representation. Then, they are processed through a fully connected layer to output predicted labels, achieving multi-scale feature fusion. Specifically:

[0097] Feature Z obtained through multi-scale analysis DFC and These correspond to the results of dynamic functional connectivity analysis and fMRI time series analysis, respectively. Since the above features have already been aligned to the same dimension through linear transformation, they are directly concatenated to form the final feature representation. Finally, a fully connected layer is used for processing to output the predicted labels.

[0098] See Figure 4 The diagram shown is a hardware architecture diagram of the spatiotemporal fusion neural network construction system 10 of this invention. Please refer to it as well. Figure 2 The system includes: a dynamic functional connectivity construction unit 101, a dynamic functional connectivity analysis unit 102, an fMRI time series analysis unit 103, and a multi-scale feature fusion unit 104. Among them:

[0099] The dynamic functional connectivity construction unit 101 is used for dynamic functional connectivity construction. That is, a DFC network is constructed based on the preprocessed fMRI time series matrix; for the data in each time window, a set of functional connectivity matrices is calculated to form a dynamic functional connectivity network; the functional connectivity matrix corresponding to each time window is modeled as a graph structure, thereby obtaining a series of graph structures, thus realizing dynamic functional connectivity construction.

[0100] The dynamic functional connectivity analysis unit 102 is used for dynamic functional connectivity analysis. That is, it performs multi-granularity analysis on the functional connectivity network corresponding to each time window, constructing a multi-granularity brain network; the spatiotemporal fusion neural network STFNN extracts spatiotemporal features through a multi-layer network structure; thereby obtaining the result Z of the dynamic functional connectivity analysis. DFC .

[0101] The fMRI time series analysis unit 103 is used for fMRI time series analysis. Specifically, it introduces a Transformer model to extract features from the time series, employs an average pooling mechanism to average the features across all time steps to obtain a global feature representation, and then performs a linear transformation to obtain the final output features, thus yielding the result of the fMRI time series analysis.

[0102] The multi-scale feature fusion unit 104 is used for multi-scale feature fusion. That is, it integrates the results of the dynamic functional connectivity analysis Z... DFC Results of fMRI time series analysis The features are directly concatenated to form the final feature representation. Then, they are processed through a fully connected layer to output predicted labels, achieving multi-scale feature fusion.

[0103] This application enriches the temporal modeling dimension by fMRI time series (reflecting the time-varying nature of brain region activity) and dynamic functional connectivity (DFC, reflecting the time-varying nature of relationships between brain regions). Ablation experiments show that the task can be completed using either of these information sources alone (e.g., using only brain region sequences or only DFC) with certain performance, but the fusion of both significantly improves classification performance (e.g., accuracy increases from 79.44% or 77.58% to 81.12%). Therefore, alternative implementations using "only DFC" or "only fMRI time series" as input should also be included in the scope of protection.

[0104] In DFC sequence modeling, this application uses a gated recurrent unit (GRU) as the base network for temporal modeling. However, experimental results show that the Long Short-Term Memory (LSTM) network also performs well in this task (accuracy of 78.31%) and can be considered an equivalent replacement for GRU. Similarly, in the fMRI time series modeling part, the Transformer model can be replaced by LSTM or GRU. Although the performance is slightly reduced (e.g., the AUC of Transformer is 0.74, while it is 0.70 when replaced by LSTM), it still has strong temporal modeling capabilities. Therefore, The components are interchangeable under this structure, and related structures can all be regarded as alternative technical solutions to this invention.

[0105] This invention aligns features from different sources and scales to the same dimension using linear transformation before splicing and fusing them, thereby achieving effective information integration. Ablation experiments show that even without the alignment strategy, the model can be trained and predicted with acceptable performance (accuracy of 79.23%). Therefore, "feature misalignment + subsequent fusion layer compensation" can also be considered an alternative solution and falls within the scope of this invention.

[0106] This invention employs the STFNN framework to simultaneously extract temporal and spatial features at each layer. Experiments also tested the effectiveness of a phased "space + time" strategy: first, GraphSAGE is used to extract spatial features, then GRU is used to model temporal dependencies, achieving an overall accuracy of 77.33%. While this strategy is not as performance-efficient as STFNN, it still achieves spatiotemporal information fusion. Therefore, this alternative structural sequence of "extracting spatial features first and then modeling temporal dependencies" can also serve as an equivalent technical solution for achieving the objectives of this invention.

[0107] In summary, to achieve the core objective of "multi-scale spatiotemporal information joint modeling" proposed in this invention, some modules possess flexible technical implementation space. Different model structures (such as GRU / LSTM, STFNN / staged structure), different feature source combination methods (such as single or dual-source time-varying features), and different fusion strategies (aligned or unaligned) can all constitute effective equivalent implementation schemes of this invention. To prevent related technologies from being circumvented at the structural substitution level, all the above solutions should be included within the protection scope of this invention.

[0108] Experimental verification:

[0109] To ensure the stability and robustness of the classification results, this embodiment strictly employs ten-fold cross-validation for model training and validation, and finally calculates the average performance index of the ten results. This embodiment uses a grid search method to systematically optimize the key hyperparameters of the model, and the final parameter configuration is shown in Tables 1-4.

[0110] Table 1. Demographic information for the MDD vs. Normal Control (NC) classification task.

[0111] (M represents male, F represents female. Mean ± standard deviation)

[0112]

[0113] Table 2. Demographic information for rMDD vs. NC classification tasks.

[0114]

[0115]

[0116] Table 3. Demographic information for FEDN vs. NC classification tasks.

[0117]

[0118] Table 4. Demographic information for the rMDD vs. FEDN classification task.

[0119]

[0120] To ensure the stability and robustness of the classification results, this invention strictly employs a 10-fold cross-validation strategy for model training and validation, ultimately calculating the average performance index of ten iterations. This invention uses a grid search method to systematically optimize the model's key hyperparameters. Specifically, model training uses the Adam optimizer with an initial learning rate of 0.001 and a fixed step-size decay strategy, where the step size is set to 30 epochs and the decay parameter gamma is set to 0.8. Simultaneously, L2 regularization is introduced to prevent overfitting. To comprehensively evaluate the model's classification performance, this invention selects ACC, SEN, SPE, and AUC as evaluation metrics and conducts comparative experiments with three different types of baseline models, focusing on spatial feature extraction, temporal feature extraction, and spatiotemporal feature fusion extraction, respectively. All comparative experiments use a 10-fold cross-validation strategy, ensuring consistent data partitioning across all models through a fixed random seed. (The bolded values ​​in the table below are the optimal values.)

[0121] Table 5. Performance comparison results of the proposed model and the baseline model on the MDD vs. NC classification task.

[0122]

[0123] Table 6. Performance comparison results of the proposed model and the baseline model on the rMDD vs. NC classification task.

[0124]

[0125]

[0126] Table 7. Performance comparison results of the proposed model and the baseline model on the FEDN vs. NC classification task.

[0127]

[0128] Table 8. Performance comparison results of the proposed model and the baseline model on the rMDD vs. FEDN classification task.

[0129]

[0130] In this invention, the ablation study results of the multi-scale analysis method in the rMDD vs. NC classification task are shown in the table below. "Brain region" indicates analysis of only the original fMRI time series, modeling the time-varying information of brain regions in the original fMRI time series. "Functional connectivity" indicates analysis of only the DFC, modeling the time-varying information in the DFC. The results show that, compared to the single-scale analysis method, the multi-scale analysis method achieved the best performance on all classification evaluation metrics. This indicates that integrating the time-varying information of fMRI time series and dynamic functional connectivity can significantly improve the model's classification ability. Furthermore, the results show that the time-varying information in dynamic functional connectivity exhibits superior performance in the classification task compared to the time-varying information in fMRI time series. This suggests that functional connectivity has stronger discriminative power in rMDD classification and can more comprehensively reflect the functional state and dynamic changes of the brain network.

[0131] Table 9 Ablation Experiment Results from Multiscale Analysis

[0132]

[0133] The ablation study results of the GRU model in the rMDD vs. NC classification task in this invention are shown in the table below. Experimental results show that GRU exhibits the best classification performance across all evaluation metrics. In contrast, the classification performance using Transformer is the lowest. This may be because Transformer is generally suitable for long sequences, while in short sequence tasks, Transformer may be too complex and less efficient than GRU in capturing short-term dependencies. The classification performance using LSTM is also lower than GRU, possibly because in short time series tasks, the complex gating mechanism and slower training speed of LSTM make its effect less direct and efficient than GRU.

[0134] Table 10 Ablation Experiment Results of GRU Model

[0135]

[0136]

[0137] The ablation study results of the Transformer model in the rMDD vs. NC classification task are shown in the table below. Experimental results show that, compared to LSTM and GRU, Transformer exhibits the best classification performance across all evaluation metrics. This may be due to the self-attention mechanism employed by Transformer, which effectively models global contextual information in longer time series, making it particularly suitable for capturing long-term dependencies in the data, thus performing better in the rMDD classification task. Compared to LSTM and GRU, Transformer is more flexible in handling complex time series relationships, especially in long-series scenarios, where it can more fully leverage its advantage in capturing long-term dependencies.

[0138] Table 11 Ablation Experiment Results of Transformer Model

[0139]

[0140] The ablation study results of the feature alignment strategy in the rMDD vs. NC classification task in this invention are shown in the table below. In this experiment, "feature misalignment" refers to the misalignment of feature dimensions at different layers, which are directly concatenated. The experimental results show that the feature alignment strategy can significantly improve the classification performance of the model. This may be because the feature alignment strategy can ensure that features at different levels are integrated under the same dimension, thereby improving feature consistency and information fusion. By aligning features at different layers, the model can better learn the relationships between time-varying patterns, reduce unnecessary noise effects, and thus effectively improve classification accuracy.

[0141] Table 12 Ablation Experiment Results of Feature Alignment Strategy

[0142]

[0143]

[0144] The ablation study results of the spatiotemporal fusion neural network in the rMDD vs. NC classification task are shown in the table below. The results show that STFNN achieved the best performance across all classification evaluation metrics, while the classification performance decreased when using the "space + time" approach. This indicates that the spatiotemporal synchronous extraction mechanism in STFNN plays a crucial role in the rMDD vs. NC classification task. By synchronously extracting spatiotemporal features, STFNN can more accurately capture the dynamic changes in temporal and spatial features, thereby improving the model's classification performance. Unlike the traditional approach of extracting spatial features first and then modeling temporal dependencies, spatiotemporal synchronous extraction can better capture the interactions and dependencies between space and time, enabling the model to comprehensively understand the functional state and dynamic changes of the brain network at the spatiotemporal level, providing more accurate classification results.

[0145] Table 13 Ablation Experiment Results of Spatiotemporal Fusion Neural Network

[0146]

[0147] This invention proposes a multi-scale spatiotemporal fusion neural network model. By introducing fine-grained, coarse-grained, and intra-module structure-based multi-scale brain network analysis methods, it achieves multi-level feature extraction of functional connectivity networks in the spatial dimension. Simultaneously, it jointly models dual-source time-varying features from fMRI time series and DFC, using Transformer and GRU to capture global and local temporal dependencies respectively. Furthermore, through a structured spatiotemporal fusion framework, it synchronously extracts and integrates spatiotemporal features, enhancing its interactive modeling capabilities. In addition, this invention designs a feature alignment mechanism and a unified fusion module, improving model efficiency and diagnostic accuracy while maintaining expressive power, providing a more efficient and accurate modeling scheme for the intelligent identification of mental illnesses such as MDD.

[0148] Although the present invention has been described with reference to the present preferred embodiments, those skilled in the art should understand that the above preferred embodiments are only used to illustrate the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for constructing a spatiotemporal fusion neural network, characterized in that, The method includes the following steps: S1. Construct a DFC network based on the preprocessed fMRI time series matrix; calculate a set of functional connectivity matrices for the data in each time window to form a dynamic functional connectivity network; model the functional connectivity matrix corresponding to each time window as a graph structure to obtain a series of graph structures, thereby realizing the construction of dynamic functional connectivity. S2, multi-granularity analysis is performed on the functional connectivity network corresponding to each time window to construct a multi-granularity brain network; the spatiotemporal fusion neural network STFNN extracts spatiotemporal features through a multi-layer network structure; thus obtaining the results of dynamic functional connectivity analysis. ; S3 introduces the Transformer model for feature extraction from the time series. An average pooling mechanism is used to average the features from all time steps, thus obtaining a global feature representation. A linear transformation is then applied to obtain the final output features, yielding the results of the fMRI time series analysis. ; S4, the results of dynamic function link analysis. Results of fMRI time series analysis These are directly spliced ​​together to form the final feature representation; Then, the predicted labels are processed through a fully connected layer to achieve multi-scale feature fusion; where: The multi-granularity brain network includes: fine-grained brain network, coarse-grained brain network, and intra-module brain network; The spatiotemporal fusion neural network consists of three modules: a spatial feature extraction module, a temporal feature extraction module, and a spatiotemporal synchronization module. The spatial feature extraction module is used to learn the representation of the graph structure using the GraphSAGE model to capture the spatial characteristics of the DFC. The temporal feature extraction module is used to extract features using a GRU network. The spatiotemporal synchronization module is used to fuse spatial and temporal features to achieve joint modeling of the two.

2. The method as described in claim 1, characterized in that, Step S1 includes: First, a DFC network is constructed using the sliding window method; given the preprocessed fMRI time series matrix... Where T is the number of time points and N is the number of brain regions; the window size is set to... The sliding step size is Then each time window The corresponding data is a time series matrix. The number of sliding windows, i.e. the number of subsequences, is calculated as shown in equation (1): （1） in, Indicates rounding down; Next, for each time window By calculating the Pearson correlation coefficients of various brain regions within the time window, a functional connectivity matrix is ​​obtained from the data within that time window. Iterate over the entire time series, moving the sliding window each time. Repeat the above calculations at each time step until all time windows have been processed, ultimately yielding a set of functional connectivity matrices. The combination of these functional connectivity matrices constitutes the dynamic functional connectivity network. ; Then, each time window Corresponding functional connection matrix Model as a graph structure This leads to a series of graph structures. .

3. The method as described in claim 2, characterized in that, Step S2 includes: Introducing a multi-granularity analysis method: for each time window Corresponding function network connection Multi-granularity analysis is performed to construct a multi-granularity brain network; subsequently, STFNN extracts spatiotemporal features through a multi-layer network structure; each layer of the network consists of a spatial feature extraction module and a temporal feature extraction module.

4. The method as described in claim 3, characterized in that, The multi-granularity brain network can reflect the fine structure, global characteristics, and modular characteristics of brain functional connectivity, thereby comprehensively depicting the multi-level spatial information of brain functional connectivity.

5. The method as described in claim 4, characterized in that, Step S2 further includes: First, a linear transformation is used to align the features to the same dimension to ensure data consistency. Then, the aligned features are concatenated to obtain the final feature representation. .

6. The method as described in claim 5, characterized in that, Step S3 includes: An average pooling mechanism is used to average the features across all time steps, thereby obtaining a global feature representation; let... The features after averaging are integrated. As shown in equation (13): （13） Through this process, the high-dimensional features corresponding to each time step of the original time series are compressed into a global feature vector with the same dimension as the number of time points in the original time series. Next, a fully connected layer is introduced to align the features to a uniform dimension through linear transformation, adapting to the requirements of subsequent tasks. The final output feature vector... As shown in equation (14): （14） in, Indicates the first Linear transformation operations of layers.

7. The method as described in claim 6, characterized in that, Step S4 includes: Feature representation obtained through multi-scale analysis and The results of dynamic functional connectivity analysis and fMRI time series analysis are directly concatenated to form the final feature representation; finally, the feature representation is processed through a fully connected layer to output the predicted label.

8. A spatiotemporal fusion neural network construction system utilizing the spatiotemporal fusion neural network construction method of claim 1, characterized in that, The system includes a dynamic functional connectivity building unit, a dynamic functional connectivity analysis unit, an fMRI time series analysis unit, and a multi-scale feature fusion unit, wherein: The dynamic functional connectivity building unit is used to construct a DFC network based on the preprocessed fMRI time series matrix. For the data in each time window, a set of functional connectivity matrices is calculated to form a dynamic functional connectivity network. The functional connectivity matrix corresponding to each time window is modeled as a graph structure, and then a series of graph structures are obtained, thereby realizing the construction of dynamic functional connectivity. The dynamic functional connectivity analysis unit is used to perform multi-granularity analysis on the functional connectivity network corresponding to each time window, constructing a multi-granularity brain network; the spatiotemporal fusion neural network STFNN extracts spatiotemporal features through a multi-layer network structure; thus obtaining the results of dynamic functional connectivity analysis. ; The fMRI time series analysis unit is used to introduce the Transformer model to extract features from the time series. It employs an average pooling mechanism to average the features from all time steps, thereby obtaining a global feature representation. A linear transformation is then performed to obtain the final output features, yielding the results of the fMRI time series analysis. ; The multi-scale feature fusion unit is used to integrate the results of dynamic functional connectivity analysis. Results of fMRI time series analysis The features are directly concatenated to form the final feature representation; then processed through a fully connected layer to output the predicted label, thus achieving multi-scale feature fusion.

Images