An emotion disorder recognition method based on multi-modal brain image fusion

Abstract

The application discloses a kind of based on multi-modal brain image fusion mood disorder identification method, belong to medical image processing and artificial intelligence auxiliary diagnosis technical field;Including the following steps: obtaining multi-modal brain image, registration to spherical space and alignment;After extracting and fusing features, spherical space-time self-attention coding and gate modulation are carried out;Modulated features are recursively down-sampled from initial level to penultimate level, and the aforementioned operation is repeated at each layer;In this level, features are processed in parallel, fused and compressed to the lowest level, while extracting clustering embedding features;Global features are aggregated at the lowest level, and after fusing with clustering embedding features, the recognition result is output by the clustering perception classifier;The application realizes multi-modal fusion and dynamic feature extraction under the maintenance of cerebral cortex topological structure, improves recognition accuracy and generalization ability.

Description

Technical Field

[0001] This invention relates to the fields of medical image processing and artificial intelligence-assisted diagnosis, and in particular to a method for identifying mood disorders based on multimodal brain image fusion. Background Technology

[0002] The cerebral cortex, as the material basis for higher cognitive and emotional processing in humans, has always been a core topic in neuroscience research due to the complex coupling mechanism between its structure and function. Resting-state functional magnetic resonance imaging (fMRI) based on blood oxygen level-dependent signals has become the mainstream method for probing the brain's spontaneous activity and intrinsic functional network architecture. However, traditional fMRI analysis is mostly based on voxels in Euclidean space or regions of interest based on prior maps. This voxel-based or coarse brain region-based analysis method often ignores the highly folded geometric and topological characteristics of the cerebral cortex. Although cortical surface reconstruction technology can map the brain into a high-resolution spherical grid, revealing microscopic morphological features that are difficult to detect with traditional voxel analysis, most deep learning models still struggle to effectively utilize this high-dimensional non-Euclidean space data. This results in the loss of a large amount of subtle pathological information hidden deep in the cortical sulci during data dimensionality reduction or spatial flattening.

[0003] To address the aforementioned issues, existing technologies can be broadly categorized into three solutions: First, methods based on Euclidean space convolutional neural networks (CNNs) extract spatial features by standardizing MRI images into a three-dimensional voxel matrix and then using a three-dimensional convolutional network. However, this approach suffers from a fundamental geometric mismatch problem. The cerebral cortex is inherently a non-Euclidean manifold, and forcibly mapping it to a regular voxel grid leads to severe topological disruption. This causes vertices that were originally adjacent on the cortical surface to be severed in voxel space, or spatially distant gyri to be incorrectly merged. This geometric distortion is particularly severe for high-resolution spherical grid data, directly resulting in the blurring of microscopic cortical features. Second, methods based on graph convolutional networks (GCNNs) typically divide the brain into regions of interest (ROIs) as graph nodes based on prior brain maps. Edges are constructed using functional or structural connections, and then graph convolutional networks are applied for classification and prediction. However, current graph convolutional network solutions are mostly limited by computational complexity, making it difficult to directly process whole-brain vertex data. They are forced to significantly downsample to low resolution, which essentially comes at the cost of sacrificing high-frequency spatial information. In addition, most existing graph network models only focus on static graph structures, making it difficult to effectively model the key dynamic functional connectivity changes in mood disorders. Thirdly, in terms of multimodal fusion strategies, existing technologies mostly adopt shallow paradigms of early or late fusion. The former directly concatenates the original data or feature vectors of different modalities at the input, while the latter performs weighted voting on the independent classification results of different modalities. These methods ignore the deep nonlinear interactions between modalities. Especially when processing functional magnetic resonance imaging data with a time dimension and static structural data, existing solutions lack effective time-space alignment mechanisms, resulting in the failure to organically integrate high-dimensional spatiotemporal dynamic information with anatomical features. Summary of the Invention

[0004] The purpose of this invention is to provide a method for identifying mood disorders based on multimodal brain image fusion, which achieves multimodal fusion and dynamic feature extraction while maintaining the topological structure of the cerebral cortex, thereby improving the recognition accuracy and generalization ability.

[0005] To address the aforementioned technical problems, embodiments of the present invention provide a method for identifying mood disorders based on multimodal brain image fusion, comprising the following steps: Structural magnetic resonance imaging (SMRI) and resting-state functional magnetic resonance imaging (fMRI) were acquired. The SMRI and resting-state fMRI images were then registered to the same standard spherical space to obtain spatially aligned spherical structural features and spherical functional time series. On a standard spherical space, spatial structural features of spherical structural features and spatiotemporal functional features of spherical functional time series are extracted; the spatial structural features are extended along the time dimension to align with the spatiotemporal functional features, and the aligned spatial structural features and spatiotemporal functional features are fused to obtain joint features; For the joint features, self-attention encoding is performed in the spherical spatial dimension and the temporal dimension respectively; the self-attention encoded features are split into functionally relevant components and structurally relevant components, and the structurally relevant components are used to modulate the functionally relevant components through a gating fusion mechanism to obtain modulated multimodal features; The modulated multimodal features are processed recursively from the first resolution level corresponding to the standard spherical space through multi-scale spherical space downsampling and mapped to the penultimate resolution spherical level. When recursively processing to each spherical level, feature extraction, alignment and fusion, self-attention encoding, and gated modulation operations are repeatedly performed on that spherical level. At the penultimate resolution spherical level, the multimodal features obtained from this spherical level are processed and fused in parallel, and the features obtained after parallel processing and fusion are compressed and mapped to the lowest resolution spherical level, while clustering embedding features are extracted. At the lowest resolution spherical level, temporal and spatial aggregation are performed on the features obtained from the compression mapping of the second-to-last resolution spherical level to obtain sample-level global features. Clustering embedding features are fused with sample-level global features to form cluster-aware global discriminative features, which are then classified using a cluster-aware classifier to output the emotional disorder identification results.

[0006] In some optional embodiments, the multimodal features obtained at the penultimate resolution spherical layer are processed and fused in parallel, and the specific steps are as follows: Multimodal features are decomposed into functional components and structural components; The functional components and structural components are expanded in terms of channel dimension. The extended structural components are aligned with the extended functional components in the time dimension through a time broadcast operation, and then concatenated in the channel dimension to obtain the intermediate joint feature representation of this level.

[0007] In some optional embodiments, the specific steps for mapping the parallel processing and fused feature compression to the lowest resolution spherical layer are as follows: Joint compression of the intermediate joint feature representation is performed on both the time and channel dimensions; The jointly compressed features are mapped to the lowest resolution spherical layer using spherical spatial pooling.

[0008] In some optional embodiments, the extraction of clustering embedding features specifically includes the following steps: Based on the functional components at this level, the functional correlation between different spherical vertices is calculated in the time dimension to construct a sample-level functional connection matrix; The functional connectivity matrix is ​​input into a multilayer perceptron network for feature projection and compression to obtain low-dimensional clustering embedding features.

[0009] In some optional embodiments, at the lowest resolution spherical level, temporal aggregation and spatial aggregation are performed on the features obtained from the compressed mapping of the second-to-last resolution spherical level to obtain sample-level global features. The specific steps are as follows: For the features obtained from the penultimate resolution spherical layer compression mapping, extract the feature parts corresponding to the left and right hemispheres respectively; For the feature parts of the left hemisphere and the feature parts of the right hemisphere, a learnable temporal attention pooling mechanism is used to weight and aggregate the time dimension to eliminate the time dimension, so as to obtain the time aggregated features of the left hemisphere and the time aggregated features of the right hemisphere. The time aggregation features of the left hemisphere and the time aggregation features of the right hemisphere are concatenated to obtain the concatenated overall features; Calculate the global space average pooling feature and the global space max pooling feature of the overall features after concatenation; The global spatial average pooling feature and the global spatial max pooling feature are concatenated to form the sample-level global feature.

[0010] In some optional embodiments, the steps of fusing clustering embedded features with sample-level global features to form cluster-aware global discriminative features, and then classifying them using a cluster-aware classifier to output the emotion disorder identification result, are as follows: Clustering embedding features are concatenated with sample-level global features to form cluster-aware global discriminative features; The global discriminative features are input into a cluster-aware classifier; wherein, the cluster-aware classifier contains multiple sub-classifiers, and the outputs of the multiple sub-classifiers are weighted and summed by the cluster membership degree to obtain the final classification result.

[0011] In some optional embodiments, the specific steps for extracting the spatial structural features of the spherical structure and the spatiotemporal functional features of the spherical functional time series are as follows: For spherical structural features, perform convolution operations along the spatial dimensions of the sphere to extract spatial structural features; For spherical functional time series, perform a joint convolution operation of spatial and temporal dimensions to extract spatiotemporal functional features.

[0012] In some optional embodiments, the steps of extending the spatial structure features along the time dimension to align with the spatiotemporal functional features, and then fusing the aligned spatial structure features with the spatiotemporal functional features to obtain joint features are as follows: By using time broadcasting, spatial structural features are copied and extended along the time dimension, aligning spatial structural features with spatiotemporal functional features in the time dimension. By splicing the time-aligned spatial structural features and spatiotemporal functional features along the channel dimension, a joint feature is obtained.

[0013] In some optional embodiments, the joint features are self-attention encoded in both the spherical spatial dimension and the temporal dimension; the self-attention encoded features are then decomposed into functionally relevant components and structurally relevant components, as follows: Self-attention encoding is performed in the time dimension: For each spherical vertex, the query, key and value vectors are constructed through linear mapping of its joint feature sequence at all time steps, and the time attention weights are calculated based on the scaling dot product attention mechanism. The value vectors are then weighted and aggregated to obtain the time-encoded features. Self-attention encoding is performed in the spherical space dimension: For each time step, the temporally encoded features of all spherical vertices are used to construct query, key, and value vectors through linear mapping, and spatial attention weights are calculated based on the scaling dot product attention mechanism. The value vectors are then weighted and aggregated to obtain the spatial-temporal self-attention encoded features. The features encoded by spatial-temporal self-attention are decomposed along their channel dimensions to obtain functionally relevant components and structurally relevant components.

[0014] In some optional embodiments, the modulation of the function-related components using the gated fusion mechanism to obtain the modulated multimodal features is carried out through the following steps: The modulation factor is generated based on the structure-related components, including inputting the structure-related components into the Sigmoid activation function for processing. The functionally correlated components are multiplied element-wise with the modulation factor to modulate the functionally correlated components and obtain the modulated multimodal features.

[0015] Embodiments of the present invention also provide a computer device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the above-described method for identifying mood disorders based on multimodal brain image fusion.

[0016] Embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which, when run by a processor, is capable of executing the above-described method for identifying mood disorders based on multimodal brain image fusion.

[0017] The mood disorder identification method based on multimodal brain image fusion provided by this invention has at least the following beneficial effects: This invention provides a method for identifying mood disorders based on multimodal brain image fusion. By achieving precise spatial alignment and deep fusion of multimodal features while fully preserving the inherent geometric topology of the cerebral cortex, it constructs an end-to-end intelligent identification framework that combines structural preservation, spatiotemporal awareness, and pathological heterogeneity modeling capabilities. This method overcomes the technical bottlenecks of geometric information loss, dynamic feature fragmentation, and superficial multimodal interaction in traditional brain image analysis. It achieves cross-scale joint representation from cortical microstructure to whole-brain functional networks, significantly improving the ability to capture and discriminate abnormal brain function patterns related to mood disorders. Simultaneously, this method demonstrates excellent stability and generalization performance in real clinical scenarios with multi-center, multi-parameter acquisition, and possesses good cross-disease transfer adaptability, requiring only minor parameter adjustments to extend to related mental illness identification tasks. This invention provides a holistic solution for the auxiliary diagnosis of mental illnesses based on brain imaging, combining theoretical originality and clinical applicability, effectively improving the reliability, interpretability, and clinical application value of the identification results. Attached Figure Description

[0018] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings:

[0019] Figure 1 This is a flowchart of a method for identifying mood disorders based on multimodal brain image fusion according to an embodiment of the present invention; Figure 2 This is a schematic diagram of a multimodal brain image preprocessing and spherical spatial alignment process according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the process of an emotion disorder identification method based on spherical multimodal fusion and clustering perception according to an embodiment of the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0021] One embodiment of the present invention relates to a method for identifying mood disorders based on multimodal brain image fusion. The implementation details of the method for identifying mood disorders based on multimodal brain image fusion in this embodiment are described in detail below. The following implementation details are provided for ease of understanding and are not necessary for implementing this solution.

[0022] The specific process of the emotion disorder recognition method based on multimodal brain image fusion in this embodiment can be described as follows: Figure 1 As shown, it includes: Step 101: Acquire structural magnetic resonance images and resting-state functional magnetic resonance images, and register the structural magnetic resonance images and resting-state functional magnetic resonance images to the same standard spherical space to obtain spatially aligned spherical structural features and spherical functional time series. In the specific implementation, the selection of subjects followed the clinical inclusion criteria of a Hamilton Depression Rating Scale (HAMD17) score of not less than 17, and strict quality control was implemented to exclude samples with an average frame shift greater than 0.3 mm or obvious image artifacts, so as to ensure that each subject had complete T1-weighted structural magnetic resonance imaging and resting-state functional magnetic resonance imaging data.

[0023] Data preprocessing employed a standardized workflow based on DeepPrep. For structural images, the Reconall workflow was first executed using FreeSurfer v7.2 software to reconstruct the cerebral cortex surface and extract five types of geometric features, including cortical thickness, curvature, sulcus depth, surface area, and volume. The reconstructed individual cortical surfaces were then registered to the fsaverage standard spherical space. For functional images, head motion correction, inter-slice temporal alignment, and boundary registration with the individual structural images were performed sequentially, and the images were mapped to the same spherical space. Noise reduction was then performed, including regression of 24 head motion parameters, white matter, and cerebrospinal fluid signals, followed by bandpass filtering at 0.01-0.1 Hz. Finally, Gaussian smoothing with a full width at half maximum (FWHM) of 6 mm was applied in the spherical space to maintain the continuity of the topological structure.

[0024] All preprocessed data are uniformly resampled to the fsaverage5 standard spherical grid, which has 10242 vertices, thus ensuring that the functional modes and structural modes are accurately corresponding in space during the input stage.

[0025] Based on the above preprocessing process, this invention uses the fsaverage5 standard spherical grid as a unified feature representation space to perform spatial alignment and joint modeling of functional magnetic resonance imaging time series and structural image features.

[0026] For a batch of sizes B The input samples, whose functional modal data are represented as having dimension 1. ,in =10242 This represents the number of vertices in the fsaverage5 spherical mesh. is the length of the functional time series. Its structural modal data is represented as having dimension . tensor The number of structural feature channels =5 These correspond to five types of geometric features extracted from each vertex: cortical thickness, curvature, groove depth, surface area, and volume. By mapping functional and structural information to the same spherical coordinate system, a precise one-to-one spatial correspondence is ensured for multimodal features during the input stage.

[0027] To support subsequent multi-scale feature learning, a hierarchical spherical representation system was pre-constructed during the preprocessing stage. Specifically, different spherical levels were defined, ranging from high resolution (ICO5) to low resolution (ICO4, ICO3, ICO2, ICO1). For each level... ℓ Vertex adjacency relationships are constructed based on the seven-neighbor topology structure of its rules. Nℓ And define the pooling mapping relationship from the previous level to the next level. Pℓ→ℓ−1 By using average pooling, features can be downsampled from the ICO5 spherical grid to the ICO4, ICO3, ICO2, and ICO1 grids in stages, thus forming a multi-resolution spherical feature representation framework that maintains strict geometric consistency, providing a foundation for subsequent feature mining at different scales.

[0028] Multimodal brain image preprocessing and spherical spatial alignment workflow as follows Figure 2 As shown in the figure, the process begins with raw data from resting-state functional magnetic resonance imaging (fMRI) and T1 structural magnetic resonance imaging (sMRI). Using the DeepPrep tool, parallel processing is performed sequentially, including AC-PC alignment, skull removal, tissue segmentation, and cortical reconstruction of the T1 structural images; discarding the starting point, correcting slice time, and correcting head movement for the functional images; spatial transformation is achieved through function-structure registration; and finally, the multimodal data is projected and resampled onto the fsaverage5 spherical grid. This complete preprocessing workflow provides a standardized data foundation with spatial alignment and topological preservation for subsequent brain image analysis.

[0029] Step 102: On the standard spherical space, extract the spatial structural features of the spherical structural features and the spatiotemporal functional features of the spherical functional time series; extend the spatial structural features along the time dimension to align with the spatiotemporal functional features, and fuse the aligned spatial structural features and spatiotemporal functional features to obtain joint features; For spherical structural features, perform a convolution operation along the spatial dimensions of the sphere to extract spatial structural features, which can be represented as: ; in Represents vertices The A spatial neighborhood, This represents the number of spatial neighborhoods corresponding to a structural mode, preferably the set of neighborhoods in a spherical mesh that are directly connected to the current vertex.

[0030] For the spherical function time series, perform a joint convolution operation of spatial and temporal dimensions to extract spatiotemporal function features, which can be represented as: ; in This represents the spatial neighborhood location associated with vertex 𝑣. This represents the offset of the corresponding neighborhood in the time dimension. This represents the total number of spatial-temporal joint neighborhoods.

[0031] By using time-based broadcasting, spatial structural features are replicated and extended along the time dimension, aligning them with spatiotemporal functional features in the time dimension. ; in (⋅) indicates that the structural features are copied and extended along the time dimension so that they share the same spatial structural prior at each time step.

[0032] By concatenating the time-aligned spatial structural features with the spatiotemporal functional features along the channel dimension, a joint feature is obtained, which can be represented as follows: ; in Indicates the time index is Functional modal characteristics, This represents the structural modal characteristics after time-based broadcasting. Concat(⋅) This indicates a splicing operation along the channel dimension.

[0033] Step 103: Perform self-attention encoding on the joint features in the spherical spatial dimension and the temporal dimension respectively; decompose the self-attention encoded features into functionally relevant components and structurally relevant components, and use the structurally relevant components to modulate the functionally relevant components through a gating fusion mechanism to obtain the modulated multimodal features; Self-attention encoding is performed in the time dimension: For each spherical vertex, the query, key and value vectors are constructed through linear mapping of its joint feature sequence at all time steps, and the time attention weights are calculated based on the scaling dot product attention mechanism. The value vectors are then weighted and aggregated to obtain the time-encoded features. In the time-dimensional encoding stage, this invention models the joint feature sequence of each spherical vertex at different time steps. Specifically, for any spherical vertex... Its joint feature across all time steps is expressed as:

[0034] ; To achieve self-attention modeling in the time dimension, we first construct query, key, and value vectors through linear mapping: ; in For learnable parameters, This represents the temporal attention subspace dimension. Subsequently, the attention weight matrix in the temporal dimension is calculated using the standard scaled dot product attention mechanism:

[0035] ; The value vector is then weighted and aggregated to obtain the time-coded output: ; The above process can adaptively model the long-term temporal dependencies of functional signals without introducing explicit temporal convolution kernels, while preserving the consistency constraints of structural priors in the temporal dimension.

[0036] After completing the temporal Transformer encoding, the temporal encoding results of all spherical vertices are recombined to obtain: ; Self-attention encoding is performed in the spherical space dimension: For each time step, the temporally encoded features of all spherical vertices are used to construct query, key, and value vectors through linear mapping, and spatial attention weights are calculated based on the scaling dot product attention mechanism. The value vectors are then weighted and aggregated to obtain the spatial-temporal self-attention encoded features. Subsequently, a spherical space Transformer encoding unit is introduced to globally model the spatial relationships between different spherical vertices within the same time step. Similarly, the Q / K / V spatial dimensions are constructed through linear mapping:

[0037] ; in , As learnable parameters, the spatial attention weights are calculated as follows: ; And based on this, the spatial encoding output is obtained: ; This spherical spatial Transformer can capture the nonlocal spatial interactions between any distant vertices on a spherical mesh without explicitly relying on the adjacency matrix or local convolution kernel, thus compensating for the shortcomings of traditional spherical convolution in long-range dependency modeling.

[0038] The features encoded by spatial-temporal self-attention are decomposed along their channel dimensions to obtain functionally relevant components and structurally relevant components.

[0039] because The joint feature is composed of structural and functional modes concatenated along the channel dimension, and its output after spatial-temporal Transformer encoding still contains both structural and functional components. Therefore, this invention re-decomposes the Transformer encoded output along the channel dimension into:

[0040] ; in ; The modulation factor is generated based on the structure-related components, including inputting the structure-related components into the Sigmoid activation function for processing. The functionally correlated components are multiplied element-wise with the modulation factor to modulate the functionally correlated components and obtain the modulated multimodal features.

[0041] Building upon this, a gated fusion mechanism based on structural prior is introduced, which modulates the functional enhancement branch step-by-step and vertex-by-vertex using the structural enhancement branch: ; in denoted by sigmoid activation function, and ⊙ denotes element-wise multiplication. Through this gating mechanism, structural modes are not directly superimposed linearly with functional modes, but rather act as dynamic modulation factors. This maintains the temporal integrity of the function while achieving global constraints and enhancements of the functional response by structural priors.

[0042] Step 104: Starting from the first resolution level corresponding to the standard spherical space, the modulated multimodal features are recursively processed and mapped to the penultimate resolution spherical level through multi-scale spherical space downsampling; when recursively processing to each spherical level, feature extraction, alignment fusion, self-attention encoding, and gated modulation operations are repeatedly performed on that spherical level. Step 105: At the penultimate resolution spherical level, the multimodal features obtained at the spherical level are processed and fused in parallel, and the features obtained after parallel processing and fusion are compressed and mapped to the lowest resolution spherical level, while extracting clustering embedding features. Multimodal features are decomposed into functional components and structural components; The functional components and structural components are expanded in terms of channel dimension. The spatiotemporal joint spherical convolution operator is used to progressively expand the dimension of the functional components, mapping the number of feature channels from 48 to 64 and then to 96, resulting in the expanded functional component representation as follows: ; Perform convolution mapping along the spherical spatial dimension, and also expand the number of channels of the structural components to 96, resulting in the expanded structural component representation as follows: ; The expanded structural components are aligned with the expanded functional components in the time dimension through a time broadcast operation, and then concatenated in the channel dimension to obtain the intermediate joint feature representation of this level. The intermediate joint feature representation is as follows: ; in This indicates the structural features after being broadcast over time.

[0043] Joint compression of the intermediate joint feature representation is performed on both the time and channel dimensions; Joint intermediate joint feature representation The input continues to the time-channel joint compression module to reduce computational redundancy caused by the time and channel dimensions while ensuring that key functional dynamic discrimination information is not lost. Let the intermediate joint feature obtained after function-structure fusion at the ICO2 spherical level be represented as...

[0044] ; in This represents the number of vertices corresponding to the ICO2 spherical mesh. and These represent the channel dimensions of the functional mode and the structural mode, respectively. The time length is represented. To simultaneously achieve downsampling in the time dimension and renormalization mapping in the channel dimension, this invention employs a two-dimensional convolution operator operating along the channel-time plane to progressively compress the features. First, the feature representation is transformed through a dimensionality rearrangement operation.

[0045] ; And apply the first-level temporal-channel joint convolution mapping: ; in Represents a two-dimensional convolution operator. This is a non-linear activation function. Through this mapping, the number of feature channels is... Compressed to a preset dimension, while the time length is reduced. Reduced to T / 2. Based on this, a second-level time-channel joint compression mapping is further applied:

[0046] ; This further compresses the channel dimension to [value] and reduces the time dimension to [value]. / 4. Subsequently, through a reverse dimension rearrangement operation, the features are restored to a representation dominated by spherical vertices:

[0047] ; The aforementioned progressive time-channel joint compression process achieves adaptive preservation of discriminative dynamic patterns in long-sequence functional signals without relying on explicit time pooling or fixed time window division.

[0048] The jointly compressed features are mapped to the lowest resolution spherical layer using spherical spatial pooling.

[0049] The spherical pooling operator maps features from the ICO2 spherical level to a lower-resolution ICO1 spherical mesh. Let the spherical pooling mapping be... Then we have:

[0050] ; in This represents the number of vertices in the ICO1 spherical mesh. The spherical pooling operation is based on a predefined spherical topological mapping relationship, and performs a weighted average of the features corresponding to the vertices of the high-resolution spherical mesh, thereby achieving spatial dimensionality reduction while maintaining the geometric consistency of the sphere.

[0051] Based on the functional components at this level, the functional correlation between different spherical vertices is calculated in the time dimension to construct a sample-level functional connection matrix; In a preferred embodiment, for functional components at the ICO2 level The functional correlation between different spherical vertices is calculated over time. For any sample... , its first With the The functional connectivity strength corresponding to each vertex of a sphere is defined as:

[0052] ; in Indicates the first The functional response sequence of each spherical vertex in the time dimension. This leads to the construction of a sample-level functional connectivity matrix:

[0053] ; The functional connectivity matrix is ​​input into a multilayer perceptron network for feature projection and compression to obtain low-dimensional clustering embedding features.

[0054] To enhance the discriminative power of the functional connectivity representation and incorporate it into an end-to-end trainable framework, this invention further performs feature projection modeling on the functional connectivity matrix. Specifically, the functional connectivity matrix is ​​flattened and then input into a multilayer perceptron network:

[0055] ; in Composed of linear mappings, nonlinear activation functions, and regularization operations, this method compresses high-dimensional functional connectivity features into low-dimensional clustering embeddings. After calculating the corresponding clustering embeddings in the left and right hemispheres respectively, they are concatenated along the feature dimension to form the low-dimensional clustering embedding features.

[0056] ; Step 106: At the lowest resolution spherical level, perform temporal aggregation and spatial aggregation on the features obtained from the compression mapping of the second-to-last resolution spherical level to obtain sample-level global features; For the features obtained from the penultimate resolution spherical layer compression mapping, extract the feature parts corresponding to the left and right hemispheres respectively; For the feature parts of the left hemisphere and the feature parts of the right hemisphere, a learnable temporal attention pooling mechanism is used to weight and aggregate the time dimension to eliminate the time dimension, so as to obtain the time aggregated features of the left hemisphere and the time aggregated features of the right hemisphere. The time aggregation features of the left hemisphere and the time aggregation features of the right hemisphere are concatenated to obtain the concatenated overall features; In a specific implementation, a learnable temporal attention pooling mechanism is introduced at the ICO1 level. For any spherical vertex... Its compressed time series characteristics can be expressed as:

[0057] ; Constructing temporal attention weights through linear mapping: ; in This represents a learnable parameter vector. Based on the aforementioned attention weights, a weighted aggregation of the time dimension is performed to obtain the spherical feature representation after removing the time dimension:

[0058] ; After performing the above temporal attention pooling operation on all spherical vertices, a sample-level spherical feature representation can be obtained: ; Considering the differences and complementarities in anatomical structure and functional organization between the left and right hemispheres, this invention preferably performs the above-mentioned modeling process separately for the left and right hemispheres, and after completing temporal attention pooling, stitches together the corresponding spherical features of the left and right hemispheres in the spatial dimension: ; Calculate the global space average pooling feature and the global space max pooling feature of the overall features after concatenation; The global spatial average pooling feature and the global spatial max pooling feature are concatenated to form the sample-level global feature.

[0059] The dimensions of the spherical vertices are aggregated using global spatial pooling operations, and the spatial average pooling feature and spatial max pooling feature are calculated respectively: ; and ; Step 107: The clustering embedding features are fused with the sample-level global features to form a cluster-aware global discriminative feature, which is then classified by a cluster-aware classifier to output the emotional disorder identification result.

[0060] In a specific embodiment, the global features are... Compared with the sample-level clustering feature vectors extracted at the ICO2 level Joint modeling is performed to form global discriminative features for cluster awareness: ; Based on this, a cluster-aware classifier is constructed, and the outputs of multiple sub-classifiers are weighted and fused using cluster membership degrees. Let the cluster membership vector be...

[0061] ; in Representing the number of clusters, then the th The output of each subclassifier is: ; The final classification output is obtained by weighted summation of the results from each sub-classifier: ; By introducing a clustering perception mechanism in the feature modeling, time compression, spatial aggregation, and classification decision stages, this invention explicitly models the potential subtype structure of brain functional patterns within an end-to-end trainable framework, thereby effectively improving the model's ability to express and discriminate the heterogeneity of complex brain imaging data.

[0062] The flowchart of the emotion disorder identification method based on spherical multimodal fusion and clustering perception is as follows: Figure 3As shown in the figure, a multimodal brain imaging method for identifying mood disorders based on spherical space is presented, using preprocessed functional time series data. X func and structural features X str As input, spatiotemporal features are extracted through spatial-temporal joint spherical convolution (3D-DINE) for functional modality and spherical spatial convolution (DINE) for structural modality, respectively. These features are then recursively processed at the ICO5 to ICO2 multi-resolution spherical level through cross-modal alignment fusion, spatial-temporal Transformer encoding, and structural prior-based gating fusion. Subsequently, clustering features (K-Means) are extracted based on the functional connectivity matrix. These features are then obtained by progressive temporal and channel compression and spatial pooling to obtain static features of the left and right hemispheres, which are then concatenated. Finally, these features are input into a clustering-aware classifier that integrates global features and cluster labels, and the output is the identification result of mood disorders (MDD / HC). This achieves accurate auxiliary diagnosis of mood disorders while maintaining the topological structure of the cerebral cortex.

[0063] A specific example: To verify the technical effectiveness of this invention, a multi-center resting-state functional magnetic resonance imaging (fMRI) dataset was used for experimental verification. The dataset covers two types of mood disorders: depressive disorder (MDD) and anxiety disorder. Samples were sourced from Xiangya Second Hospital of Central South University, Gansu Provincial People's Hospital, the Second Hospital of Lanzhou University, the Neuropsychiatric Phenome Consortium, and public databases, as shown in Table 1. Significant differences exist among the centers in terms of the number of subjects, disease type, and the length of the fMRI time points, effectively simulating the heterogeneity of image data under multi-center, multi-parameter acquisition conditions in real clinical applications.

[0064] Table 1. Composition of the experimental dataset In a binary classification task of depressive disorders (patient / healthy control), a five-fold cross-validation strategy was used, and the experiment was repeated three times. The classification performance is shown in Table 2. The results show that the method of the present invention can stably extract functional network discriminant features related to depressive disorders under multi-center and different time length conditions. The classification accuracy, precision, recall, F1 score and AUC all remain at a high level, demonstrating good stability and generalization ability.

[0065] Table 2 Results of the Depression Disorder Classification Experiment To further verify the applicability and transferability of the method of this invention across disease scenarios, fine-tuning was performed on a training model for depression disorder, and then transferred to an anxiety disorder classification task. During fine-tuning, the basic spatiotemporal representation learning module, temporal compression module, functional connectivity construction module, and most of the spherical convolutional backbone network were fixed; only the parameters of the function-structure cross-attention, cross-hemispheric interactive attention, temporal attention pooling, cluster assignment and cluster classification modules, and the final classification layer were updated. Experimental results are shown in Table 3, indicating that the method of this invention can still achieve ideal classification performance with only a few parameter adjustments, and the learned brain functional structural representations have good disease independence and transferability.

[0066] Table 3 Results of the Anxiety Disorder Classification Experiment The experimental results above show that the mood disorder recognition method based on multimodal brain image fusion provided by the present invention not only exhibits excellent discrimination performance in multicenter depressive disorder recognition tasks, but can also be efficiently extended to related mental illness scenarios such as anxiety disorders through lightweight fine-tuning. It has technical advantages of strong stability, good generalization ability and wide application range under complex multi-source brain image conditions.

[0067] The steps of the various methods described above are only for clarity. In practice, they can be combined into one step or some steps can be split into multiple steps. As long as they include the same logical relationship, they are all within the protection scope of this invention. Adding insignificant modifications or introducing insignificant designs to the algorithm or process, without changing the core design of the algorithm and process, are also within the protection scope of this invention.

[0068] Another embodiment of the present invention relates to a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the method embodiments described above.

[0069] That is, those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0070] Those skilled in the art will understand that the above embodiments are specific embodiments for implementing the present invention, and in practical applications, various changes can be made to them in form and detail without departing from the spirit and scope of the present invention.

Claims

1. A method for identifying mood disorders based on multimodal brain image fusion, characterized in that, The method includes: Structural magnetic resonance imaging (SMRI) and resting-state functional magnetic resonance imaging (fMRI) were acquired. The SMRI and resting-state fMRI images were then registered to the same standard spherical space to obtain spatially aligned spherical structural features and spherical functional time series. On a standard spherical space, spatial structural features of spherical structural features and spatiotemporal functional features of spherical functional time series are extracted; the spatial structural features are extended along the time dimension to align with the spatiotemporal functional features, and the aligned spatial structural features and spatiotemporal functional features are fused to obtain joint features; For the joint features, self-attention encoding is performed in the spherical spatial dimension and the temporal dimension respectively; the self-attention encoded features are split into functionally relevant components and structurally relevant components, and the structurally relevant components are used to modulate the functionally relevant components through a gating fusion mechanism to obtain modulated multimodal features; The modulated multimodal features are processed recursively from the first resolution level corresponding to the standard spherical space through multi-scale spherical space downsampling and mapped to the penultimate resolution spherical level. When recursively processing to each spherical level, feature extraction, alignment and fusion, self-attention encoding, and gated modulation operations are repeatedly performed on that spherical level. At the penultimate resolution spherical level, the multimodal features obtained from this spherical level are processed and fused in parallel, and the features obtained after parallel processing and fusion are compressed and mapped to the lowest resolution spherical level, while clustering embedding features are extracted. At the lowest resolution spherical level, temporal and spatial aggregation are performed on the features obtained from the compression mapping of the second-to-last resolution spherical level to obtain sample-level global features. Clustering embedding features are fused with sample-level global features to form cluster-aware global discriminative features, which are then classified using a cluster-aware classifier to output the emotional disorder identification results.

2. The method for identifying mood disorders based on multimodal brain image fusion as described in claim 1, characterized in that, In the penultimate resolution spherical layer, the multimodal features obtained from this spherical layer are processed and fused in parallel. The specific steps are as follows: Multimodal features are decomposed into functional components and structural components; The functional components and structural components are expanded in terms of channel dimension. The extended structural components are aligned with the extended functional components in the time dimension through a time broadcast operation, and then concatenated in the channel dimension to obtain the intermediate joint feature representation of this level.

3. The method for identifying mood disorders based on multimodal brain image fusion as described in claim 1, characterized in that, The specific steps for compressing and mapping the features obtained after parallel processing and fusion to the lowest resolution spherical layer are as follows: Joint compression of the intermediate joint feature representation is performed on both the time and channel dimensions; The jointly compressed features are mapped to the lowest resolution spherical layer using spherical spatial pooling.

4. The method for identifying mood disorders based on multimodal brain image fusion as described in claim 1, characterized in that, The specific steps for extracting clustering embedding features are as follows: Based on the functional components at this level, the functional correlation between different spherical vertices is calculated in the time dimension to construct a sample-level functional connection matrix; The functional connectivity matrix is ​​input into a multilayer perceptron network for feature projection and compression to obtain low-dimensional clustering embedding features.

5. The method for identifying mood disorders based on multimodal brain image fusion as described in claim 1, characterized in that, At the lowest resolution spherical layer, temporal and spatial aggregation are performed on the features obtained from the compression mapping of the second-to-last resolution spherical layer to obtain sample-level global features. The specific steps are as follows: For the features obtained from the penultimate resolution spherical layer compression mapping, extract the feature parts corresponding to the left and right hemispheres respectively; For the feature parts of the left hemisphere and the feature parts of the right hemisphere, a learnable temporal attention pooling mechanism is used to weight and aggregate the time dimension to eliminate the time dimension, so as to obtain the time aggregated features of the left hemisphere and the time aggregated features of the right hemisphere. The time aggregation features of the left hemisphere and the time aggregation features of the right hemisphere are concatenated to obtain the concatenated overall features; Calculate the global space average pooling feature and the global space max pooling feature of the overall features after concatenation; The global spatial average pooling feature and the global spatial max pooling feature are concatenated to form the sample-level global feature.

6. The method for identifying mood disorders based on multimodal brain image fusion as described in claim 1, characterized in that, The process of fusing clustering embedded features with sample-level global features to form cluster-aware global discriminative features, and then classifying them using a cluster-aware classifier to output the emotion disorder identification result, is detailed below: Clustering embedding features are concatenated with sample-level global features to form cluster-aware global discriminative features; The global discriminative features are input into a cluster-aware classifier; wherein, the cluster-aware classifier contains multiple sub-classifiers, and the outputs of the multiple sub-classifiers are weighted and summed by the cluster membership degree to obtain the final classification result.

7. The method for identifying mood disorders based on multimodal brain image fusion as described in claim 1, characterized in that, The specific steps for extracting the spatial structural features and spatiotemporal functional features of the spherical structure and the time series of spherical functions are as follows: For spherical structural features, perform convolution operations along the spatial dimensions of the sphere to extract spatial structural features; For spherical functional time series, perform a joint convolution operation of spatial and temporal dimensions to extract spatiotemporal functional features.

8. The method for identifying mood disorders based on multimodal brain image fusion as described in claim 1, characterized in that, The steps for extending the spatial structure features along the time dimension to align with the spatiotemporal functional features, and then fusing the aligned spatial structure features with the spatiotemporal functional features to obtain joint features are as follows: By using time broadcasting, spatial structural features are copied and extended along the time dimension, aligning spatial structural features with spatiotemporal functional features in the time dimension. By splicing the time-aligned spatial structural features and spatiotemporal functional features along the channel dimension, a joint feature is obtained.

9. The method for identifying mood disorders based on multimodal brain image fusion as described in claim 1, characterized in that, The joint features are self-attention encoded in both the spherical spatial dimension and the temporal dimension; the self-attention encoded features are then decomposed into function-related components and structure-related components, as detailed below: Self-attention encoding is performed in the time dimension: For each spherical vertex, the query, key and value vectors are constructed through linear mapping of its joint feature sequence at all time steps, and the time attention weights are calculated based on the scaling dot product attention mechanism. The value vectors are then weighted and aggregated to obtain the time-encoded features. Self-attention encoding is performed in the spherical space dimension: For each time step, the temporally encoded features of all spherical vertices are used to construct query, key, and value vectors through linear mapping, and spatial attention weights are calculated based on the scaling dot product attention mechanism. The value vectors are then weighted and aggregated to obtain the spatial-temporal self-attention encoded features. The features encoded by spatial-temporal self-attention are decomposed along their channel dimensions to obtain functionally relevant components and structurally relevant components.

10. The method for identifying mood disorders based on multimodal brain image fusion as described in claim 1, characterized in that, The method of using a gated fusion mechanism to modulate the function-related components with the structure-related components to obtain modulated multimodal features is as follows: The modulation factor is generated based on the structure-related components, including inputting the structure-related components into the Sigmoid activation function for processing. The functionally correlated components are multiplied element-wise with the modulation factor to modulate the functionally correlated components and obtain the modulated multimodal features.

Images