Multi-modal magnetic resonance image striatum segmentation method and system based on multi-scale hypergraph

Abstract

The invention discloses a multi-modal magnetic resonance image striatum segmentation method and system based on a multi-scale hypergraph, and belongs to the field of medical image processing and computer vision. According to the method, by combining MRI data of multiple modals and multi-scale hypergraph construction, the complex structure and functional characteristics of the striatum can be captured more comprehensively, so that the precision and robustness of a segmentation task are improved; according to the method, two different information transmission mechanisms of a hypergraph neural network and an external graph attention mechanism are utilized to process fused hypergraphs under different scales, so that information from different scales and modals is effectively integrated, comprehensive feature description of each corpus striatum voxel is provided, details of the corpus striatum can be described more accurately, and the accuracy of corpus striatum modeling is improved. Richer feature support is provided for subsequent clustering and segmentation tasks; unnecessary calculation is reduced by adopting the sparse representation hypergraph, and the data processing efficiency is further optimized by adopting the dimension reduction operation provided by the auto-encoder.

Description

Technical Field

[0001] The present invention belongs to the field of medical image processing and computer vision, and specifically relates to a multi-modal magnetic resonance image striatum segmentation method and system based on a multi-scale hypergraph. Background Art

[0002] As the most complex structure in the human body, the exploration of the physiological functions and pathological mechanisms of the brain has always been an important topic in scientific research. Over the past century, scientists have been working hard to reveal the behavior and function of the brain, and this ongoing exploration reflects a deep curiosity about the nature of human cognition. Among the many brain regions, the striatum has received special attention due to its functional complexity and importance in a variety of neurological diseases.

[0003] The striatum is a key component of the basal ganglia that works closely with the cerebral cortex to regulate a variety of functions and shows significant structural heterogeneity. It is generally believed that the striatum includes three major anatomical components: the caudate nucleus, which is crucial for motor control and cognitive learning processes; the putamen, which regulates movement and promotes learning between sensory and motor patterns; and the nucleus accumbens (NAcc), which regulates reward and motivation mechanisms. However, the striatum receives different topographical projections from the cerebral cortex, and the diversity of these projections requires us to reflect them through more detailed structural and functional divisions. In addition, the structure and function of the striatum are closely related to a variety of neurological and psychiatric diseases, including dopamine-related diseases such as Parkinson's disease and schizophrenia. Therefore, accurate segmentation and study of the striatum will not only help to understand its role in behavior and cognition, but also facilitate the study of striatal variability in clinical patient populations, which is a core goal in the fields of cognitive neuroscience and clinical neuroscience.

[0004] In the process of analyzing the complex network of the brain, neural connections, as the main channels for information transmission, constitute the basis of its complex structure and function. With the advancement of science and technology, especially the development of magnetic resonance imaging (MRI) technology, it has provided key tools for mapping the structure and function of these complex networks, opening a new chapter in the study of brain operations. The versatility of MRI technology makes it possible to comprehensively evaluate brain activity from different time and space dimensions. Among them, diffusion MRI (dMRI) is extremely sensitive to the structural integrity of white matter, providing key insights into understanding brain structural connections. The anatomical features obtained by dMRI are extremely important for studying individual differences in the striatum. In addition, functional MRI (fMRI) can capture instantaneous changes in brain activity. By measuring the changes in the ratio of oxygenated hemoglobin and deoxygenated hemoglobin, it can intuitively show the functional activity of various parts of the brain in specific tasks or thinking activities, thus providing a powerful perspective and method for studying cognitive functions and their neural basis.

[0005] Brain segmentation techniques derived from connectivity information extracted from MRI imaging have become crucial in revealing complex neuroanatomical data. These techniques make it possible to more effectively identify and classify different structural and functional regions within the brain. Numerous methods have been developed to better achieve connectivity-based brain region segmentation. These methods include connectivity-based localization (CBL) techniques, various clustering techniques, and methods using deep learning. Summary of the invention

[0006] In order to solve the problems existing in the prior art, the present invention proposes a multi-modal magnetic resonance imaging striatum segmentation method and system based on a multi-scale hypergraph, which makes full use of multiple modal MRIs to construct hypergraph data of multiple scales, and uses a hypergraph convolutional network to fuse the data of two modalities at the same scale, and then fuses the multi-modal hypergraph fused at multiple scales through an external graph attention mechanism to obtain the bimodal multi-scale features of each voxel of the striatum, and finally achieves segmentation through autoencoders and clustering processing. The present invention improves the accuracy and robustness of the striatum segmentation task, and effectively reduces the computational load and memory requirements of the model.

[0007] In one aspect, the present invention provides a multi-modal magnetic resonance imaging striatum segmentation method based on a multi-scale hypergraph, the method comprising the following steps:

[0008] MRI preprocessing: inputting MRI samples of multiple modalities collected by a magnetic resonance imaging device and performing data preprocessing on them to output MRI hypermaps of multiple modalities of the MRI samples; wherein the data preprocessing includes: extracting connectivity features of striatal voxels of MRI samples of each modality, and constructing MRI hypermaps of each modality based on the extracted connectivity features of striatal voxels;

[0009] MRI hypergraph fusion: The MRI hypergraphs of two modalities at different scales are fused to obtain dual-modal fusion hypergraphs of multiple scales. Then, the dual-modal fusion hypergraphs of multiple scales are fused into one hypergraph based on the external graph attention mechanism to obtain the dual-modal multi-scale features of each striatal voxel.

[0010] MRI clustering: An autoencoder is used to reduce the dimensionality of the bimodal multi-scale features of each striatal voxel, and then the segmentation results of the striatum are obtained based on clustering processing.

[0011] Furthermore, the input magnetic resonance imaging MRI sample includes two modalities, namely, the dMRI modality and the fMRI modality, and the corresponding striatal voxel connectivity feature extraction processing specifically includes:

[0012] For the dMRI modality, the HCP (Human Connectome Project) minimal preprocessing pipeline was used to preprocess the dMRI magnetic resonance imaging images, and then the obtained image data was used to extract the connectivity features of the striatal voxels:

[0013] Voxel-based probabilistic fiber tractography algorithms estimate fiber orientation and uncertainty;

[0014] The region of interest (ROI) of the striatum obtained by the brain neuroimaging data analysis tool is used as the voxel tracking seed point to track the ipsilateral cerebral cortex and generate the preliminary fiber bundle connection characteristics from each voxel of the striatum to the cortex;

[0015] Classify the whole brain fibers and create a ζ-dimensional binary label to characterize whether the voxel belongs to the fiber bundle k, and then downsample to the specified resolution to obtain the ζ-dimensional fiber bundle space, where the value of ζ is consistent with the classification number of the whole brain fibers;

[0016] The preliminary fiber bundle connection features of each voxel of the striatum to the cortex are converted into connection features in the fiber bundle space, and a multi-scale hypergraph of the dMRI modality is constructed based on the connection features in the ζ-dimensional fiber bundle space of each voxel;

[0017] For fMRI modality, the HCP minimal preprocessing pipeline was used to preprocess the fMRI magnetic resonance imaging images, and then the obtained image data was used to extract the connectivity features of striatal voxels:

[0018] A connectivity matrix was constructed based on the Pearson correlation coefficient between the time series of striatal voxels and the time series of other gray matter (GM) voxels. The connectivity matrix was subjected to Fisher Z transformation to obtain the fMRI connectivity feature matrix. A multi-scale hypergraph of the fMRI modality was then constructed based on the fMRI connectivity features of each voxel.

[0019] Furthermore, the multi-scale hypergraph is constructed as follows:

[0020] Define X = [x 1 ,...,x m ,...,x M ] T ∈R M×d Represented as a striatal object with a total of M voxels, where x m represents the feature sequence of the mth voxel, and d is the length of the feature sequence;

[0021] The feature sequence x of each voxel m is regarded as a response vector and estimated using a linear combination of the feature sequences of the other M-1 voxels. The estimated expression is:

[0022] x m =A mα m +τ m ,m=1,2,...,M

[0023] Among them, A m =[x 1 ,...,x m-1 ,0,x m+1 ,...x M ] represents the data matrix containing all feature sequences except the mth voxel, α m represents the weight vector that quantifies the influence of other voxels on the m-th voxel, τ m ∈R d Represents the noise term. Note that a zero element in the weight vector means that the corresponding voxel is not important in estimating the feature sequence.

[0024] Use sparse learning to optimize m Estimated objective function:

[0025]

[0026] Among them, λ>0 is the regularization parameter that controls the sparsity of the model. Different λ values ​​correspond to different sparsity solutions. The larger the λ value, the sparser the model, that is, α m The more elements in are zero;

[0027] For each striatum, a hypergraph network is constructed by sparsely representing each striatal voxel used as a node, with hyperedges e m Including the centroid voxel (i.e., the mth voxel) and other voxels (containing only their weight vector α m Different values ​​of λ determine the interaction relationships between voxels at different levels, that is, hypergraphs of different scales.

[0028] Furthermore, MRI hyperimage fusion specifically includes:

[0029] The structural data of the MRI hypergraph of the same scale of the two modalities are obtained, and the bimodal fusion features of each hypergraph node are obtained through the bimodal hypergraph fusion network and sent to the multi-scale hypergraph fusion network;

[0030] Among them, the bimodal hypergraph fusion network includes a hypergraph convolution layer, normalized batch processing and an activation function (such as RELU) in sequence. The hypergraph convolution layer simultaneously adopts two strategies to update node features during forward propagation: one is to first fuse the features of the nodes connected to it through hyperedges in the same modality, and then fuse the features of the node in the hypergraph of the other modality; the other is to first fuse the features of the node in the hypergraph of the other modality, and then fuse the features of the nodes connected to it through hyperedges in the same modality.

[0031] The multi-scale hypergraph fusion network uses an external graph attention mechanism to fuse the bimodal hypergraphs fused at different scales to obtain the bimodal multi-scale features of each node that are ultimately used for clustering.

[0032] Furthermore, the autoencoder is a stacked structure including a plurality of stacked unit modules, wherein the stacked unit modules sequentially include a Transformer-based graph convolutional layer and an activation function (such as a ReLU activation function).

[0033] Furthermore, the dual-modal hypergraph fusion network, multi-scale hypergraph fusion network and autoencoder of the hypergraph fusion processing are jointly trained, and the total loss used in the training is set as the reconstruction loss of the autoencoder Cluster-based average distance loss Deep Embedding Clustering and clustering-based distance margin loss The weighted sum of . For the reconstruction loss, the output of the autoencoder is reconstructed through the decoder, and the hidden layer features of the autoencoder output are mapped back to the input dimension of the autoencoder, and then the loss between the reconstructed features and the input features of the autoencoder is calculated.

[0034] On the other hand, the present invention also provides a multi-modal magnetic resonance imaging striatum segmentation system based on a multi-scale hypergraph, which comprises an MRI preprocessing module, an MRI hypergraph fusion module and an MRI clustering module connected in sequence;

[0035] The input of the MRI preprocessing module is a magnetic resonance image MRI of multiple modalities collected by a magnetic resonance imaging device, that is, the input of the MRI preprocessing module is an MRI sample, and after data preprocessing of the input MRI sample, MRI hypergraph structure data of multiple modalities of the MRI sample is output;

[0036] The MRI hypergraph fusion module fuses the two-modal MRI hypergraphs output by the MRI preprocessing module at different scales, and then fuses the dual-modal hypergraphs at different scales into one hypergraph according to the external graph attention mechanism to obtain the dual-modal multi-scale features of each striatal voxel;

[0037] The MRI hypergraph clustering module includes an autoencoder and a clustering processing submodule, wherein the autoencoder is used to reduce the dimension of the bimodal multi-scale features of each striatal voxel and then send them to the clustering processing submodule for clustering processing (such as using the KMeans clustering method) to output the segmentation results of the striatum.

[0038] Further, the MRI hypergraph fusion module includes a dual-modality hypergraph fusion submodule and a multi-scale hypergraph fusion submodule connected in sequence;

[0039] Among them, the bimodal hypergraph fusion submodule is used to obtain the MRI hypergraph structure data of the same scale of two modalities, and obtain the bimodal fusion features of each hypergraph node through the hypergraph convolutional network;

[0040] The bimodal hypergraph fusion submodule includes a hypergraph convolution layer, a normalized batch processing, and an activation function in sequence; and the node update strategy adopted by the hypergraph convolution layer of the bimodal hypergraph fusion submodule during forward propagation is simultaneously the following two methods:

[0041] 1) First, the features of the nodes connected to it through hyperedges in the same modality are integrated, and then the features of the node in the hypergraph of the other modality are integrated;

[0042] 2) First, fuse the features of the node in the hypergraph of another modality, and then fuse the features of the nodes connected to it through hyperedges in the same modality;

[0043] The multi-scale graph fusion submodule uses an external graph attention mechanism to fuse the bimodal hypergraphs fused at different scales to obtain the bimodal multi-scale features of each node that are ultimately used for clustering.

[0044] Furthermore, the autoencoder of the MRI hypergraph clustering module is configured as a stacked structure of stacked units of Transformer-based graph convolutional layers and activation functions.

[0045] Furthermore, the segmentation system also includes a model training module, which is configured to perform model training on the MRI hypergraph fusion module and the MRI hypergraph clustering module according to the joint loss function and using a deep learning algorithm; and the total loss used during training is set to the reconstruction loss of the autoencoder Cluster-based average distance loss Deep Embedding Clustering and clustering-based distance margin loss The weights assigned to multiple losses are hyperparameters that balance the impact of each loss component. The setting of hyperparameters is to ensure that each loss function value remains at the same order of magnitude, thereby promoting stable convergence of the model.

[0046] The technical solution provided by the present invention brings at least the following beneficial effects:

[0047] By combining MRI data of multiple modalities and multi-scale hypergraph construction, the present invention can more comprehensively capture the complex structure and functional characteristics of the striatum, thereby improving the accuracy and robustness of the segmentation task; at the same time, by using two different information transmission mechanisms of the hypergraph neural network and using the external graph attention mechanism to process the fused hypergraphs at different scales, the system can effectively integrate information from different scales and modalities, and provide a comprehensive feature description of each striatal voxel, which helps to more accurately characterize the details of the striatum and provide richer feature support for subsequent clustering and segmentation tasks; finally, the present invention adopts sparse learning and autoencoder technology to effectively reduce the computational load and memory requirements of the model. The sparsely represented hypergraph reduces unnecessary calculations, and the dimensionality reduction operation provided by the autoencoder further optimizes the efficiency of data processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0049] Figure 1 It is a main structural schematic diagram of a multi-modal magnetic resonance imaging striatum segmentation system based on a multi-scale hypergraph provided by an embodiment of the present invention;

[0050] Figure 2 It is a basic framework schematic diagram of the MRI hypergraph fusion module in an embodiment of the present invention;

[0051] Figure 3 is a schematic structural diagram of a dual-modal hypergraph fusion submodule in an embodiment of the present invention;

[0052] Figure 4 is a schematic diagram of the structure of the external graph attention mechanism in an embodiment of the present invention;

[0053] Figure 5 Schematic diagram of the structure of the MRI clustering module in an embodiment of the present invention. DETAILED DESCRIPTION

[0054] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0055] On the one hand, an embodiment of the present invention provides a multi-modal magnetic resonance image striatum segmentation method based on a multi-scale hypergraph (i.e., a multi-modal magnetic resonance image assisted classification method based on a multi-branch fusion graph network), which includes: MRI preprocessing: inputting magnetic resonance image MRI samples of multiple modalities collected by a magnetic resonance imaging device and performing data preprocessing on them to output MRI hypergraphs of multiple modalities of the MRI samples; wherein the data preprocessing includes: extracting connectivity features of striatal voxels of MRI samples of each modality, and constructing MRI hypergraphs of each modality based on the extracted connectivity features of striatal voxels, preferably, the hypergraph can be constructed based on a sparse representation method; MRI hypergraph fusion: fusing MRI hypergraphs of two modalities at different scales to obtain dual-modal fusion hypergraphs of multiple scales, and then fusing the dual-modal fusion hypergraphs of multiple scales into a hypergraph according to an external graph attention mechanism to obtain dual-modal multi-scale features of each striatal voxel; MRI clustering: using an autoencoder to reduce the dimension of the dual-modal multi-scale features of each striatal voxel, and then obtaining the segmentation result of the striatum based on clustering processing. The method of the embodiment of the present invention can more comprehensively capture the complex structure and functional characteristics of the striatum by combining MRI data of multiple modalities and multi-scale hypergraph construction, thereby improving the accuracy and robustness of the segmentation task; at the same time, by using two different information transmission mechanisms of the hypergraph neural network and using the external graph attention mechanism to process the fused hypergraphs at different scales, the method of the embodiment of the present invention can effectively integrate information from different scales and modalities, and provide a comprehensive feature description of each striatal voxel, which helps to more accurately characterize the details of the striatum and provide richer feature support for subsequent clustering and segmentation tasks; finally, the method of the embodiment of the present invention adopts sparse learning and autoencoder technology to effectively reduce the computational load and memory requirements of the model. The sparsely represented hypergraph reduces unnecessary calculations, and the dimensionality reduction operation provided by the autoencoder further optimizes the efficiency of data processing.

[0056] On the other hand, an embodiment of the present invention further provides a multi-modal magnetic resonance imaging striatum segmentation system based on a multi-scale hypergraph, such as Figure 1 As shown, the classification system includes an MRI preprocessing module, an MRI hypergraph fusion module and an MRI clustering module; wherein the MRI preprocessing module is configured to obtain magnetic resonance images MRI of multiple modalities acquired by a magnetic resonance imaging device, and output MRI hypergraph structural data of multiple modalities of the MRI sample; the MRI hypergraph fusion module is configured to fuse the MRI hypergraphs of the same scale of the two modalities output by the MRI preprocessing module, and then fuse the dual-modal hypergraphs fused at different scales into a hypergraph according to an external graph attention mechanism to obtain the final features of each striatal voxel; the MRI clustering module is configured to use an autoencoder to reduce the dimensionality according to the dual-modal multi-scale features of each striatal voxel and then perform KMeans clustering to obtain the segmentation result.

[0057] In an embodiment of the present invention, the MRI preprocessing module is further configured to perform the following operations:

[0058] This embodiment uses FreeSurfer software to extract the regions of interest (ROI) of the left and right hemisphere striatum, the left and right hemisphere cerebral cortex, and the left and right hemisphere white matter from T1-weighted (T1w) data. Specifically, after using FreeSurfer to segment the individual brain, the corresponding mask is extracted using the corresponding label in FreeSurferColorLUT. In order to balance computational efficiency and spatial accuracy, the ROI mask is downsampled to 3mm resolution. In the MRI data modality selection, diffusion magnetic resonance imaging (d-MRI) and functional magnetic resonance imaging (fMRI) are used. Both modal data need to be registered with T1w data to accurately extract the features of the striatum.

[0059] In this embodiment, the connectivity features of striatal voxels are extracted from dMRI magnetic resonance imaging, and the structural connectivity index of each vertex (striatal voxel) is calculated, including: (1) probabilistic fiber bundle tracking: a voxel-based probabilistic fiber bundle imaging algorithm is applied, and the fiber direction and uncertainty are estimated using FSL's BEDPOSTX_GPU algorithm. Since the T1w and d-MRI images have been registered, the striatal ROI area extracted from the T1w data can be directly used as the seed point for fiber tracking. The PROBTRACKX2_GPU algorithm is used to track 5000 streamlines for each vertex, with the end point being the ipsilateral cerebral cortex. The ROI extraction method for the cortex and white matter regions is the same as that for the striatum. According to the established tracking parameters (curvature threshold = 0.2, fiber orientation threshold = 0.01), the resulting brain fiber bundle map is obtained. (N vs is the number of striatal voxels of subject s, N bs is the number of brain voxels of subject s), and downsampled it to a 3 mm equal-cubic resolution space to obtain the preliminary connection characteristics of each striatal voxel to all cortical voxels. (2) Whole-brain fiber classification: Using the pre-trained deep learning model of the TractSeg tool (v2.3), the whole-brain fibers were classified into 72 categories and binary labels were created for the whole-brain voxels. The resulting fiber bundle mask is also downsampled to 3 mm isodirectional resolution. (3) Connection feature generation: The connection feature of each striatal voxel to the cortex is converted into a connection feature in the fiber bundle space to generate The calculation formula is as follows:

[0060]

[0061] in, Connection feature matrix Each element f in i,t N represents the probability that any fiber bundle t exists in the fiber bundle imaging map of vertex i. bs is the number of brain voxels of subject s, This is the resulting diagram of the brain fiber bundle starting from the striatum.

[0062] In this embodiment, the connectivity features of striatal voxels are extracted from fMRI magnetic resonance imaging, and the functional connectivity index of each vertex (striatal voxel) is calculated, including: (1) ROI time series extraction: the fMRI time series of each voxel of the striatum and all gray matter voxels outside the striatum are extracted using the ROI mask. (2) Correlation matrix calculation: the correlation matrix of the voxels in the striatum and the gray matter voxels is calculated using the Pearson coefficient, and the Fisher Z transformation is performed.

[0063] In this embodiment, the multi-scale hypergraph is constructed in the following manner:

[0064] X=[x 1 ,...,x m ,...,x M ] T ∈R M×d Represented as a striatal object with a total of M voxels, where x m represents the feature sequence of the mth voxel, and d is the length of the feature sequence. Then, the feature sequence of each voxel (i.e., x m ) can be viewed as a response vector and can be estimated using a linear combination of the feature sequences of the other M-1 voxels as follows:

[0065] x m =A m α m +τ m ,m=1,2,...,M

[0066] Among them A m =[x 1 ,...,x m-1 ,0,x m+1 ,...x M ] represents the data matrix containing all feature sequences except the mth voxel, α m represents the weight vector that quantifies the influence of other voxels on the m-th voxel, τ m ∈R d Represents the noise term. Note that a zero element in the weight vector means that the corresponding voxel is not important in estimating the feature sequence.

[0067] Use sparse learning to optimize the following objective function:

[0068]

[0069] Where λ>0 is the regularization parameter that controls the sparsity of the model. Different λ values ​​correspond to different sparsity solutions. The larger the λ value, the sparser the model, that is, α m The more elements in are zero.

[0070] For each striatum, a hypernetwork is constructed by sparsely representing each striatal voxel used as a node, with hyperedges e m Including the centroid voxel (i.e., the mth voxel) and other voxels (containing only their weight vector α m Different values ​​of λ determine the interaction relationships between voxels at different levels, that is, hypergraphs of different scales.

[0071] In a preferred technical solution provided in this embodiment, Figure 2 As shown, the MRI hypergraph fusion module of this embodiment includes a bimodal hypergraph fusion submodule and a multiscale hypergraph fusion submodule connected in sequence. Among them, the bimodal hypergraph fusion submodule fuses the same-scale MRI hypergraphs of the two modalities output by the MRI preprocessing module, and the multiscale hypergraph fusion submodule fuses the bimodal hypergraphs fused at different scales into one hypergraph according to the external graph attention mechanism. The bimodal hypergraph fusion submodule includes a hypergraph convolution layer, a normalized batch process, and a ReLU activation function in sequence.

[0072] The hypergraph convolutional network of the bimodal hypergraph fusion submodule is as follows:

[0073] set up is a hypergraph with N vertices and M hyperedges. Each hyperedge ∈∈E is assigned a positive weight W ∈∈ , all weights are stored in a diagonal matrix In. Supermap Through an association matrix To represent. When the hyperedge ∈∈E and the vertex v i ∈V, H i∈ = 1, otherwise 0. Then, the degree of the vertex is defined as:

[0074]

[0075] The degree of a hyperedge is defined as:

[0076]

[0077] in, and are all diagonal matrices.

[0078] The main obstacle in defining the convolution operation in a hypergraph is to measure the transition probability between two vertices, in this way, the embedding (or features) of each vertex can be propagated in the graph neural network. To achieve this, two assumptions are held: 1) more propagation should be done between vertices connected by common hyperedges, and 2) hyperedges with larger weights should be trusted more in this propagation. Then, a hypergraph convolution step is defined as:

[0079]

[0080] in, is the embedding of the ith vertex in layer l, and σ(·) is a nonlinear activation function such as LeakyReLU and ReLU. is the weight matrix between the (l)th layer and the (l+1)th layer. The above equation can be written in matrix form as:

[0081] X (l+1) =σ(HWH T X (l) P)

[0082] in, and are the inputs of the (l)th layer and the (l+1)th layer respectively.

[0083] However, HWH T The spectral radius of X is not restricted, which means that (l) The scale of may change. When optimizing neural networks, stacking multiple hypergraph convolutional layers may lead to an increased risk of numerical instability and gradient explosion / vanishing. Therefore, proper normalization is required. Therefore, the hypergraph convolutional network is finally defined as:

[0084] X (l+1) =σ(D -1 / 2 HWB -1 H T D -1 / 2 X (l) P)

[0085] In one embodiment, Figure 3 As shown, in the forward propagation process of the dual-modal hypergraph fusion submodule of the present invention, this embodiment adopts two strategies to fuse node features.

[0086] Strategy 1 is to first perform intra-modal fusion. For each modality’s hypergraph, first aggregate the node features connected by hyperedges. Use the above hypergraph convolutional network formula for processing:

[0087]

[0088] Among them, X (1) and X(2) They represent the results after updating the node features of the two modes respectively.

[0089] Then, cross-modal fusion is performed, that is, the updated features of the two modalities are fused, which can be achieved through simple weighted sum, splicing or other complex nonlinear fusion mechanisms:

[0090] X′=f(X (1) ,X (2) )

[0091] Among them, f() is a fusion function, which can be MLP, attention mechanism, etc.

[0092] The second strategy is to perform cross-modal fusion first, that is, first fuse the original features of the two modalities:

[0093] X inter =f(X( 1 ),X( 2) )

[0094] Among them, f() can be the same as the fusion function in strategy one.

[0095] Then perform intra-modal fusion, that is, use the fused feature X inter Propagate in the hypergraph of each mode:

[0096] X′=D -1 HWB -1 H T X inter

[0097] Finally, the node features obtained by the two strategies are spliced ​​together to complete a node update.

[0098] In one embodiment, the multi-scale hypergraph fusion submodule fuses the bimodal hypergraphs fused at different scales into one hypergraph according to an external graph attention mechanism to obtain the bimodal multi-scale features of each striatal voxel.

[0099] Among them, the external graph attention mechanism is as follows Figure 4 As shown, it computes the attention between the node features of the input graph and the external unit using the following formula:

[0100]

[0101] in, is a learnable parameter independent of the input graph and can be viewed as an external unit with S nodes, providing a shared memory for all input graphs. SelfRepresents the similarity between the input graph nodes, while in the external graph attention mechanism (Graph External Attention, GEA), A GE Represents the similarity between the input graph node and the external unit. Considering the sensitivity of the attention matrix to the input feature scale, this embodiment uses A GE The double normalization technique is applied on . This double normalization process normalizes the columns and rows separately and can be expressed as:

[0102]

[0103] in, represents the original attention matrix output, represents the row-normalized attention matrix, α i,j Represents the attention matrix after column normalization.

[0104] In practical applications, in order to improve the capacity of the network, two different external units are used to process key information and value information. In addition, in order to utilize the edge information in the input graph, an additional external unit is used to process the edge features, and a shared unit is used to store the connection between the edge and the node:

[0105]

[0106] in It is a shared unit used to store connections between edges and nodes; is the external unit for the node.

[0107] In one embodiment, Figure 5 As shown in the figure, the MRI clustering module uses an autoencoder to reduce the dimension of each striatal voxel based on the bimodal multi-scale features and then performs KMeans clustering on the reduced voxel features to obtain the segmentation result.

[0108] Among them, the autoencoder is defined as follows:

[0109] Consider a single voxel in the input individual striatum map data G = (V, E) Multi-head attention graph convolution encodes a single voxel into a low-dimensional representation through non-linear mapping:

[0110]

[0111] in, is x i A lower dimensional representation in the hidden layer. σ() represents the activation function (e.g. ReLU) applied after the linear transformation. ij represents the weight of the edge between node i and node j.

[0112] The attention coefficient is calculated by the dot product attention mechanism:

[0113]

[0114] For the multi-head self-attention mechanism, each head k independently calculates the attention coefficient

[0115]

[0116] in, and is the learnable weight matrix of the kth head, d k is the embedding dimension of each head (usually where d is the original embedding dimension).

[0117] Each head k produces an output The outputs of all heads are concatenated and then used with another weight matrix W O Project to the final output space:

[0118]

[0119] Among them, || represents splicing.

[0120] Therefore, the update formula of the 2-head attention graph convolution layer in this embodiment is:

[0121]

[0122] The multi-head self-attention mechanism enables the model to focus on the input data from multiple perspectives and capture rich information and features. The decoder part maps the hidden layer features back to the input dimension to obtain the reconstructed input Specifically, the formula of the decoder part is as follows:

[0123]

[0124] in, is the output of reconstruction, is the attention coefficient of the decoder, W 8 is the learnable weight matrix of the kth head.

[0125] Autoencoders usually use reconstructions of all nodes on the computation graph With the original input x i The reconstruction loss between is used for model training. The reconstruction loss is usually the mean square error (MSE):

[0126]

[0127] In order to improve the feature representation ability of AE, SAE is obtained by stacking multiple autoencoders.

[0128] In one embodiment, the K-means algorithm is applied to the features output by the autoencoder for preliminary clustering, and then a joint loss mechanism is introduced to fine-tune the parameters of the autoencoder and the hypergraph fusion module according to the clustering results to optimize the clustering effect. The joint loss mechanism includes four loss functions: autoencoder reconstruction loss, average distance loss, deep embedding clustering (DEC) loss, and distance boundary loss. These loss functions are designed to improve the accuracy and robustness of clustering while maintaining the sensitivity of the model to the structural and functional characteristics of the striatum.

[0129] Average Distance Loss measures the average distance of a voxel to its cluster center, promoting compact and consistent clustering:

[0130]

[0131] Among them, z i represents the low-dimensional latent space representation of voxel i, Represents the center point of the cluster to which voxel i belongs. By minimizing the distance between the voxel and its cluster center, this loss function ensures the compactness and consistency within the cluster, thereby improving the robustness and interpretability of the clustering.

[0132] The Deep Embedded Clustering (DEC) loss reduces the variance within clusters and forms clear cluster boundaries by minimizing the Kullback-Leibler (KL) divergence between the soft cluster assignment Q and the target distribution P:

[0133]

[0134] Among them, Q ij is the probability that voxel i belongs to cluster j, P ij is the target distribution. The target distribution is usually derived from Q by emphasizing higher confidence assignments, making the cluster assignment clearer. The core idea of ​​DEC loss is to ensure the stability and clarity of the cluster structure by regularizing the cluster assignment. However, since KL divergence is unbounded and asymmetric, the improved Jensen-Shannon (JS) divergence is used instead of KL divergence to measure q i and p i The similarities between are as follows:

[0135]

[0136] That is, in this embodiment, As calculated

[0137] The distance boundary loss ensures clear separation between cluster centers and prevents cluster overlap:

[0138]

[0139] Among them, μ i and μ j are the center points of cluster i and cluster j respectively, and m is the distance boundary hyperparameter. By applying the ReLU function, this loss function will only contribute to the loss when the minimum distance between different cluster centers is less than the boundary m. This mechanism ensures the formation of a clear and well-separated cluster structure by penalizing the situation where the cluster centers are too close.

[0140] The overall loss function during fine-tuning is the weighted sum of the individual sub-losses:

[0141]

[0142] where λ recon ,λ avgdist ,λ DEC and λ disbound are the weights assigned to each loss term, which are hyperparameters that balance the impact of each loss component. The setting of hyperparameters is to ensure that each loss function value remains at the same order of magnitude, thereby promoting stable convergence of the model.

[0143] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

[0144] The above are only some embodiments of the present invention. For those skilled in the art, several modifications and improvements can be made without departing from the creative concept of the present invention, which all belong to the protection scope of the present invention.

Claims

1. A multi-modal magnetic resonance imaging striatum segmentation method based on a multi-scale hypergraph, characterized in that: The following steps are involved: MRI preprocessing: inputting MRI samples of multiple modalities collected by a magnetic resonance imaging device and performing data preprocessing on them to output MRI hypermaps of multiple modalities of the MRI samples; wherein the data preprocessing includes: extracting connectivity features of striatal voxels of MRI samples of each modality, and constructing MRI hypermaps of each modality based on the extracted connectivity features of striatal voxels; MRI hypergraph fusion: The MRI hypergraphs of two modalities at different scales are fused to obtain dual-modal fusion hypergraphs of multiple scales. Then, the dual-modal fusion hypergraphs of multiple scales are fused into one hypergraph based on the external graph attention mechanism to obtain the dual-modal multi-scale features of each striatal voxel. MRI clustering: An autoencoder is used to reduce the dimensionality of the bimodal multi-scale features of each striatal voxel, and then the segmentation results of the striatum are obtained based on clustering processing.

2. The method according to claim 1, characterized in that The input magnetic resonance imaging (MRI) samples include two modalities, namely, dMRI modality and fMRI modality. The corresponding striatal voxel connectivity feature extraction processing specifically includes: For the dMRI modality, the Human Connectome Project HCP minimal preprocessing pipeline was used to preprocess the dMRI magnetic resonance imaging images, and then the obtained image data was used to extract the connectivity features of the striatal voxels: Voxel-based probabilistic fiber tractography algorithms estimate fiber orientation and uncertainty; Based on the region of interest of the striatum obtained by the brain neuroimaging data analysis tool, the region is used as the voxel tracking seed point to track the ipsilateral cerebral cortex, generating the preliminary fiber bundle connection characteristics from each voxel of the striatum to the cortex; Classify the whole brain fibers and create a ζ-dimensional binary label to characterize whether the voxel belongs to the fiber bundle k, and then downsample to the specified resolution to obtain the ζ-dimensional fiber bundle space, where the value of ζ is consistent with the classification number of the whole brain fibers; The preliminary fiber bundle connection features of each voxel of the striatum to the cortex are converted into connection features in the fiber bundle space, and a multi-scale hypergraph of the dMRI modality is constructed based on the connection features in the ζ-dimensional fiber bundle space of each voxel; For fMRI modality, the HCP minimal preprocessing pipeline was used to preprocess the fMRI magnetic resonance imaging images, and then the obtained image data was used to extract the connectivity features of striatal voxels: A connectivity matrix was constructed based on the Pearson correlation coefficient between the time series of striatal voxels and the time series of other gray matter voxels. The connectivity matrix was subjected to Fisher Z transformation to obtain the fMRI connectivity feature matrix. A multi-scale hypergraph of the fMRI modality was then constructed based on the fMRI connectivity features of each voxel.

3. The method according to claim 1, characterized in that The construction method of multi-scale hypergraph is as follows: Define X = [x1,...,x m ,...,x M ] T ∈R M×d Represented as a striatal object with a total of M voxels, where x m represents the feature sequence of the mth voxel, and d is the length of the feature sequence; The feature sequence x of each voxel m is regarded as a response vector and estimated using a linear combination of the feature sequences of the other M-1 voxels. The estimated expression is: x m =A m a m +t m ,m=1,2,...,M Among them, A m =[x1,...,x m-1 ,0,x m+1 ,...x M ] represents the data matrix containing all feature sequences except the mth voxel, α m represents the weight vector that quantifies the influence of other voxels on the m-th voxel, τ m ∈R d represents the noise term; Use sparse learning to optimize m Estimated objective function: Among them, λ>0 is the regularization parameter that controls the sparsity of the model; For each striatum, a hypergraph network is constructed by sparsely representing each striatal voxel used as a node, with hyperedges e m Including the centroid voxel represented by the m-th voxel and other voxels whose voxel weight vectors are non-zero.

4. The method according to claim 1, characterized in that MRI hyperimage fusion specifically includes: The structural data of the MRI hypergraph of the same scale of the two modalities are obtained, and the bimodal fusion features of each hypergraph node are obtained through the bimodal hypergraph fusion network and sent to the multi-scale hypergraph fusion network; The bimodal hypergraph fusion network includes a hypergraph convolution layer, a normalized batch processing, and an activation function in sequence. The node update strategy adopted by the hypergraph convolution layer during forward propagation is any one of the following two methods: 1) First, the features of the nodes connected to it through hyperedges in the same modality are integrated, and then the features of the node in the hypergraph of the other modality are integrated; 2) First, fuse the features of the node in the hypergraph of another modality, and then fuse the features of the nodes connected to it through hyperedges in the same modality; The multi-scale hypergraph fusion network uses an external graph attention mechanism to fuse the bimodal hypergraphs fused at different scales to obtain the bimodal multi-scale features of each node that are ultimately used for clustering.

5. The method according to claim 1, characterized in that The autoencoder is a stacked structure including several stacked unit modules, wherein the stacked unit modules sequentially include Transformer-based graph convolutional layers and activation functions.

6. A multi-modal magnetic resonance imaging striatum segmentation system based on a multi-scale hypergraph, characterized in that: It includes an MRI preprocessing module, an MRI hypergraph fusion module and an MRI clustering module which are connected in sequence; The input of the MRI preprocessing module is the MRI samples of multiple modalities collected by the magnetic resonance imaging device. After data preprocessing of the input MRI samples, the MRI hypergraph structure data of multiple modalities of the MRI samples is output; The MRI hypergraph fusion module fuses the two-modal MRI hypergraphs output by the MRI preprocessing module at different scales, and then fuses the dual-modal hypergraphs at different scales into one hypergraph according to the external graph attention mechanism to obtain the dual-modal multi-scale features of each striatal voxel; The MRI hypergraph clustering module includes an autoencoder and a clustering processing submodule, wherein the autoencoder is used to reduce the dimension of the bimodal multi-scale features of each striatal voxel and then send them to the clustering processing submodule for clustering processing to output the segmentation result of the striatum.

7. The system according to claim 6, characterized in that The MRI hypergraph fusion module includes a dual-modality hypergraph fusion submodule and a multi-scale hypergraph fusion submodule connected in sequence; Among them, the bimodal hypergraph fusion submodule is used to obtain the MRI hypergraph structure data of the same scale of two modalities, and obtain the bimodal fusion features of each hypergraph node through the hypergraph convolutional network; The bimodal hypergraph fusion submodule includes a hypergraph convolution layer, a normalized batch processing, and an activation function in sequence; and the hypergraph convolution layer of the bimodal hypergraph fusion submodule simultaneously adopts the following two node update strategies during forward propagation: 1) First, the features of the nodes connected to it through hyperedges in the same modality are integrated, and then the features of the node in the hypergraph of the other modality are integrated; 2) First, fuse the features of the node in the hypergraph of another modality, and then fuse the features of the nodes connected to it through hyperedges in the same modality; The multi-scale graph fusion submodule uses an external graph attention mechanism to fuse the bimodal hypergraphs fused at different scales to obtain the bimodal multi-scale features of each node that are ultimately used for clustering.

8. The system according to claim 6, characterized in that The autoencoder of the MRI hypergraph clustering module is configured as a stacked structure of stacked units of Transformer-based graph convolutional layers and activation functions.

9. The system according to claim 6, characterized in that The segmentation system also includes a model training module, wherein the model training module is configured to perform model training on the MRI hypergraph fusion module and the MRI hypergraph clustering module according to a joint loss function and using a deep learning algorithm; and the total loss used during training is set to the reconstruction loss of the autoencoder Cluster-based average distance loss Deep Embedding Clustering and clustering-based distance margin loss The weighted sum of .

10. The system according to claim 9, characterized in that The weighted terms of the total loss used during training are specifically set as: The decoder is used to reconstruct the output of the autoencoder, map the hidden layer features of the autoencoder output back to the input dimension of the autoencoder, and then use the mean square error to calculate the reconstruction loss. Average distance loss It is used to measure the average distance of a voxel to its cluster center; Deep Embedding Clustering By minimizing the Kullback-Leibler divergence between the soft clustering distribution Q and the target distribution P; Distance Boundary Loss Among them, μ i and μ j are the center points of cluster i and cluster j respectively, and m is the distance boundary hyperparameter, the preset value.

Images