Neural typing method based on multi-modal brain map and semi-supervised deep clustering

Abstract

The invention belongs to the technical field of video quality evaluation, and particularly relates to a neural typing method based on a multi-modal brain map and semi-supervised deep clustering. Comprising the following steps: constructing a multi-modal brain network according to multi-modal neural image data; the brain network of each mode comprises a normal brain network and an abnormal brain network; performing pre-training on the MBVAE model by adopting a multi-modal brain network to obtain a pre-trained MBVAE model; performing fine tuning training on the pre-trained MBVAE model in combination with a clustering module to obtain a trained deep embedded clustering model; obtaining multi-modal neural image data of a user, constructing a multi-modal brain network, and inputting the multi-modal brain network into the trained deep embedded clustering model for processing to obtain a neural typing result; according to the method, the accuracy and reliability of ASD neural typing results are improved, the problems of gender imbalance and multi-site heterogeneity existing in ASD data are effectively solved, and the robustness and generalization ability of the model are enhanced.

Description

Technical Field

[0001] This invention belongs to the field of video quality assessment technology, specifically relating to a neural typing method based on multimodal brain mapping and semi-supervised deep clustering. Background Technology

[0002] Autism Spectrum Disorder (ASD) is a highly hereditary and heterogeneous neurodevelopmental disorder, and accurate neurotyping is crucial for early diagnosis and targeted intervention. Current technologies primarily utilize neuroimaging techniques, such as structural magnetic resonance imaging (sMRI) and resting-state functional magnetic resonance imaging (rs-fMRI), to analyze the brain structure and function of ASD patients in order to identify their neurological subtypes. However, current neurotyping methods still have the following major limitations:

[0003] Limitations of feature extraction: Existing studies often use brain region connectivity values ​​as input features, focusing only on the strength of brain connections while neglecting the topological structure of brain networks and the fusion of multimodal information. This results in insufficient discriminative power of the extracted features, failing to fully characterize the complex heterogeneity of the ASD brain.

[0004] The challenge of data complexity: ASD brain network data exhibits complex characteristics such as multimodal heterogeneity, multisite heterogeneity, and extreme gender imbalance (male:female approximately 3:1). Existing methods have failed to effectively address these challenges, limiting the accuracy and reliability of the typing results.

[0005] Limitations of fractal models: Traditional unsupervised clustering methods (such as k-means and hierarchical clustering) are easily affected by confounding factors (such as age and site). Furthermore, two-stage methods that perform representation learning before clustering often lead to suboptimal solutions because the representation learning is not optimized for the clustering task. While semi-supervised deep clustering methods exist, most are based on generative adversarial networks (GANs), which suffer from difficulties in model convergence and pattern collapse.

[0006] Therefore, there is an urgent need for a new method for ASD neuromorphology that can fully utilize multimodal brain network information, effectively cope with data complexity, and achieve end-to-end optimized learning. Summary of the Invention

[0007] To address the shortcomings of existing technologies, this invention proposes a neuromorphic classification method based on multimodal brain mapping and semi-supervised deep clustering. The method includes: acquiring the user's multimodal neuroimaging data and constructing a multimodal brain network; inputting the multimodal brain network into a trained deep embedding clustering model for processing to obtain the neuromorphic classification result; the deep embedding clustering model includes an MBVAE model and a clustering module.

[0008] The training process of a deep embedding clustering model includes:

[0009] S1: Acquire multimodal neuroimaging data for training and construct a multimodal brain network based on the multimodal neuroimaging data; the brain network for each modality includes a normal brain network and an abnormal brain network;

[0010] S2: The MBVAE model is pre-trained using a multimodal brain network to obtain a pre-trained MBVAE model;

[0011] S3: Fine-tune the pre-trained MBVAE model together with the clustering module to obtain a trained deep embedding clustering model.

[0012] Preferably, the MBVAE model has an encoder-decoder structure, with the encoder consisting of three sub-encoders. The pre-training process of the MBVAE model includes:

[0013] The three sub-encoders encode the brain network for each modality, resulting in three types of coding features for each modality: normal coding features, shared coding features, and abnormal coding features.

[0014] An expert product module is used to fuse the corresponding coding features across all modalities, resulting in three comprehensive coding features. These three comprehensive coding features are then sampled to obtain three joint latent embeddings, namely, normal joint latent embeddings. Shared joint implicit embedding Combined with anomalies and hidden embedding ;

[0015] Each mode is assigned a decoder, with the input being... At that time, the output reconstructs the normal brain network; the input is... When the time comes, output the reconstructed abnormal network;

[0016] The total loss of the MBVAE model is calculated by reconstructing the normal brain network, reconstructing the abnormal network, and using three joint implicit embeddings. The model parameters are then adjusted based on the total loss of the MBVAE model to obtain the pre-trained MBVAE model.

[0017] Furthermore, each sub-encoder is a graph convolutional network or a graph attention network; the decoder consists of fully connected layers and inner product layers.

[0018] Furthermore, the formula for calculating the total loss of the MBVAE model is as follows:

[0019]

[0020]

[0021]

[0022] in, This represents the total loss of the MBVAE model. Indicates the number of samples. This represents the ELBO loss for sample n. This represents the total relevance loss for sample n. This represents the modal alignment loss of sample n. Indicates the total relevance weight. Indicates modal alignment weights, This indicates the calculation of KL divergence. Denotes the true joint posterior distribution of the three joint latent embeddings. Indicates encoder pair The posterior distribution estimate, Indicates encoder pair The posterior distribution estimate, Indicates the inference network pair The posterior distribution estimate, This represents the grouping information of sample n. Represents the latent variable The scoring function, Indicates encoder pair The posterior distribution estimate, This represents the expectation operation. This represents the set of multimodal latent variables.

[0023] Furthermore, the formula for calculating the ELBO loss is as follows:

[0024]

[0025]

[0026]

[0027] in, This indicates an ELBO loss. This represents the ELBO loss term for normal samples. This represents the ELBO loss term for outlier samples. This represents the conditional probability of the generative model. This represents the m-th modality of the n-th sample. Indicates the number of modes. Indicates encoder pair The posterior distribution estimate, This represents the grouping information of sample n. The general symbol for representing a prior distribution. This indicates the encoder's understanding of common latent variables. The posterior distribution estimate, This indicates the calculation of KL divergence. Indicates encoder pair The posterior distribution estimate, This indicates the expected operation.

[0028] Preferably, the process of fine-tuning the pre-trained MBVAE model together with the clustering module includes:

[0029] The readout operation is used to process the anomalous joint latent embeddings of the MBVAE model to obtain a graph-level representation;

[0030] The similarity between the graph representation and the cluster centers is calculated using the Student t-distribution, and the clustering probability of the samples corresponding to the graph representation is obtained, which is the neural typing result.

[0031] The divergence loss of the clustering module is calculated based on the clustering probability of the samples. The weighted sum of the divergence loss of the clustering module and the total loss of the MBVAE model is used as the total loss of the deep embedding clustering model. The model parameters are adjusted according to the total loss of the deep embedding clustering model to obtain the trained deep embedding clustering model.

[0032] Furthermore, the formula for calculating the similarity between the graph-level representation and the cluster centers is:

[0033]

[0034] in, This represents the probability of assigning sample n to cluster k. The graph-level representation of sample n, This represents the degrees of freedom of the student's t-distribution. The graph-level representation of the center of the k-th cluster. The graph-level representation of the center of the k'-th cluster.

[0035] Furthermore, the formula for calculating the divergence loss of the clustering module is:

[0036]

[0037] in, This represents the divergence loss of the clustering module. Indicates the number of samples. Indicates the number of neurological subtypes. This represents the confidence level that sample n belongs to the k-th neurotype. This represents the probability of assigning sample n to cluster k.

[0038] The beneficial effects of this invention are as follows:

[0039] Strong feature discrimination: By designing a deep embedding clustering model based on GNN, the effective fusion of multimodal brain map information is achieved, and individual-specific map features for clustering tasks are learned, which greatly improves the discriminative power of the features.

[0040] High classification accuracy: Semi-supervised depth map clustering is used, and NC information is incorporated as a supervision signal, which effectively reduces the interference of confounding factors, making the ASD neural classification results more accurate and reliable.

[0041] Strong robustness and generalization ability: By proposing a model optimization technique based on subtype-specific mind map synthesis, the problem of gender imbalance and multi-site heterogeneity in ASD data is effectively solved, thereby enhancing the robustness and generalization ability of the model.

[0042] End-to-end optimization: It realizes the joint optimization of feature learning and clustering tasks, avoids the suboptimal solution problem of traditional two-stage methods, and improves the overall classification performance. Attached Figure Description

[0043] Figure 1 This is a schematic diagram of the deep embedding clustering model in this invention;

[0044] Figure 2 This is a schematic diagram of the MBVAE data flow when the input is an ASD mind map in this invention;

[0045] Figure 3 This is a schematic diagram of the probabilistic graphical model for generating multimodal brain maps in this invention;

[0046] Figure 4 This is a schematic diagram of self-supervised mode alignment in this invention. Detailed Implementation

[0047] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0048] To overcome the problems of insufficient feature extraction, inadequate handling of data complexity, and limitations of classification models in existing ASD neural typing methods, this invention provides a neural typing method for ASD that achieves high accuracy and reliability, namely a neural typing method based on multimodal brain mapping and semi-supervised deep clustering. The method includes the following steps: acquiring the user's multimodal neuroimaging data and constructing a multimodal brain network; inputting the multimodal brain network into a trained deep embedding clustering model for processing to obtain the neural typing result; the deep embedding clustering model designed in this invention includes an MBVAE model and a clustering module.

[0049] The training process of a deep embedding clustering model includes:

[0050] This invention designs an alignment and fusion strategy for multimodal brain maps, fully mining the consistency and complementarity information contained in multimodal brain networks to obtain high-quality ASD-specific embedding representations for clustering, achieving accurate ASD neurotyping, such as... Figure 1 As shown. Figure 1 This paper presents a complete framework for identifying subtypes of autism spectrum disorder. The process begins with the input module in the lower left corner, receiving multimodal brain network data from autistic patients and normal controls from multiple sites. This data is first pre-trained on our "Variational Graph Autoencoder Based on Multimodal Brain Networks," whose core is a three-branch encoder designed to decouple the input information into three parts: site-invariant common features, diagnostic group-specific features, and technical noise introduced by the scanning sites. Here, "site" refers to the data acquisition source, such as MRI equipment from different hospitals or research centers, used to eliminate the influence of multi-site heterogeneity. Subsequently, these decoupled features are fused and reconstructed through a "product expert" module, aiming to generate a clean brain network representation that removes site differences. Finally, these high-quality features are input into a deep embedding clustering module, which automatically classifies autistic patients into subtypes with different brain network patterns in an unsupervised learning manner, thus providing a basis for precision medicine; specifically:

[0051] S1: Acquire multimodal neuroimaging data for training and construct a multimodal brain network based on the multimodal neuroimaging data; the brain network for each modality includes a normal brain network and an abnormal brain network.

[0052] Preferably, multimodal neuroimaging data, such as structural magnetic resonance imaging (sMRI) and resting-state functional magnetic resonance imaging (rs-fMRI), provided by the ABIDE public dataset (including ABIDEI and ABIDEII), as well as self-collected clinical datasets (planned to collect data from 100 ASD patients and 100 normal controls), are utilized, which contain rich clinical information.

[0053] Multimodal neuroimaging data underwent preprocessing, including brain region segmentation and feature extraction of sMRI data using FreeSurfer software, and time-series correction and denoising of rs-fMRI data using Connectome Workbench. Subsequently, multimodal brain networks were constructed based on the multimodal neuroimaging data, including morphological brain networks (based on correlations of features such as cortical thickness) and functional brain networks (based on correlations of brain region time series). Each modality included a normal brain network (normal control group NC) and an abnormal brain network (ASD patient group).

[0054] S2: The MBVAE model is pre-trained using a multimodal brain network to obtain a pre-trained MBVAE model.

[0055] Multimodal brain network data is characterized by high dimensionality and heterogeneity, leading to problems such as the curse of dimensionality and difficulties in feature alignment when analyzed directly. The Multimodal Brain Variational Autoencoder (MBVAE) based on multimodal brain networks employs a feature decoupling and fusion strategy to map multimodal data to a unified low-dimensional latent space, thereby extracting more representative feature representations. This model, based on probabilistic graphical theory, a unique network architecture, and a carefully designed loss function, achieves efficient encoding and decoding of multimodal brain network data; specifically:

[0056] The foundation of probabilistic graphical models is the probabilistic framework of variational autoencoders. Probabilistic graphical models use graphs to represent the probabilistic dependencies of variables, combining knowledge of probability theory and graph theory to represent the joint probability distribution of variables related to the model. This paper designs a variational graphical autoencoder (MBVAE) based on a multimodal brain network, using generative probabilistic graphical models to learn individual-specific features of ASD for neurodetyping. In MBVAE, this invention assumes that observed data are generated by latent variables, and the generation process follows a certain probability distribution. For multimodal data, the model needs to learn a shared latent space, allowing data from different modalities to align and interact within this space. For multimodal cases, specific fusion strategies need to be designed to handle the latent representations of different modalities; common methods include early fusion, late fusion, and hybrid fusion strategies. The MBVAE model uses a conditional probabilistic graphical model to describe the relationship between latent variables and observed variables. In this representation, nodes represent random variables, and edges represent the probabilistic dependencies between variables. Directed edges are used to encode the generation process, i.e., the generation path from latent variable z to observed data x; while undirected edges are used to represent the interaction relationships between different modalities. This probabilistic graph provides a mathematical foundation for the joint modeling of multimodal brain network data, and also provides guiding principles for the design of subsequent inference algorithms.

[0057] MBVAE's network architecture employs an encoder-decoder structure, specifically optimized for the characteristics of multimodal brain network data. The encoder maps input data to the latent space, while the decoder reconstructs the input data from the latent representation. Unlike traditional autoencoders, MBVAE's encoder does not directly generate latent representations; instead, it generates distribution parameters (mean and variance) of the latent space and then samples these parameters from the distribution using reparameterization techniques. For its multimodal nature, MBVAE uses a multi-branch encoder design, with each modality corresponding to a dedicated encoding sub-network. Each modality's encoder outputs its own posterior distribution parameters, which are then integrated into a unified latent representation using a specific fusion strategy. This fusion strategy is a key aspect of MBVAE's architecture design. The decoder employs a shared-dedicated hybrid architecture. The shared decoder decodes common features from the fused latent representations and then reconstructs the original modality data using dedicated decoder heads for each modality. This design ensures effective information sharing between modalities while also considering the unique properties of each modality, promoting the learning of decoupled representations while maintaining reconstruction quality.

[0058] The MBVAE model uses an encoder-decoder structure, with the encoder consisting of three sub-encoders. The pre-training process of the MBVAE model includes:

[0059] Step 1: The three sub-encoders encode the brain network for each modality, obtaining three types of coding features for each modality: normal coding features, shared coding features, and anomalous coding features. Specifically:

[0060] Given the subject Multimodal brain networks, represented as brain maps. ,in Let T be a set of T nodes; Let be the set of edges; It is a picture The adjacency matrix, Non-zero elements in the array represent an edge between two nodes; It is the feature matrix of the nodes, and the feature vector of each node is defined as its correlation with all other nodes (which can be obtained from...). (obtained from) Representing modalities, this invention focuses on morphological brain networks constructed based on sMRI. and functional brain networks constructed based on rs-fMRI ,Right now Modalities can also be added as needed.

[0061] In some embodiments of the present invention, for each mode For the brain map, MBVAE uses three independent encoders based on graph convolution (GCN) or graph attention (GAT) layers. , , Encode it. Among them, Input NC samples of mode m, Input ASD samples of mode m, The input consists of NC and ASD samples stacked in dimensions, and its output (shared encoded features) acts as a bridge connecting the generation processes of NC and ASD, which is the key to the model's ability to achieve multimodal fusion and anti-interference capability.

[0062] Step 2: The expert product module is used to fuse the corresponding coding features under all modalities to obtain three comprehensive coding features; the three comprehensive coding features are sampled separately to obtain three joint latent embeddings, namely normal joint latent embeddings. Shared joint implicit embedding Combined with anomalies and hidden embedding .

[0063] Multimodal information is fused using the Product of Experts (PoE). For each distribution, PoE integrates the probability distributions of multiple modalities through variational inference to obtain a comprehensive probability distribution, which is the consensus representation of all experts in the latent space. The hidden posterior probability distribution of each channel can be decomposed as follows:

[0064]

[0065] in, This indicates that the latent variables inferred from all modalities of the mind map all conform to a Gaussian distribution. The latent variables are obtained by multiplying the individual Gaussian variables. It also conforms to a Gaussian distribution, with a mean of and variance for:

[0066]

[0067] in, and It is a latent variable The mean and variance of the distribution. These are the Hadamard products. These parameters are used to define the Gaussian distribution of the latent variables and to compute the statistical properties of the joint latent embeddings during the fusion process.

[0068] Sampling is performed on the three integrated coding features, i.e., the integrated probability distributions, to obtain the corresponding joint latent embeddings, i.e., the normal joint latent embeddings. Shared joint implicit embedding Combined with anomalies and hidden embedding .

[0069] Step 3: Set up a decoder for each modality, with the input being... At that time, the output reconstructs the normal brain network; the input is... When the time comes, output the reconstructed abnormal network.

[0070] MBVAE establishes a decoder for each modality. When the subject is NC, the input is When the subject has ASD, the input is: At this time, the data stream in MBVAE is as follows: Figure 2 As shown. Figure 2 This paper presents a deep learning model architecture for processing neurotypical (NC) and autism spectrum disorder (ASD) brain mapping data. The main body of the graph consists of multiple boxes and arrows representing neural network modules, clearly showing the direction of information flow. The model processes data from left to right. The input on the left side consists of two types of graph data: structural graphs and functional graphs, each containing an adjacency matrix (A) and node features (X). This data is fed into three parallel encoding paths: an upper NC-specific path, a middle shared path, and a lower ASD-specific path. Each path contains an encoder (E) to extract latent representations. The core of the model is the middle shared module, which contains a "product expert" (PoE) module, represented by a dark exploded icon, for integrating information. The latent variables generated by the three paths are NC-specific (…). ),shared( ) and ASD specificity ( — These variables are fed into the decoder (D) on the right. The decoder reconstructs the input graph data by selectively combining these variables; for example, when reconstructing NC data, only NC-specific and shared variables are used, while ASD-specific variables are set to zero.

[0071] The entire architecture, through this separation and recombination mechanism, aims to learn effective data representations that highlight the differences and commonalities among different groups. The decoder consists of fully connected layers and inner product layers, reconstructing the adjacency matrix using the following formula:

[0072]

[0073] in, T represents the number of brain regions / nodes. For the sigmoid function, The weight matrix is The output of the fully connected layer.

[0074] Generating multimodal brain maps The probabilistic graphical model representation is shown in [link to the original text]. Figure 3 . Figure 3 This diagram presents a conceptual model for generating multimodal brain maps, primarily comparing two groups: normal control (NC) and autism spectrum disorder (ASD). The diagram features three key information source nodes: NC-specific information at the top, NC-ASD shared information in the middle, and ASD-specific information at the bottom. This information is input into the generation module on the right via arrows. The generation module on the right comprises two main parts: the upper part generates multimodal brain maps for NC, described by conditional probabilities, indicating that only NC and shared information are used when generating NC brain maps, while ASD information is set to zero; the lower part generates multimodal brain maps for ASD, indicating that only ASD and shared information are used when generating ASD brain maps, while NC information is set to zero. The labels ∀m ∈ M indicate that the model is applicable to multiple modalities, and ∀n ∈ N indicate that it is applicable to multiple samples. Overall, this diagram illustrates how to generate brain maps for different groups by separating and combining group-specific and shared latent information. Generated from three low-dimensional latent variables, including common features of NC and ASD. and their respective individual-specific characteristics and Among them, common features In addition to encoding multimodal brain network information shared by both the ASD and NC groups, it also implicitly includes technical noise from different sites (shared across individuals). This makes the extracted ASD-specific information more meaningful. This process filters out interference from multi-site heterogeneity, allowing for better decoupling of pathological features associated with ASD for subsequent neurological subtyping. It distinguishes... and This helps to more accurately distinguish between health and disease states in subsequent tasks such as typing or classification. The observed samples are composed of latent variables. function The determined conditional distribution was obtained through random sampling. When the subjects... When it is NC, there is When the subject has ASD, there is .

[0075] Step 4: Calculate the total loss of the MBVAE model based on the reconstructed normal brain network, the reconstructed abnormal network, and the three joint implicit embeddings. Adjust the model parameters based on the total loss of the MBVAE model to obtain the pre-trained MBVAE model.

[0076] By utilizing joint latent embeddings and reconstructed data, the model performance is evaluated through multiple loss functions, and the parameters are back-tuned to train the MBVAE model. This design ensures that the MBVAE model can learn discriminative features, laying the foundation for subsequent clustering fine-tuning. The total loss of the MBVAE model consists of three components, as follows:

[0077] Variational inference ELBO loss: It can be derived that the variational lower bounds for the brain network sets of NC and ASD subjects are respectively:

[0078]

[0079]

[0080] in, Kullback-Leibler divergence of two probability distributions This indicates an ELBO loss. This represents the ELBO loss term for normal samples. This represents the ELBO loss term for outlier samples. This represents the conditional probability of the generative model (decoder). This represents the m-th modality of the n-th sample. Indicates the number of modes. This indicates that the inference network (encoder) supports... The posterior distribution estimate, This represents the grouping information of sample n. The general symbol for representing a prior distribution. This indicates the encoder's understanding of common latent variables. The posterior distribution estimate, This indicates the calculation of KL divergence. Indicates encoder pair The posterior distribution estimate, This indicates the expected operation.

[0081] Define the ELBO loss as the negative sum of two ELBOs: .

[0082] Total correlation loss: To promote the independence of the distributions among the three latent variables, a total correlation loss is introduced:

[0083]

[0084] in, Latent variables The joint conditional probability distribution is obtained by minimizing... Make The product approximates the marginal distribution. This term can be approximated using the density ratio technique.

[0085] Modality alignment loss: Aligning data from different modalities in the latent space helps subsequent variational inference. We propose constructing a proxy task within a self-supervised learning framework to implement multimodal alignment. This task promotes alignment between modalities by regularizing the joint space of the single-modality variational posterior and using the dispersion of the latent variables as a penalty. Figure 4 As shown, this is the flow of a probability model: the two source data on the left. and respectively through conditional distribution ( | )and ( | The two sources are encoded into a joint space; this joint space is then connected to the penalty term on the right-hand side by joint probability regularization, which uses discreteness (such as KL divergence) as a regularization means to constrain the latent representations of the two sources to ensure that they remain consistent and correlated in this space.

[0086] This process can be quantified using the following modal alignment loss:

[0087]

[0088] Where ch represents a channel, including three channels: NC-specific, ASD-specific, and Common. It is a set of multimodal latent variables; The joint distribution of the single-modal variational posterior; As a measure of the dispersion of latent variables, it serves as a regularization term to promote consistency in representations across different modalities.

[0089]

[0090]

[0091] in, for The mean, The distance is Euclidean.

[0092] The deep embedding clustering model training of this invention includes two stages. First, MBVAE is pre-trained to obtain a reasonable embedding representation. For all samples... Each iteration selects NC and ASD sample sets with a batch size of B. and Minimize the overall loss function of MBVAE:

[0093]

[0094] in, For ELBO's loss, For total correlation loss, For modal alignment loss, , Weights for controlling various losses.

[0095] S3: Fine-tune the pre-trained MBVAE model together with the clustering module to obtain a trained deep embedding clustering model.

[0096] The second stage of training incorporates a clustering module for fine-tuning, specifically:

[0097] After pre-training the MBVAE module, a clustering module was added for fine-tuning. Clustering analysis was performed in the ASD-specific embedding latent space. To solve the graph-level clustering task, the readout operation was used to aggregate embedding node information to obtain a graph-level representation of each participant's multimodal brain network: Readout operations can include summation, mean calculation, max pooling, etc. For the nth ASD subject and the kth cluster, the data representation of the ASD-specific embedding space is computed using the Student's t-distribution. and cluster center Similarities between them:

[0098]

[0099] in, Initialized by k-means, These are the degrees of freedom of the student's t-distribution. This represents the probability of assigning sample n to cluster k, from which the soft-label distribution of ASD samples can be obtained. Further by Define a target distribution with higher confidence. :

[0100]

[0101] The loss of the clustering module is defined as minimizing the distribution. and distribution KL divergence between:

[0102]

[0103] in, For ASD sample size, The number of clusters represents the number of neural subtypes. By minimizing this loss, the target distribution... This approach uses self-supervised learning to help the model learn better clustering feature representations.

[0104] The weighted sum of the divergence loss of the clustering modules and the total loss of the MBVAE model is used as the total loss of the deep embedding clustering model, and the total model loss is minimized.

[0105]

[0106] in, For the clustering module loss, Parameters contributing to loss control are defined. During backpropagation, network parameters and cluster centers are updated. When the total loss converges or reaches the maximum preset number of iterations, the model parameters are maintained. During the validation phase, semi-synthetic data (simulating ASD subtypes by adjusting the brain network characteristics of NC data) is used to evaluate clustering performance. Indicators such as normalized mutual information and adjusted RAND index are employed, and the optimal number of subtypes is determined by leaving room for cross-validation. After successful validation, a trained deep embedding clustering model is obtained. Multimodal neuroimaging data of users is acquired and a multimodal brain network is constructed. The multimodal brain network is then input into the trained deep embedding clustering model for processing to obtain neurotyping results.

[0107] The clustering results of nodes / samples n, i.e., the neural typing results of ASD, can be derived from the target distribution. It is obtained by calculation using the following formula:

[0108]

[0109] In summary, this invention designs a variational graph autoencoder (MBVAE) model based on multimodal brain maps. This model encodes brain network data through graph convolutional layers or graph attention layers, decoupling common features from individual-specific features, and utilizes expert products for multimodal fusion to reduce heterogeneity interference from multiple sites. Model training consists of two stages: pre-training and fine-tuning. During pre-training, variational lower bound loss, total relevance loss, and modality alignment loss are minimized. During fine-tuning, a deep embedding clustering module is added, and subtype classification is performed by calculating similarity using the Student's t-distribution. Finally, the model is applied to perform neurogenotyping on real ASD data, analyzing differences in brain network indicators (such as topological properties and clinical symptoms) between subtypes to reveal heterogeneity patterns. The entire process emphasizes data reliability and model generalization, providing a new method for ASD research.

[0110] The above-described embodiments further illustrate the purpose, technical solution, and advantages of the present invention. It should be understood that the above-described embodiments are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made to the present invention within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A neural typing method based on multimodal brain mapping and semi-supervised deep clustering, characterized in that, include: The system acquires multimodal neuroimaging data from users and constructs a multimodal brain network. The multimodal brain network is then input into a trained deep embedding clustering model for processing to obtain neuromorphic classification results. Deep embedding clustering models include MBVAE models and clustering modules; The training process of a deep embedding clustering model includes: S1: Acquire multimodal neuroimaging data for training and construct a multimodal brain network based on the multimodal neuroimaging data; the brain network for each modality includes a normal brain network and an abnormal brain network; S2: The MBVAE model is pre-trained using a multimodal brain network to obtain a pre-trained MBVAE model; S3: Fine-tune the pre-trained MBVAE model together with the clustering module to obtain a trained deep embedding clustering model.

2. The neural typing method based on multimodal brain mapping and semi-supervised deep clustering according to claim 1, characterized in that, The MBVAE model is an encoder-decoder structure, with the encoder consisting of three sub-encoders. The pre-training process of the MBVAE model includes: The three sub-encoders encode the brain network for each modality, resulting in three types of coding features for each modality: normal coding features, shared coding features, and abnormal coding features. An expert product module is used to fuse the corresponding coding features across all modalities, resulting in three comprehensive coding features. These three comprehensive coding features are then sampled to obtain three joint latent embeddings, namely, normal joint latent embeddings. Shared joint implicit embedding Combined with anomalies and hidden embedding ; Each mode is assigned a decoder, with the input being... At that time, the output reconstructs the normal brain network; the input is... When the time comes, output the reconstructed abnormal network; The total loss of the MBVAE model is calculated by reconstructing the normal brain network, reconstructing the abnormal network, and using three joint implicit embeddings. The model parameters are then adjusted based on the total loss of the MBVAE model to obtain the pre-trained MBVAE model.

3. The neural typing method based on multimodal brain mapping and semi-supervised deep clustering according to claim 2, characterized in that, Each sub-encoder is a graph convolutional network or a graph attention network; the decoder consists of fully connected layers and inner product layers.

4. The neural typing method based on multimodal brain mapping and semi-supervised deep clustering according to claim 2, characterized in that, The formula for calculating the total loss of the MBVAE model is: ； ； ； in, This represents the total loss of the MBVAE model. Indicates the number of samples. This represents the ELBO loss for sample n. This represents the total relevance loss for sample n. This represents the modal alignment loss of sample n. Indicates the total relevance weight. Indicates modal alignment weights, This indicates the calculation of KL divergence. Denotes the true joint posterior distribution of the three joint latent embeddings. Indicates encoder pair The posterior distribution estimate, Indicates encoder pair The posterior distribution estimate, Indicates the inference network pair The posterior distribution estimate, This represents the grouping information of sample n. Represents the latent variable The scoring function, Indicates encoder pair The posterior distribution estimate, This represents the expectation operation. This represents the set of multimodal latent variables.

5. The neural typing method based on multimodal brain mapping and semi-supervised deep clustering according to claim 4, characterized in that, The formula for calculating ELBO loss is: ； ； ； in, This indicates an ELBO loss. This represents the ELBO loss term for normal samples. This represents the ELBO loss term for outlier samples. This represents the conditional probability of the generative model. This represents the m-th modality of the n-th sample. Indicates the number of modes. Indicates encoder pair The posterior distribution estimate, This represents the grouping information of sample n. The general symbol for representing a prior distribution. This indicates that the encoder has access to common latent variables. The posterior distribution estimate, This indicates the calculation of KL divergence. Indicates encoder pair The posterior distribution estimate, This indicates the expected operation.

6. The neural typing method based on multimodal brain mapping and semi-supervised deep clustering according to claim 1, characterized in that, The process of fine-tuning the pre-trained MBVAE model together with the clustering module includes: The readout operation is used to process the anomalous joint latent embeddings of the MBVAE model to obtain a graph-level representation; The similarity between the graph representation and the cluster centers is calculated using the Student t-distribution, and the clustering probability of the samples corresponding to the graph representation is obtained, which is the neural typing result. The divergence loss of the clustering module is calculated based on the clustering probability of the samples. The weighted sum of the divergence loss of the clustering module and the total loss of the MBVAE model is used as the total loss of the deep embedding clustering model. The model parameters are adjusted according to the total loss of the deep embedding clustering model to obtain the trained deep embedding clustering model.

7. A neural typing method based on multimodal brain mapping and semi-supervised deep clustering according to claim 6, characterized in that, The formula for calculating the similarity between graph-level representations and cluster centers is: ； in, This represents the probability of assigning sample n to cluster k. The graph-level representation of sample n, This represents the degrees of freedom of the student's t-distribution. The graph-level representation of the center of the k-th cluster. The graph-level representation of the center of the k'-th cluster.

8. A neural typing method based on multimodal brain mapping and semi-supervised deep clustering according to claim 6, characterized in that, The formula for calculating the divergence loss of the clustering module is: ； in, This represents the divergence loss of the clustering module. Indicates the number of samples. Indicates the number of neurological subtypes. This represents the confidence level that sample n belongs to the k-th neuromorphic subtype. This represents the probability of assigning sample n to cluster k.

Images