A cross-site neural image mental illness recognition method and system based on multi-modal fusion

Abstract

This invention discloses a method and system for cross-site neuroimaging-based mental illness identification based on multimodal fusion. The method includes: acquiring and preprocessing multimodal neuroimaging data to obtain a constructed multimodal neuroimaging dataset; constructing a mental illness identification model by inputting the multimodal neuroimaging dataset into the model to obtain site prediction results and disease prediction results; calculating site classification loss and disease classification loss based on the site prediction results and disease prediction results, and establishing an objective function; training the mental illness identification model based on the objective function and the multimodal neuroimaging dataset until a preset training cutoff condition is met, and outputting the trained mental illness identification model; and using the trained model for auxiliary diagnosis of mental illness in new subjects. This invention effectively overcomes the interference of heterogeneity in multi-center data and achieves high-precision, high-generalization objective diagnosis of mental illness.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and adaptive analysis of medical images, and in particular to a cross-site neuroimaging method and system for identifying mental illnesses based on multimodal fusion. Background Technology

[0002] As one of the leading causes of disability worldwide, the objective identification of severe mental illnesses is of great clinical significance. Currently, clinical identification of mental illnesses, such as major depressive disorder (MDD), relies primarily on subjective interviews and assessments of symptoms by physicians, lacking quantifiable biomarkers. This leads to low consistency in identification and significant difficulties in differentiation. Furthermore, the overlap of clinical symptoms with other mental illnesses further complicates differential diagnosis. Although neuroimaging techniques offer new perspectives on revealing the pathological mechanisms of mental illnesses, existing research methods have not yet established objective diagnostic criteria that meet clinical requirements.

[0003] In the field of neuroimaging, structural magnetic resonance imaging (sMRI) and functional magnetic resonance imaging (fMRI) can provide complementary information from the perspectives of brain structure and functional activity, respectively, providing important evidence for the study of mental illnesses. However, existing analytical methods have significant limitations: on the one hand, most studies are still limited to single-modal analysis or use shallow fusion strategies such as simple feature splicing, failing to fully utilize the complementary advantages of sMRI and fMRI in function-structure coupling; in particular, traditional cross-modal fusion methods, such as unidirectional cross-attention mechanisms, suffer from information flow asymmetry and insufficient feature utilization, limiting the model's ability to integrate and represent complex pathological patterns of mental illnesses; on the other hand, methods based on traditional machine learning rely on artificial feature engineering and shallow models, making it difficult to effectively capture complex nonlinear pathological patterns of mental illnesses, resulting in insufficient model generalization ability.

[0004] Furthermore, building robust disease identification models using multi-center data is an inevitable trend, but the resulting site-to-site differences pose a significant challenge. Factors such as equipment and scanning parameters in different imaging centers lead to systematic biases in the data. Existing site correction techniques are mostly based on static assumptions unrelated to biological signals, resulting in limited correction effectiveness and a tendency to produce false results when sample heterogeneity is high. Although adaptive methods based on adversarial learning and other fields have emerged recently, these methods are mostly validated in scenarios with a limited number of sites. When faced with massive amounts of data from numerous centers, their scalability, complexity, and stability face severe challenges. Summary of the Invention

[0005] The purpose of this invention is to provide a cross-site neuroimaging method and system for identifying mental illnesses based on multimodal fusion, so as to solve the problems mentioned in the background art.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] In a first aspect, the present invention provides a cross-site neuroimaging method for identifying mental illnesses based on multimodal fusion, characterized in that it includes:

[0008] Acquire multimodal neural imaging data and preprocess the neural imaging data to construct a multimodal neural imaging dataset;

[0009] The multimodal neuroimaging data includes structural imaging data and functional imaging data;

[0010] The mental illness identification model is constructed by inputting preprocessed multimodal neuroimaging data into the mental illness identification model to obtain site prediction results and disease prediction results.

[0011] Based on the site prediction results and disease prediction results, the site classification loss and disease classification loss are calculated. The objective function is established with the goal of minimizing the disease classification loss and maximizing the site classification loss.

[0012] Based on the objective function and multimodal neuroimaging dataset, a mental illness recognition model is trained until the preset training cutoff condition is met, and the trained mental illness recognition model is output.

[0013] The multimodal neuroimaging data of the subject to be identified is input into the trained mental illness identification model to obtain the final disease identification result.

[0014] In an optional implementation, the construction of a mental illness identification model and the input of the multimodal neuroimaging dataset into the mental illness identification model to obtain site prediction results and disease prediction results specifically include the following steps:

[0015] Construct a model for identifying mental illness;

[0016] The mental illness identification model includes a two-stream feature extraction network, a feature fusion network, and a classifier network;

[0017] The classifier network includes a site classifier and a disease classifier;

[0018] Based on a dual-stream feature extraction network, feature extraction is performed on the preprocessed structural image data and functional image data respectively to obtain structural image feature representation and temporal feature representation;

[0019] Based on the feature fusion network, bidirectional cross-attention fusion is performed on the structural image feature representation and the temporal feature representation to obtain a multimodal fused representation;

[0020] The multimodal fusion representation is input into the site classifier to obtain the site prediction result;

[0021] The multimodal fusion representation, structural image feature representation, and temporal feature representation are concatenated and then input into the disease classifier to obtain the disease prediction result.

[0022] In an optional implementation, the step of extracting features from the preprocessed structural image data and functional image data based on a dual-stream feature extraction network to obtain structural image feature representations and temporal feature representations specifically includes the following steps:

[0023] The preprocessed structural image data and functional image data are input into a two-stream feature extraction network;

[0024] The dual-stream feature extraction network includes a structural image feature extraction branch based on the 3D ResNet algorithm and a temporal feature extraction branch based on the BiLSTM-Transformer algorithm;

[0025] Based on the structural image feature extraction branch, hierarchical spatial feature extraction is performed on the preprocessed structural image data, and the extracted spatial features are then subjected to three-dimensional global average pooling and linear transformation to obtain a structural image feature representation.

[0026] Based on the BiLSTM unit in the temporal feature extraction branch, the preprocessed functional image data is processed to obtain a preliminary temporal feature representation containing bidirectional temporal information. The preliminary temporal feature representation is input into the Transformer unit, and the effective time step outputs of the Transformer unit are aggregated and then processed by linear transformation to obtain the temporal feature representation.

[0027] In an optional implementation, the feature fusion network performs bidirectional cross-attention fusion on the structural image feature representation and the temporal feature representation respectively to obtain a multimodal fused representation, specifically including the following steps:

[0028] The structural image feature representation and temporal feature representation are input into the feature fusion network;

[0029] Using temporal features as the query vector and structural image features as the key and value vectors, the first cross-attention is calculated to obtain the enhanced temporal feature representation;

[0030] Using structural image features as query vectors and temporal features as key and value vectors, a second cross-attention is calculated to obtain enhanced structural image feature representations.

[0031] Based on the enhanced temporal feature representation and the enhanced structural image feature representation, a first fusion weight and a second fusion weight are calculated respectively; based on the first fusion weight and the second fusion weight, the enhanced temporal feature representation and the enhanced structural image feature representation are weighted and summed to obtain a multimodal fusion representation.

[0032] In an optional implementation, the step of calculating site classification loss and disease classification loss based on site prediction results and disease prediction results, with the optimization objective being to minimize the disease classification loss and maximize the site classification loss, specifically includes the following steps:

[0033] Based on the predicted site results and the actual site labels, the site classification loss is calculated using the cross-entropy loss function.

[0034] Based on the disease prediction results and the actual disease labels, the disease classification loss is calculated using the cross-entropy loss function.

[0035] Based on site classification loss and disease classification loss, an objective function is established with the optimization objective of minimizing the disease classification loss and maximizing the site classification loss;

[0036] The expression for the objective function is:

[0037]

[0038] In the formula, Total loss; For disease classifier parameters; For site classifier parameters; Loss due to disease classification; Losses are categorized by site; This is the loss weight.

[0039] In an optional implementation, the step of training a mental illness recognition model based on an objective function and a multimodal neural image dataset until a preset training cutoff condition is met, and then outputting the trained mental illness recognition model, specifically includes the following steps:

[0040] A1: Initialize the parameters of the mental illness identification model;

[0041] A2: The mental illness recognition model is trained based on the multimodal neuroimaging dataset;

[0042] A3: Repeat step A2 until the mental illness recognition model reaches the preset convergence condition, and obtain the trained mental illness recognition model;

[0043] Step A2 specifically includes the following steps:

[0044] The data from the multimodal neuroimaging dataset is input into the mental illness recognition model, and then passed through the dual-stream feature extraction network, feature fusion network, and classifier network for forward propagation to obtain site prediction results and disease prediction results.

[0045] Based on the site prediction results and the disease prediction results, the total loss is calculated using an objective function, and backpropagation is performed to update the parameters of the mental illness identification model. During the backpropagation process, a gradient reversal strategy is executed on the gradients passed to the site classifier, and the parameters of the two-stream feature extraction network and the feature fusion network are updated using the reversed gradients, so that the identification model learns domain-invariant features to eliminate site specificity among multi-site data.

[0046] Secondly, the present invention provides a cross-site neuroimaging system for identifying mental illnesses based on multimodal fusion, characterized in that it includes:

[0047] The data acquisition module is used to acquire multimodal neural image data and preprocess the neural image data to construct a multimodal neural image dataset.

[0048] The multimodal neuroimaging data includes structural imaging data and functional imaging data;

[0049] The model building module is used to build a mental illness identification model. It inputs preprocessed multimodal neuroimaging data into the mental illness identification model to obtain site prediction results and disease prediction results.

[0050] The objective function module is used to calculate the site classification loss and the disease classification loss based on the site prediction results and the disease prediction results, with the optimization objective being to minimize the disease classification loss and maximize the site classification loss.

[0051] The model training module is used to train a mental illness recognition model based on an objective function and a multimodal neuroimaging dataset until the preset training cutoff condition is met, and then output the trained mental illness recognition model.

[0052] The disease identification module is used to input the multimodal neuroimaging data of the subject to be identified into the trained mental illness identification model to obtain the final disease identification result.

[0053] Thirdly, the present invention provides an electronic device, comprising:

[0054] At least one processor;

[0055] At least one memory storing computer-executable instructions, wherein, when executed by the at least one processor, the at least one processor causes the at least one processor to perform the multimodal fusion-based cross-site neuroimaging method for identifying mental disorders as described above.

[0056] Fourthly, the present invention provides a computer-readable storage medium in which instructions, when executed by at least one processor, cause the at least one processor to perform the cross-site neuroimaging method for identifying mental disorders based on multimodal fusion as described above.

[0057] Fifthly, the present invention provides a computer program product including computer instructions, which, when executed by at least one processor, cause the at least one processor to perform the multimodal fusion-based cross-site neuroimaging method for identifying mental disorders as described above.

[0058] The beneficial effects of the embodiments provided by the present invention include:

[0059] This invention utilizes a dual-stream deep coding network combining 3D ResNet and BiLSTM-Transformer to optimize the extraction of spatial and temporal features, thereby enabling automatic learning of deep nonlinear patterns in high-dimensional image data and overcoming the limitations of traditional models that rely on manual features and have limited representation capabilities.

[0060] This invention achieves deep interaction between structural and functional modalities through a bidirectional cross-attention fusion mechanism, overcoming the information asymmetry defect of traditional unidirectional fusion strategies and significantly improving the ability to integrate multidimensional pathological features.

[0061] This invention employs an adversarial training method based on minimax optimization, combined with a gradient reversal strategy, which can effectively remove site-specific biases in multi-center data, thereby enabling the feature extraction network to learn site-invariant features and thus improving the cross-site generalization performance of the mental illness identification model.

[0062] This invention constructs and trains a mental illness identification model using deep feature extraction, multimodal fusion, and domain adaptation techniques. This model effectively overcomes the clinical challenges of low consistency and insufficient standardization in mental illness identification caused by reliance on subjective doctor assessments and patient self-reports in existing technologies. It achieves objective quantitative analysis based on multimodal image data, providing high-precision and reproducible evidence for clinical identification of mental illnesses, and significantly improving the reliability and consistency of mental illness identification results. Attached Figure Description

[0063] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0064] Figure 1 A flowchart of the cross-site neuroimaging method for identifying mental disorders, as illustrated in the embodiments of this specification, is shown.

[0065] Figure 2 A schematic diagram of the framework of the mental illness identification model in the embodiments of this specification is shown;

[0066] Figure 3 A schematic diagram of the training process of the mental illness identification model in the embodiments of this specification is shown;

[0067] Figure 4 A visual comparison chart of the fused feature distribution in the embodiments of this specification is shown;

[0068] in, Figure 4 (a) is the distribution map of fused features without using the gradient inversion strategy; Figure 4 (b) is the fusion feature distribution map after using the gradient reversal multi-task learning strategy;

[0069] Figure 5 A block diagram of the cross-site neuroimaging mental illness identification system in an embodiment of this specification is shown. Detailed Implementation

[0070] The features and exemplary embodiments of various aspects of the present invention will now be described in detail. In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the invention. However, it will be apparent to those skilled in the art that the invention may be practiced without requiring some of these specific details. The following description of embodiments is merely intended to provide a better understanding of the invention by illustrating examples of the invention.

[0071] In some of the processes described in the specification, claims, and accompanying drawings of this invention, multiple operations appear in a specific order. However, it should be clearly understood that these operations may not be performed in the order they appear herein, or they may be performed in parallel. The operation numbers, such as S1, S2, etc., are merely used to distinguish different operations and do not themselves represent any execution order. Furthermore, these processes may include more or fewer operations, and these operations may be performed sequentially or in parallel.

[0072] The following explains the technical terms used in the embodiments:

[0073] 3D ResNet: 3D ResNet (Three-Dimensional Residual Network) is a deep residual convolutional neural network suitable for processing three-dimensional volumetric data. By introducing a cross-layer residual connection structure, this network effectively alleviates the gradient vanishing problem in deep network training and can extract discriminative spatial hierarchical features from volumetric data such as brain structure images through multi-layer three-dimensional convolutional operations.

[0074] BiLSTM-Transformer: BiLSTM-Transformer is a hybrid temporal feature extraction architecture that combines a bidirectional long short-term memory network with the Transformer architecture. This architecture first uses BiLSTM to capture bidirectional long-term dependencies in time series, and then uses the self-attention mechanism of Transformer to enhance the feature representation of key time steps. It is suitable for deep feature encoding of time series signals such as fMRI.

[0075] BiLSTM: Bidirectional Long Short-Term Memory (BiLSTM) is a bidirectional long short-term memory network that uses forward and backward LSTM units to process time series data from both the forward and backward directions, thereby comprehensively capturing the contextual dependencies in time series and effectively improving the ability to model dynamic features of time series.

[0076] Transformer: The Transformer is a deep learning architecture based on the self-attention mechanism. It dynamically focuses on key information and enhances features by calculating the correlation weights between elements of the input sequence. This model abandons the traditional recurrent neural network structure and has the characteristics of high parallel computing efficiency and strong long-range dependency modeling ability, making it widely applicable to sequence data processing tasks.

[0077] Example 1

[0078] like Figure 1 As shown, this embodiment provides a cross-site neuroimaging method for identifying mental illnesses based on multimodal fusion, including:

[0079] S1: Acquire multimodal neural image data and preprocess the neural image data to construct a multimodal neural image dataset;

[0080] In this embodiment, the multimodal neuroimaging data includes structural imaging data and functional imaging data; wherein, the structural imaging data is structural magnetic resonance imaging data, and the functional imaging data is functional magnetic resonance imaging data.

[0081] For example, preprocessing of neuroimaging data specifically includes:

[0082] For structural magnetic resonance imaging data, spatial normalization, craniotomy, tissue segmentation and spatial smoothing are performed, and the structural images of each subject are registered to the standard MNI space to obtain gray matter volume data, i.e. preprocessed structural image data.

[0083] For functional magnetic resonance imaging data, time-layer correction, head motion correction, spatial standardization, and smoothing filtering are performed sequentially, and covariates such as white matter signal, cerebrospinal fluid signal, and global average signal are regressed. Finally, brain region time-series data are extracted based on AAL brain atlas, which is the preprocessed functional imaging data.

[0084] In some embodiments, to adapt to the input requirements of subsequent deep learning models, the above data can be normalized:

[0085] The time series data is length-standardized, and sequences shorter than the maximum time step are zero-paddinged to form a unified 116×240-dimensional input representation.

[0086] The gray matter volume data is subjected to min-max normalization to eliminate the influence of dimensions.

[0087] S2: The mental illness identification model is constructed by inputting the preprocessed multimodal neuroimaging data into the mental illness identification model to obtain site prediction results and disease prediction results;

[0088] like Figure 2 As shown, exemplarily, step S2 specifically includes the following steps:

[0089] S201: Constructing a model for identifying mental illnesses;

[0090] In this embodiment, the mental illness identification model includes a two-stream feature extraction network, a feature fusion network, and a classifier network;

[0091] The classifier network includes site classifiers and disease classifiers;

[0092] S202: Based on a dual-stream feature extraction network, feature extraction is performed on the preprocessed structural image data and functional image data respectively to obtain structural image feature representation and temporal feature representation;

[0093] In some embodiments, step S202 specifically includes the following steps:

[0094] S2021: Input the preprocessed structural image data and functional image data into the dual-stream feature extraction network;

[0095] The dual-stream feature extraction network includes a structural image feature extraction branch based on the 3D ResNet algorithm and a temporal feature extraction branch based on the BiLSTM-Transformer algorithm.

[0096] S2022: Based on the structural image feature extraction branch, hierarchical spatial feature extraction is performed on the preprocessed structural image data, and the extracted spatial features are subjected to three-dimensional global average pooling and linear transformation to obtain a structural image feature representation.

[0097] In this embodiment, the structural image feature extraction branch adopts a deep convolutional neural network 3D ResNet-34 architecture, which includes a four-level residual structure. Each level consists of a preset number of residual blocks, and each residual block includes at least one residual mapping path composed of a three-dimensional convolutional layer and one skip connection path.

[0098] The skip connection path is used to add the input of the residual block to the output of the residual mapping path element by element, and then process the addition result through the activation function to form the output of the residual block, which can ensure the effective propagation of gradients during training.

[0099] Specifically, the structural image feature extraction branch performs hierarchical spatial feature extraction on the preprocessed structural image data through a series of cascaded residual blocks to obtain a high-dimensional spatial feature map. Then, a three-dimensional global average pooling operation is performed on the spatial feature map to reduce its spatial dimension to a feature vector. Finally, the feature vector is mapped to a high-dimensional space through the first linear transformation layer, and the output is a structural image feature representation.

[0100]

[0101] In the formula, The structure image features are represented by Softmax(·); Softmax(·) is the Softmax activation function. and These are the weight matrix and bias vector of the first linear transformation layer, respectively; Flatten(·) is the flattening operation; AvgPool(·) is the three-dimensional global average pooling operation; For cascading of fourth-level residual blocks; The computation performed for the i-th residual block in level l; nl represents the number of residual blocks contained in level l, where n takes the value [3,4,6,3]. This refers to the preprocessed structural image data;

[0102] S2023: Based on the BiLSTM unit in the temporal feature extraction branch, the preprocessed functional image data is processed to obtain a preliminary temporal feature representation containing bidirectional temporal information. The preliminary temporal feature representation is input into the Transformer unit, and the effective time step outputs of the Transformer unit are aggregated and then processed by linear transformation to obtain the temporal feature representation.

[0103] In this embodiment, the preprocessed functional image data is first processed using a BiLSTM unit. The hidden state of each time step is obtained through its forward and backward propagation, and the bidirectional hidden states of the corresponding time steps are concatenated to generate a preliminary temporal feature representation containing bidirectional temporal dependencies.

[0104] Subsequently, the preliminary temporal feature representation is input into the Transformer unit. The Transformer unit automatically ignores zero-padding time steps during computation through its built-in attention masking mechanism, models only the global dependencies between valid time steps identified by the mask, and performs weighted average aggregation on the output sequence of the Transformer unit. Finally, the temporal feature representation is obtained through the second linear transformation layer.

[0105]

[0106]

[0107] In the formula, The output sequence of the Transformer unit; This represents the preliminary temporal characteristics. B is the set of real numbers; B is the batch size, i.e., the number of samples processed; T is the time step; d is the feature dimension. It represents the temporal characteristics; and These are the weight matrix and bias vector of the second linear transformation layer, respectively;

[0108] It should be noted that the two feature extraction branches mentioned above adopt a parameter-independent design scheme. The structural image feature extraction branch focuses on the extraction of spatial dimension features, while the temporal feature extraction branch focuses on the capture of temporal dynamic features. This division of labor and cooperation mechanism ensures the complementarity and integrity of different modal features.

[0109] S203: Based on the feature fusion network, bidirectional cross-attention fusion is performed on the structural image feature representation and the temporal feature representation respectively to obtain a multimodal fusion representation;

[0110] In this embodiment, the feature fusion network includes two symmetrical cross-attention units and a modality weight allocation unit;

[0111] In some embodiments, step S203 specifically includes the following steps:

[0112] S2031: Input the structural image feature representation and temporal feature representation into the feature fusion network;

[0113] S2032: In a cross-attention unit, the query vector is represented by temporal features, and the key vector and value vector are represented by structural image features. The first cross-attention is calculated to obtain the enhanced temporal feature representation.

[0114] Specifically, through a cross-attention mechanism, key information from structural image feature representation is dynamically integrated into temporal feature representation to obtain enhanced temporal feature representation. Its functional expression is as follows:

[0115]

[0116]

[0117] In the formula, To enhance the representation of temporal features; This is the first cross-attention; , , Let be the first set of learnable parameter matrices, where d is the feature dimension. Used for gradient stabilization;

[0118] S2033: In another cross-attention unit, the structural image features are represented as query vectors, and the temporal features are represented as key vectors and value vectors. The second cross-attention is calculated to obtain the enhanced structural image feature representation.

[0119] Specifically, through a cross-attention mechanism, key information from temporal feature representations is dynamically integrated into structural image feature representations to obtain enhanced structural image feature representations. The functional expression for this enhancement is:

[0120]

[0121]

[0122] In the formula, To enhance the representation of structural image features; This is for the second cross attention; , , This is the second set of learnable parameter matrices;

[0123] S2034: Based on the enhanced temporal feature representation and the enhanced structural image feature representation, calculate the first fusion weight and the second fusion weight respectively;

[0124] In this embodiment, the calculation of the first fusion weight and the second fusion weight is implemented through a modality weight allocation unit (MWAM). This unit takes the concatenation result of the two enhanced feature vectors mentioned above as input, processes it through a fully connected layer and an activation function, and finally outputs the fusion weights of each modality through a softmax function, specifically expressed as follows:

[0125]

[0126] In the formula, , For learnable weight matrix, For vector concatenation operations, ω=[ , [ ] represents the weighting coefficients assigned to the two modes;

[0127] S2035: Based on the first fusion weight and the second fusion weight, the enhanced temporal feature representation and the enhanced structural image feature representation are weighted and summed to obtain a multimodal fusion representation.

[0128] Specifically, the expression for multimodal fusion representation is:

[0129]

[0130] In the formula, For multimodal fusion representation; First fusion weight; Second fusion weight;

[0131] This embodiment achieves equal interaction between functional and structural modalities through a bidirectional cross-attention mechanism, effectively overcoming the information asymmetry limitations of traditional one-way fusion and fully exploring the complementary information between multimodal data. At the same time, the weight allocation mechanism can adaptively evaluate the importance of each modal feature in the mental illness identification task and perform weighted fusion accordingly to achieve optimal fusion.

[0132] S204: Input the multimodal fusion representation into the site classifier to obtain the site prediction result;

[0133] In some embodiments, site classifier The multimodal fusion representation is used as input, and the predicted probability of a sample belonging to each data collection station is output.

[0134] In this embodiment, a gradient reversal strategy is introduced during the model training phase to eliminate the differences in data distribution between different scanning sites.

[0135] Specifically, during the backpropagation of the site classifier, its loss gradient is multiplied by a negative coefficient and then backpropagated to the feature extraction network. Through this adversarial training, the feature extraction network is driven to suppress site-related features, thereby learning site-independent domain-invariant features.

[0136] S205: The multimodal fusion representation, structural image feature representation and temporal feature representation are concatenated and then input into the disease classifier to obtain the disease prediction result.

[0137] In some embodiments, disease classifier For the final disease-aided identification, the input is not a single feature, but a comprehensive representation constructed through the fusion of multi-level features.

[0138] Specifically, the temporal feature representation, structural image feature representation, and cross-modal fusion representation are concatenated to form a comprehensive feature vector that integrates modality-specific information and cross-modal correlation information. The disease classifier processes this comprehensive feature vector and outputs the predicted probability that the subject is identified as a mental illness patient or a healthy control, thus completing the assisted identification.

[0139] S3: Based on the site prediction results and disease prediction results, calculate the site classification loss and the disease classification loss, and establish an objective function with the optimization objective of minimizing the disease classification loss and maximizing the site classification loss;

[0140] For example, step S3 specifically includes the following steps:

[0141] Based on the predicted site results and the actual site labels, the site classification loss is calculated using the cross-entropy loss function.

[0142] Based on the disease prediction results and the actual disease labels, the disease classification loss is calculated using the cross-entropy loss function.

[0143] Based on site classification loss and disease classification loss, an objective function is established with the optimization objective of minimizing the disease classification loss and maximizing the site classification loss;

[0144] Specifically, the expression for the cross-entropy loss function is:

[0145]

[0146] Where C represents the total number of categories (for site categories, C is the total number of sites). This is an indicator function (it takes a value of 1 when the true class of the sample is i, and 0 otherwise); Predict the probability that the sample belongs to the i-th class for the classifier;

[0147] Specifically, the expression for the objective function is:

[0148]

[0149] In the formula, Total loss; For disease classifier parameters; For site classifier parameters; Loss due to disease classification; Losses are categorized by site; This is the loss weight.

[0150] It should be noted that the real site labels and real disease labels mentioned in this embodiment are pre-existing supervised information independent of the image data: real site labels are derived from the acquisition metadata of multi-center image data, and are inherent site identifiers recorded at the time of data acquisition for each subject; real disease labels are established based on strict clinical diagnostic criteria, and the disease status of the subjects is independently assessed and confirmed by experienced clinicians according to internationally accepted diagnostic criteria. In step S1, when constructing the dataset, these pre-determined labels are associated with the preprocessed neuroimaging data to jointly constitute the supervised dataset used for model training.

[0151] S4: Based on the objective function and multimodal neural image dataset, train the mental illness recognition model until the preset training cutoff condition is met, and output the trained mental illness recognition model.

[0152] For example, step S4 specifically includes the following steps:

[0153] S401: Initialize the parameters of the mental illness identification model;

[0154] S402: Train the mental illness recognition model based on the multimodal neuroimaging dataset;

[0155] like Figure 3 As shown, in some embodiments, step S402 specifically includes the following steps:

[0156] The data from the multimodal neuroimaging dataset is input into the mental illness recognition model, and then passed through the dual-stream feature extraction network, feature fusion network, and classifier network for forward propagation to obtain site prediction results and disease prediction results.

[0157] Based on the site prediction results and the disease prediction results, the total loss is calculated using an objective function, and backpropagation is performed to update the parameters of the mental illness identification model. During backpropagation, a gradient reversal strategy is executed on the gradients passed to the site classifier, and the parameters of the two-stream feature extraction network and the feature fusion network are updated using the reversed gradients, so that the learning domain invariant features of the mental illness identification model can be made to eliminate site specificity among multi-site data.

[0158] S403: Repeat step S402 until the mental illness recognition model reaches the preset convergence condition, and obtain the trained mental illness recognition model.

[0159] The preset convergence condition is that the total loss function no longer decreases significantly for multiple consecutive training epochs on the validation set, or reaches the preset maximum number of training epochs.

[0160] It should be noted that, in order to achieve the above-mentioned optimization objective of minimizing the disease classification loss and maximizing the site classification loss, this embodiment adopts a gradient reversal multi-task learning strategy. This strategy drives the feature extraction network to minimize the disease classification loss while maximizing the site classification loss through adversarial training, thereby learning domain-invariant features that are independent of the site.

[0161] In some embodiments, the training of the mental illness identification model employs a five-fold cross-validation strategy to ensure the fairness of the evaluation and the reliability of the results.

[0162] Specifically, the training process for the mental illness identification model uses the AdamW optimizer, with an initial learning rate set to 5×10⁻. 5 The training process consists of 50 cycles and can be completed on a compute node equipped with four Tesla V100 graphics processors interconnected via NVLink.

[0163] In a preferred embodiment, after the mental illness identification model has been trained, accuracy, recall, area under the receiver operating characteristic curve, and F1 score can be used as performance evaluation indicators to evaluate the performance of the trained mental illness identification model on an independent test set.

[0164] S5: Input the multimodal neuroimaging data of the subject to be identified into the trained mental illness identification model to obtain the final disease identification result.

[0165] It should be emphasized that the multimodal fusion recognition framework provided in this embodiment is universal and can be applied to different categories of mental illness auxiliary recognition tasks. It can be trained using a neuroimaging dataset collected for a specific mental illness to obtain a dedicated recognition model applicable to that specific mental illness.

[0166] To clearly illustrate the implementation of the present invention, the following description uses Major Depressive Disorder (MDD) as an example, that is, a model is trained using a multicenter depression image dataset to obtain a dedicated model for assisting in the identification of depression.

[0167] In one specific embodiment, the selected dataset is derived from the internationally publicly available multicenter depression imaging database REST-meta-MDD. After submitting an application and obtaining authorization, the dataset was used. This dataset integrates data from 2,428 subjects from 25 research centers in 17 hospitals in China, including 1,300 clinically diagnosed patients with major depressive disorder and 1,128 healthy controls. It covers subject groups at different stages of disease and medication status, and has good clinical representativeness and statistical reliability. The specific demographic information of this dataset is shown in Table 1.

[0168]

[0169] To verify the effectiveness of the proposed framework, a mental illness identification model was trained and evaluated on this dataset. The training of the mental illness identification model adopted a five-fold cross-validation strategy to ensure robustness of the results, and precision, recall, AUC and F1 score were used as core evaluation metrics.

[0170] Model performance comparison analysis: The mental illness identification model in this embodiment was compared with several existing baseline methods. The experimental results are shown in Table 2.

[0171] In single-modal classification tasks, the structural image model based on 3D ResNet-34 achieves a classification accuracy of 73.44%, which is better than traditional methods such as support vector machine (SVM, 63.97%).

[0172] In multimodal fusion classification, the fusion framework proposed in this invention (labeled as...) It achieved the best overall performance, with an accuracy of 77.76±1.02%, an AUC of 77.34±0.92%, and an F1 score of 79.92±1.91%, which is significantly better than baseline methods such as feature splicing.

[0173]

[0174] Cross-center generalization capability verification: To verify the model's ability to overcome the heterogeneity of multi-center data and achieve cross-site generalization, a leave-one-site verification experiment was designed. Specifically, sites 25, 1, 8, and 21 were selected as independent test sets, and the data from the remaining 24 sites were used to train the model. The performance difference before and after the adoption of the gradient inversion multi-task learning strategy (GRL-MTL) was compared (see Table 3).

[0175] Experimental results show that:

[0176] When the gradient reversal multi-task learning strategy is not enabled, the diagnostic performance of the model fluctuates significantly across different sites (accuracy ranges from 52.67% to 61.54%), indicating that the model is severely affected by the inter-site domain offset problem.

[0177] When the gradient reversal multi-task learning strategy was enabled, the model maintained stable and high diagnostic performance across all test sites (accuracy remained stable between 62.00% and 65.79%).

[0178] Therefore, based on the above results, it is proven that the gradient reversal multi-task learning strategy in this embodiment can effectively alleviate the heterogeneity bias of multi-center data, drive the model to learn the disease discrimination features that are station invariant, and thus significantly improve the model's cross-center generalization ability.

[0179]

[0180] Ablation experiment analysis: To explore the contribution of each core module, an ablation experiment was designed, and the results are summarized in Table 4.

[0181] The baseline model is a model that does not use the bidirectional cross-attention fusion mechanism and gradient reversal multi-task learning strategy, and its performance is low and highly volatile (accuracy 67.46±7.11%).

[0182] Enabling the bidirectional cross-attention fusion mechanism alone significantly improved performance and enhanced stability (accuracy 75.41±1.45%), indicating that the bidirectional cross-attention fusion mechanism can effectively establish cross-modal dynamic alignment.

[0183] While the performance was not optimal when the gradient reversal multi-task learning strategy was enabled alone, the model achieved the best performance (accuracy 77.76±1.02%) and the highest stability when both mechanisms were enabled simultaneously.

[0184] Therefore, based on the above results, it is demonstrated that the bidirectional cross-attention fusion mechanism focuses on mining deep cross-modal associations, while the gradient reversal multi-task learning strategy is dedicated to removing site-specific biases. The two work together to achieve further performance improvement.

[0185]

[0186] Feature Distribution Visualization Analysis: To further visualize the site correction effect, the features learned by the model are visualized using the t-SNE method, such as... Figure 4 As shown, Figure 4 To compare the distribution of fusion features before and after site heterogeneity treatment: Figure 4 (a) A t-SNE visualization of fusion features including site heterogeneity; Figure 4 (b) is a t-SNE visualization of the fusion features after removing site heterogeneity using the GRL-MTL strategy.

[0187] The fused features without site correction are scattered and mixed in the t-SNE dimensionality reduction plot, indicating that the features are severely affected by site differences. However, the features processed by the gradient inversion multi-task learning strategy show a clearer clustering trend, and the mixing of data points from different sites is reduced, which intuitively proves the effectiveness of the method in extracting site-invariant features.

[0188] Model decision interpretability analysis: The SHAP method was used to quantify the decision basis of the model and identify the brain region features that contribute the most to the classification. Some key results are summarized in Table 5.

[0189] The study found that the functional modality's contribution was mainly concentrated in the cerebellar vermis, insula, and anterior cingulate cortex, suggesting functional abnormalities in the emotion regulation and perception-integration circuits; while the structural modality's contribution was prominent in the globus pallidus, thalamus, and prefrontal cortex, suggesting structural changes related to motivation, information transfer, and higher cognition.

[0190] The above findings not only align with existing pathophysiological understanding of depression, but also demonstrate that the multimodal model constructed in this invention can capture neurobiological representations of different dimensions of the disease, enhancing the biological credibility and clinical interpretability of the model's diagnostic results.

[0191]

[0192] The comprehensive verification through the above embodiments shows that the mental illness identification model provided in this embodiment not only achieves superior diagnostic performance on multi-center datasets, but also has strong cross-site generalization ability and good decision interpretability, providing an effective technical solution for solving the clinical challenge of objective diagnosis of depression.

[0193] Example 2

[0194] like Figure 5 As shown, this embodiment provides a cross-site neuroimaging mental illness identification system 100 based on multimodal fusion, including:

[0195] The data acquisition module 101 is used to acquire multimodal neural image data and preprocess the neural image data to construct a multimodal neural image dataset.

[0196] The multimodal neuroimaging data includes structural imaging data and functional imaging data;

[0197] The model building module 102 is used to build the mental illness identification model by inputting the preprocessed multimodal neuroimaging data into the mental illness identification model to obtain site prediction results and disease prediction results.

[0198] The objective function module 103 is used to calculate the site classification loss and the disease classification loss based on the site prediction results and the disease prediction results, and to establish an objective function with the optimization objective of minimizing the disease classification loss and maximizing the site classification loss.

[0199] The model training module 104 is used to train a mental illness recognition model based on an objective function and a multimodal neural image dataset until a preset training cutoff condition is met, and output the trained mental illness recognition model.

[0200] The disease diagnosis module 105 is used to input the multimodal neuroimaging data of the subject to be diagnosed into the trained mental illness recognition model to obtain the final disease diagnosis result.

[0201] Example 3

[0202] This embodiment provides an electronic device, including at least one control processor and a memory for communicatively connecting to the at least one control processor;

[0203] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0204] The non-transitory software program and instructions required to implement the cross-site neuroimaging mental illness identification method based on multimodal fusion in the above embodiments are stored in memory. When executed by a processor, the cross-site neuroimaging mental illness identification method based on multimodal fusion in the above embodiments is executed, for example, the method described above is executed. Figure 1 Method steps S1 to S5;

[0205] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0206] Example 4

[0207] This embodiment provides a computer-readable storage medium storing computer-executable instructions for causing a computer to perform a cross-site neuroimaging method for identifying mental illnesses based on multimodal fusion as described in Embodiment 1.

[0208] It should be noted that the computer-readable storage medium in this embodiment may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof.

[0209] More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0210] In this embodiment, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this embodiment, the computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying a computer-readable program. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof.

[0211] The computer-readable signal medium may also be any computer-readable storage medium other than a computer-readable storage medium, which can send, propagate or transmit a program for use by or in connection with an instruction execution system, apparatus or device.

[0212] The computer program contained on the computer-readable storage medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof. The aforementioned computer-readable storage medium can be used to write a computer program for performing this embodiment in one or more programming languages ​​or combinations thereof. These programming languages ​​include object-oriented programming languages—such as Python, Java, and C++—and conventional procedural programming languages—such as C or similar programming languages. The program can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0213] The above description is merely a preferred embodiment of the present invention. It will be apparent to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects. The scope of the present invention is defined by the appended claims rather than the foregoing description, and thus all variations falling within the meaning and scope of the equivalents of the claims are intended to be included within the present invention.

Claims

1. A cross-site neuroimaging method for identifying mental illnesses based on multimodal fusion, characterized in that, include: Acquire multimodal neural imaging data and preprocess the neural imaging data to obtain a multimodal neural imaging dataset; The multimodal neuroimaging data includes structural imaging data and functional imaging data; A mental illness identification model was constructed, and preprocessed multimodal neuroimaging data was input into the mental illness identification model to obtain site prediction results and disease prediction results; Based on the site prediction results and disease prediction results, the site classification loss and disease classification loss are calculated. The objective function is established with the goal of minimizing the disease classification loss and maximizing the site classification loss. Based on the objective function and multimodal neuroimaging dataset, a mental illness recognition model is trained until the preset training cutoff condition is met, and the trained mental illness recognition model is output. The multimodal neuroimaging data of the subject to be identified is input into the trained mental illness identification model to obtain the final disease identification result.

2. The method according to claim 1, characterized in that, The construction of the mental illness identification model involves inputting preprocessed multimodal neuroimaging data into the model to obtain site prediction results and disease prediction results. Specifically, this includes the following steps: Construct a model for identifying mental illness; The mental illness identification model includes a two-stream feature extraction network, a feature fusion network, and a classifier network; The classifier network includes a site classifier and a disease classifier; Based on a dual-stream feature extraction network, feature extraction is performed on the preprocessed structural image data and functional image data respectively to obtain structural image feature representation and temporal feature representation; Based on the feature fusion network, bidirectional cross-attention fusion is performed on the structural image feature representation and the temporal feature representation to obtain a multimodal fused representation; The multimodal fusion representation is input into the site classifier to obtain the site prediction result; The multimodal fusion representation, structural image feature representation, and temporal feature representation are concatenated and then input into the disease classifier to obtain the disease prediction result.

3. The method according to claim 2, characterized in that, The dual-stream feature extraction network extracts features from the preprocessed structural image data and functional image data respectively to obtain structural image feature representation and temporal feature representation. Specifically, it includes the following steps: The preprocessed structural image data and functional image data are input into a two-stream feature extraction network; The dual-stream feature extraction network includes a structural image feature extraction branch based on the 3D ResNet algorithm and a temporal feature extraction branch based on the BiLSTM-Transformer algorithm; Based on the structural image feature extraction branch, hierarchical spatial feature extraction is performed on the preprocessed structural image data, and the extracted spatial features are then subjected to three-dimensional global average pooling and linear transformation to obtain a structural image feature representation. Based on the BiLSTM unit in the temporal feature extraction branch, the preprocessed functional image data is processed to obtain a preliminary temporal feature representation containing bidirectional temporal information. The preliminary temporal feature representation is input into the Transformer unit, and the effective time step outputs of the Transformer unit are aggregated and then processed by linear transformation to obtain the temporal feature representation.

4. The method according to claim 2, characterized in that, The feature fusion network performs bidirectional cross-attention fusion on structural image feature representation and temporal feature representation respectively to obtain multimodal fusion representation, specifically including the following steps: The structural image feature representation and temporal feature representation are input into the feature fusion network; Using temporal features as the query vector and structural image features as the key and value vectors, the first cross-attention is calculated to obtain the enhanced temporal feature representation; Using structural image features as query vectors and temporal features as key and value vectors, a second cross-attention is calculated to obtain enhanced structural image feature representations. Based on the enhanced temporal feature representation and the enhanced structural image feature representation, a first fusion weight and a second fusion weight are calculated respectively; based on the first fusion weight and the second fusion weight, the enhanced temporal feature representation and the enhanced structural image feature representation are weighted and summed to obtain a multimodal fusion representation.

5. The method according to claim 1, characterized in that, Based on the site prediction results and disease prediction results, site classification loss and disease classification loss are calculated. The objective function is established with the goal of minimizing the disease classification loss and maximizing the site classification loss. This process includes the following steps: Based on the predicted site results and the actual site labels, the site classification loss is calculated using the cross-entropy loss function. Based on the disease prediction results and the actual disease labels, the disease classification loss is calculated using the cross-entropy loss function. Based on site classification loss and disease classification loss, an objective function is established with the optimization objective of minimizing the disease classification loss and maximizing the site classification loss; The expression for the objective function is: ； In the formula, Total loss; For disease classifier parameters; For site classifier parameters; Loss due to disease classification; Losses are categorized by site; This is the loss weight.

6. The method according to claim 2, characterized in that, The process of training a mental illness recognition model based on an objective function and a multimodal neuroimaging dataset until a preset training cutoff condition is met, and then outputting the trained mental illness recognition model, specifically includes the following steps: A1: Initialize the parameters of the mental illness identification model; A2: The mental illness recognition model is trained based on the multimodal neuroimaging dataset; A3: Repeat step A2 until the mental illness recognition model reaches the preset convergence condition, and obtain the trained mental illness recognition model; Step A2 specifically includes the following steps: The data from the multimodal neuroimaging dataset is input into the mental illness recognition model, and then passed through the dual-stream feature extraction network, feature fusion network, and classifier network for forward propagation to obtain site prediction results and disease prediction results. Based on the site prediction results and the disease prediction results, the total loss is calculated using an objective function, and backpropagation is performed to update the parameters of the mental illness identification model. During the backpropagation process, a gradient reversal strategy is executed on the gradients passed to the site classifier, and the parameters of the two-stream feature extraction network and the feature fusion network are updated using the reversed gradients, so that the identification model learns domain-invariant features to eliminate site specificity among multi-site data.

7. A cross-site neuroimaging system for identifying mental illnesses based on multimodal fusion, characterized in that, include: The data acquisition module is used to acquire multimodal neural image data and preprocess the neural image data to construct a multimodal neural image dataset. The multimodal neuroimaging data includes structural imaging data and functional imaging data; The model building module is used to build a mental illness identification model. It inputs preprocessed multimodal neuroimaging data into the mental illness identification model to obtain site prediction results and disease prediction results. The objective function module is used to calculate the site classification loss and the disease classification loss based on the site prediction results and the disease prediction results, with the optimization objective being to minimize the disease classification loss and maximize the site classification loss. The model training module is used to train a mental illness recognition model based on an objective function and a multimodal neuroimaging dataset until the preset training cutoff condition is met, and then output the trained mental illness recognition model. The disease identification module is used to input the multimodal neuroimaging data of the subject to be identified into the trained mental illness identification model to obtain the final disease identification result.

8. An electronic device, characterized in that, include: At least one processor; At least one memory storing computer-executable instructions, wherein, when executed by the at least one processor, the computer-executable instructions cause the at least one processor to perform the cross-site neuroimaging mental illness identification method based on multimodal fusion as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor causes the at least one processor to perform the cross-site neuroimaging method for identifying mental disorders based on multimodal fusion as described in any one of claims 1 to 6.

10. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by at least one processor, the at least one processor causes the processor to perform the cross-site neuroimaging method for identifying mental disorders based on multimodal fusion as described in any one of claims 1 to 6.

Images