A deep interpretable multi-modal discriminative representation method

Abstract

The application discloses a kind of deep explainable multimodal discriminative representation method.Deep explainable multimodal discriminative representation network model is proposed in the application, the method aligns heterogeneous modal in discriminative common space to improve classification accuracy, and provides a sensitive feature analysis idea.Specifically, the application uses multilayer perceptron as an encoder, combines self-representation constraints embedded with structural information and generalized canonical correlation analysis constraints, maps multimodal data into discriminative common space, and improves recognition performance.In addition, the application decouples sensitive features by compressing complex networks through knowledge distillation to enhance the explainability of the model.

Description

Technical Field

[0001] This invention presents a deep interpretable multimodal discriminant representation method, which relates to the field of computer-aided diagnosis and is mainly used to determine the stage of Alzheimer's disease. It plays an important role in disease prevention and precision treatment. Background Technology

[0002] Alzheimer's disease is a progressive neurodegenerative disease that commonly affects the elderly, manifesting as motor impairment, memory decline, and language difficulties. Because Alzheimer's disease is irreversible, intervention is only possible in its early stages to slow its progression. Mild cognitive impairment is a prodromal phase of Alzheimer's disease, characterized by a decline in cognitive abilities. Neuroimaging is widely used in early diagnosis, revealing structural or functional damage to the brain and allowing us to differentiate patients from healthy individuals based on the extent of brain lesions.

[0003] In recent years, machine learning methods have received increasing attention in the early diagnosis of Alzheimer's disease. Early research focused on modeling single modalities, such as magnetic resonance imaging (MRI) or positron emission tomography (PET). However, single-modality models are insufficient to reveal the complex structural and functional changes in the brain, while multimodal data can supplement more potential complementary information from different perspectives. Due to the high-dimensional nonlinearity of neural images, shallow models cannot effectively utilize the hierarchical and structural information between different modalities. Deep learning, by separating complex patterns in the input, can classify more effectively than shallow structures and has advantages in inferring and modeling the hierarchical structure of brain networks. However, existing deep learning-based early diagnostic methods neglect the structural information embedded in multimodal data. Structural information can serve as important prior knowledge, allowing for the selection of more discriminative features by rationally utilizing the differences and commonalities between samples. Furthermore, the interpretability of diagnostic models has always been a concern in the medical field. In neural networks, due to the involvement of activation functions, network weights and inputs cannot be analyzed independently, resulting in poor clinical interpretability of deep learning-based diagnostic models. Existing deep models are unable to provide information on the influence of brain regions in diagnosis, and thus offer limited clinical guidance. Summary of the Invention

[0004] Purpose of the invention: To address the problems of neglecting sample discriminative structure information and low model interpretability in existing methods, this invention provides a deep interpretable multimodal discriminative representation method that combines discriminative structure to improve diagnostic accuracy and enhance model interpretability.

[0005] Technical solution: A deep, interpretable multimodal discriminant representation method, the specific steps of which are as follows:

[0006] Step 1: Data Preprocessing and Network Initialization

[0007] For the acquired multimodal data, imaging measurements of relevant brain regions are extracted based on the brain region template. The i-th modality data is defined as X. i The label is defined as Y. Furthermore, the network model parameters are initialized according to a normal distribution.

[0008] Step 2: Reconstructing multimodal data

[0009] Modal data X i The hierarchical representation f is reconstructed using the corresponding multilayer perceptron. i (X i It simulates the hierarchical structure of the brain.

[0010] Step 3: Inter-modal and intra-modal sample alignment

[0011] In multimodal data, although the same sample may exhibit different data representations in different modalities, the relationships between samples remain consistent. For example, similar samples in modality 1 should also be similar in modality 2. This invention introduces self-representation learning, whereby a sample in a subspace can be represented by a linear combination of other samples. The self-representation coefficients Z capture the similarity of samples within a modality and share this information across modalities to maintain consistency in sample similarity between different modalities. The multilayer perceptron can then construct a low-dimensional common subspace under the constraint of shared self-representations. This invention further applies block diagonal constraints to the shared self-representation coefficients to achieve alignment between and within modalities. Due to fluctuations in the similarity of intra- and inter-class samples, the self-representation coefficients will have a diagonal block structure when input samples are sorted by category. We employ block diagonal constraints embedded with semantic information to strengthen the diagonal block structure. The mutual representation of intra-class samples is enhanced, while the mutual representation of inter-class samples is suppressed. This prompts the multilayer perceptron to generate a differentiated common subspace and obtain a differentiated coordinated representation Zf. i (X i ).

[0012] Step 4: Generate a common discriminant space

[0013] In multimodal research, heterogeneity is a common problem. Since different modalities reside in different representation spaces with significant differences, directly classifying heterogeneous modalities leads to poor classification performance and low robustness. Therefore, it is necessary to map multimodal data to a common space before classification. This invention uses optimized generalized canonical correlation analysis to address this problem, improving the Zf... i (X iThis is further mapped to a common space. Generalized Canonical Correlation Analysis (GCCA) is an extension of canonical correlation analysis that maps three or more views to a common space while maximizing the correlation coefficients between the views.

[0014] Step 5: Predict the stage of the disease

[0015] Discriminative self-representation constraints and GCCA constraints together form the loss function to facilitate the generation of a discriminative common space by the multilayer perceptron. We will use the multimodal common subspace f obtained during training... i (X i SVM was used for classification to predict the stage of the disease.

[0016] Step Six: Extract Interpretable Weights

[0017] To improve the interpretability of the model, this invention proposes a linear multimodal knowledge distillation module. This module is used to learn collaborative representations under multimodal conditions, compressing the complex nonlinear mapping relationships in the multimodal network into direct linear relationships. The mean square error between the collaborative representation and the distillation representation is used as the model loss to train the model and extract the mapping weights.

[0018] Beneficial Effects: This invention addresses the problem of brain disease diagnosis by proposing a diagnostic method for Alzheimer's disease based on deep, interpretable multimodal discriminant representations. Compared to existing technologies, this invention offers the following technical advantages:

[0019] 1) This invention takes into account the inherent similarity of samples and uses self-representation learning in multimodal deep networks to reduce the non-uniformity in sample representations, thereby promoting more effective feature selection.

[0020] 2) This invention effectively combines the discriminative structure information of samples, implements diagonal block constraints on the self-representation coefficients, realizes sample alignment within and between modes, promotes the generation of discriminative models, and improves classification performance.

[0021] 3) To improve the interpretability of the model, this invention uses a linear multimodal knowledge distillation module to capture the direct correlation between brain regions and diagnoses. Attached Figure Description

[0022] Figure 1 This is a general framework diagram of an Alzheimer's disease diagnostic method based on deep interpretable multimodal discriminant representation;

[0023] Figure 2 This is a flowchart of the deep multimodal discrimination representation module;

[0024] Figure 3 This is the flowchart for the linear multimodal knowledge distillation module; Detailed Implementation

[0025] The present invention will now be described in further detail with reference to the accompanying drawings.

[0026] Figure 1 The overall framework of the model is presented. This invention can be divided into two modules: a deep multimodal discriminative representation module and a linear multimodal knowledge distillation module, which are used to solve the problems of the model ignoring sample discriminative structural information and low interpretability, respectively.

[0027] Figure 2 The flowchart of the deep multimodal discriminative representation module is shown, and its specific experimental steps are as follows:

[0028] 1) Data preprocessing and network initialization

[0029] For the acquired multimodal data, imaging measurements of relevant brain regions are extracted based on brain region templates. Each modality is defined as follows: s represents the number of samples, d i Let be the number of features of mode i, and n be the number of modes. The label is defined as Y∈R. s×c c represents the disease stage. Furthermore, the network model parameters are initialized, including the multilayer perceptron parameters θ. i The self-expression coefficient Z follows a normal distribution.

[0030] 2) Training the deep multimodal discriminative representation module

[0031] This module consists of three parts: an encoder, a discriminative self-representation constraint, and a generalized canonical correlation analysis constraint. This invention uses a multilayer perceptron as the encoder, and the modal data X... i The hierarchical representation f is reconstructed using the corresponding multilayer perceptron. i (X i It simulates the hierarchical structure of the brain.

[0032] Due to the self-representation properties within a mode and the similarity of sample representations between modes, we introduce shared sparse self-representation coefficients Z∈R. s×s In our fusion model, preliminary sample alignment is achieved. The self-representation property means that data points in the same subspace can be represented as a linear combination of other points. Based on this property, the following formula is proposed:

[0033] Zf i (X i )=f i (X i stdiag(Z) = 0

[0034] Z reflects the similarity between samples. Furthermore, to avoid trivial solutions Z = 1, we set the constraint diag(Z) = 0.

[0035] When multimodal data is input into the model in class order, a diagonal block structure appears in the shared self-representation coefficients, with each diagonal block corresponding to a class. To address this, we embed supervision information into the shared self-representation coefficients and impose diagonal block constraints, ensuring that only samples of the same class can represent each other, thus further enhancing intra-modal and inter-modal sample alignment. We define the out-of-class matrix as:

[0036]

[0037] The shared sparse self-representation and its diagonal block constraint together constitute the loss function of the self-representation layer:

[0038]

[0039] in Let denote the Frobenius norm of the matrix, and ⊙ denote the dot product. Specifically, the first term preserves the sparsity of Z, the second term guides Z to have shared self-representation capabilities, and finally, the third term makes Z discriminative by minimizing the out-of-block self-representation coefficients.

[0040] This invention employs generalized canonical correlation analysis (GCCA) as a constraint on the representation space, projecting each modality into a common space, thereby integrating the latent information in multimodal data. We define the objective function of GCCA as:

[0041]

[0042] in It is a template for public spaces.

[0043] We further optimize the objective function of GCCA, setting the output of the i-th modality to M. i =Zf i (X i ;θ i )∈R s ×d′ In addition, M i T M i It is called the scattering matrix, M i (M i T M i ) -1 M i T Let r be the projection matrix. We can add r to the scattering matrix. i I is used for regularization, and the projection matrix of the regularization is expressed as:

[0044]

[0045] We define the sum of the projection matrices as The common space template G can be computed as the left singular vector of the rank-m SVD decomposition on Q:

[0046]

[0047] The objective function equation of GCCA can be further expressed as:

[0048]

[0049] Where m is the required rank of SVD, and n is the number of modes. Furthermore, minimizing the GCCA objective is equivalent to maximizing Tr(G T QG), Tr(G) T QG) is the sum of the first m largest singular values ​​of Q. The objective function of GCCA can ultimately be expressed as:

[0050]

[0051] Finally, we combine the loss function of the self-expression layer and the optimized GCCA constraint to form the final loss function, which is used to train the deep multimodal discriminative representation module, and we use stochastic gradient descent to optimize the network model parameters.

[0052]

[0053] 3) Diagnosis during the disease stage

[0054] This invention uses a support vector machine to represent the common subspace f under each modality. i (X i The parameters are classified and grid search is used to find the optimal parameters.

[0055] Figure 3 The flowchart of the linear multimodal knowledge distillation module is shown, and its specific experimental steps are as follows:

[0056] 1) Training the linear multimodal knowledge distillation module

[0057] In this invention, each modality has a corresponding distillation module. Since increasing the depth of a linear multilayer perceptron does not increase its expressive power, this module consists of a single-layer linear mapping, compressing complex weights and preserving the latent relationships learned from the multimodalities. Its structure is consistent with the input and output layers of the multilayer perceptron in the deep multimodal discriminative representation module. Training is performed using the following loss function, and stochastic gradient descent is also used to optimize the network model parameters:

[0058]

[0059] In the formula, f i (X i) is a collaborative representation under deep multimodal discriminant representation, f i ′(X i ) is the representation under the linear multimodal knowledge distillation module, and MSE is the mean squared error.

[0060] 2) Diagnosis during the disease stage

[0061] This invention uses a support vector machine to represent the common subspace f under each modality. i (X i The parameters are classified and grid search is used to find the optimal parameters.

[0062] 3) Analysis of the importance of brain regions

[0063] The feature mapping weights under the distillation module are multiplied by the feature scores in the support vector machine, and the absolute value is calculated as the influence score of the brain region to analyze which brain regions play an important role in the process of diagnosing diseases.

[0064] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of relevant knowledge in the art, various changes can be made without departing from the spirit of the present invention.

Claims

1. A deep, interpretable multimodal discriminant representation method, characterized in that, Includes the following steps: (1) Preprocess the multimodal data and initialize the deep network parameters, including extracting relevant brain region imaging measurement values ​​based on the brain region template; Each modal data is defined as , , Represented as the number of samples, Represented as modality The number of features, For modal number, the label is defined as , The disease stage; Multilayer perceptron parameters in deeply interpretable network models With shared sparse self-representation coefficients Follows a normal distribution; (2) Modal data of each heterogeneous system The hierarchical representation is reconstructed using the corresponding multilayer perceptron. It is used to simulate the hierarchical structure of the brain; (3) Using discriminative self-representation learning with diagonal block constraints to align intra-modal and inter-modal samples of the reconstructed multimodal data, and calculating the discriminative self-representation loss; this step includes using self-representation learning and implementing diagonal block constraints to achieve intra-modal and inter-modal sample alignment, the specific steps of which include: Introducing shared sparse self-representation coefficients in the fusion model To achieve preliminary sample alignment, self-representation features refer to the ability of data points in the same subspace to be represented as linear combinations of other points, expressed mathematically as: Among them, the shared sparse self-representation coefficients To reflect the similarity between samples, constraints are set to avoid trivial solutions. ; Further embedding supervisory information into the shared sparse self-representation coefficients The system is then configured to represent samples within and between modalities by imposing diagonal block constraints, allowing only samples of the same class to represent each other, thereby enhancing intramodal and intermodal sample alignment. Define the out-of-class matrix as: Shared sparse self-representation coefficients Together with the diagonal block constraints, they constitute the loss function of the self-representation layer: in, Denotes the Frobenius norm of a matrix. The above expression represents the dot product, with the first term used to preserve the shared sparse self-representation coefficients. The sparsity of the second term is used to guide the sharing of sparse self-representation coefficients. It possesses shared self-representation capabilities, and the third term enables shared sparse self-representation coefficients by minimizing out-of-block self-representation coefficients. It has discriminative properties; (4) By applying the generalized canonical correlation analysis, the correlation coefficient maximization constraint is applied to the aligned multimodal data, and the multimodal data is mapped to the common space. The loss of the generalized canonical correlation analysis is calculated as follows: Generalized canonical correlation analysis is used as a constraint on the representation space to project each modality into a common space, thereby integrating the latent information in multimodal data. Its objective function is defined as: Among them, public space templates It is a public space template; Further optimize the objective function of the generalized canonical correlation analysis, setting the output of the i-th mode as... ; Here is the scattering matrix. The projection matrix is ​​added through the scattering matrix. To perform regularization, the projection matrix of the regularization is represented as: The sum of the projection matrices is defined as Public space template Can be calculated as The left singular vector of the upper rank-m SVD decomposition is represented as: The objective function equation for generalized canonical correlation analysis is expressed as: in It is the required rank of SVD. It is the modal number; Furthermore, minimizing the objective of generalized canonical correlation analysis is equivalent to maximizing , It is the sum of the first m largest singular values ​​of Q. The objective function of generalized canonical correlation analysis is ultimately expressed as: (5) The discriminative self-representation loss and the generalized canonical correlation analysis loss are used as the total loss of the multimodal discriminative representation module, and the gradient stochastic descent method is used to optimize the model parameters; (6) Use support vector machines to classify the common subspace obtained from training, in order to predict the stage of the disease; (7) Using knowledge distillation, a linear mapping model is used to learn the collaborative representation of each modality in the common space. The mean square error of the linear representation and the collaborative representation is used as the loss function of the multimodal interpretable module. This includes using a linear multimodal knowledge distillation module to capture the direct correlation between features and labels. The specific process is as follows: Each modality has a corresponding distillation module, which consists of a single-layer linear mapping that compresses complex weights and preserves the latent relationships learned from the multimodal model. Its structure is consistent with the input and output layers of the multilayer perceptron in the deep multimodal discriminative representation module. The network model is trained using the following loss function, and stochastic gradient descent is also used to optimize the network model parameters: ， In the formula, It is a deep multimodal discriminant representation. yes The distillation is represented by MSE, which is the mean square error. (8) Use the gradient stochastic descent method to optimize the model parameters and compress the complex weights of the multilayer perceptron for analysis of sensitive features.

Images