A Multimodal Brain Image Feature Learning Method

Abstract

The present invention discloses a multi-modal brain image feature learning method, and its steps include: respectively preprocessing multi-modal brain image data to obtain a feature matrix; calculating the correlation coefficient between the features of each modal data to obtain a feature correlation matrix, and performing linear fusion to obtain feature correlation regularization; performing weighted fusion on the feature matrix, calculating the adjacent nodes of the features to obtain an adjacency matrix, constructing a feature graph Laplacian matrix, and obtaining feature structure regularization; embedding the two regularizations into a multi-task model introducing low-rank constraints to perform feature learning, and screening feature vectors with good representations; subsequently performing linear fusion, training the training set using a support vector machine, and using the test set to test the classification performance of the model. The present invention learns the potential relationships between multi-modal brain image features and the local spatial geometric structure of the features, screens associated features, and improves the classification performance while enhancing the interpretability of feature learning.

Description

Technical Field

[0001] The present invention relates to the field of biomedical information technology, and in particular, to a multi-modal brain image feature learning method. Background Art

[0002] The brain is the most complex organ of the human body and one of the most complex systems in the world so far. The operating mechanism of the human brain has not been fully understood yet. Nowadays, brain science research aiming at studying various aspects of the brain has gradually become the core content of natural science research. The development of brain science research enables humans to understand the brain more comprehensively. In recent years, with the continuous development of brain imaging technology, a series of non-invasive methods have been provided to obtain brain information. Commonly used brain imaging technologies can be divided into functional and structural imaging technologies. In the research on exploring brain thinking and cognitive states, machine learning has been widely used in brain image-related research. Since it can automatically analyze and obtain laws from data, and use the laws to predict unknown data and assist in finding brain network features that may be sensitive to diseases, the brain image feature learning method based on machine learning has become a hot research topic in brain image research, attracting more and more researchers.

[0003] In practice, doctors can obtain a large number of brain images with different structures and functions, which can assist doctors in diagnosing the subjects from different angles and strengthening the understanding of disease-causing factors. Traditional single-modal brain images only start from one perspective, obviously ignoring the information complementarity between different modal brain images. This will inevitably lead to insufficient features obtained and affect the final disease diagnosis. In multi-modal brain image feature learning, the most important point is to perform joint feature learning on the features extracted from multiple brain images, screen out the features associated with diseases, and improve the classification performance while reducing the feature dimension. In existing machine learning methods, multi-task learning is often used for feature learning related to diseases. Its advantage lies in being able to reveal the potential common characteristics between different task data, sharing information between tasks, and having good generalization.

[0004] Multi-modal brain image feature learning can be divided into three ways: filtering, wrapping, and embedding. Embedded feature learning is widely used. By combining the learner with the feature learning process, feature learning is automatically completed when the learner is learning. Regularization technology is often used in embedded feature learning. In the existing embedded feature learning methods, feature selection is not sufficient when performing multi-modal feature selection based on samples. Summary of the Invention

[0005] The purpose of the present invention is to solve the deficiencies of traditional multi-modal feature learning, and provides a new multi-modal brain image feature learning method, adopting the following technical solutions:

[0006] A multi-modal brain image feature learning method, the steps of which include:

[0007] Step 1: Select the structural magnetic resonance imaging (sMRI) and positron emission tomography (PET) of the same subject from the sample set. Extract the average gray matter density of the brain region of interest from the structural magnetic resonance imaging as the feature of the structural magnetic resonance imaging, and extract the glucose metabolism of the brain region of interest from the positron emission tomography as the feature of the positron emission tomography;

[0008] Step 2: Calculate the correlation coefficient between the features of each modality data respectively to obtain the corresponding feature correlation matrix, and perform linear fusion to obtain feature correlation regularization;

[0009] Step 3: Perform weighted fusion on the feature matrix, calculate the adjacent nodes of the features to obtain the adjacency matrix, and construct the feature graph Laplacian matrix according to the cosine distance method to obtain feature structure regularization;

[0010] Step 4: Embed the feature correlation regularization and the feature structure regularization into a multi-task model introducing low-rank constraints for feature learning to obtain a feature learning model;

[0011] Step 5: Screen out the feature vectors with good representation through the feature learning model, perform standardization respectively, and perform linear fusion on the features extracted from the multi-modal data to obtain a fused feature matrix;

[0012] Step 6: Divide the fused feature matrix of the sample set into a test set and a training set, use the training set to train the support vector machine and generate a model, use the test set to test the classification performance of the model, and finally calculate the classification performance index; classify the feature vectors using the trained model.

[0013] Further, the specific content of Step 1 is as follows:

[0014] Step 1.1: Extract the average gray matter density from the structural magnetic resonance imaging: Perform spatial standardization on the original image of the structural magnetic resonance imaging using a standard brain template, make each region of the original image correspond one by one with the template region, and then segment the image into three brain tissue structures: gray matter, white matter, and cerebrospinal fluid; finally, use the brain partition template to extract the average gray matter density of the region of interest as the feature of the structural magnetic resonance imaging;

[0015] Step 1.2: Extract the average glucose metabolism from positron emission tomography (PET) imaging: Perform head motion correction on the original PET imaging, register it to a standard brain template for normalization and smoothing operations, and finally use a brain parcellation template to extract the average glucose metabolism of brain regions as the PET imaging feature.

[0016] Further, the specific steps of Step 2 are as follows:

[0017] Calculate the correlation coefficients between the feature data of each modality respectively, and define the matrix Record the correlation coefficient matrix of the i-th modality, where p represents the number of features; obtain the feature correlation regularization through linear fusion, and its calculation formula is:

[0018]

[0019] In the formula, tr(·) represents the trace norm of the matrix, represents the correlation matrix after linear fusion, where m represents the number of modalities, and then represents the feature weight matrix.

[0020] Further, the specific steps of Step 3 are as follows:

[0021] Step 3.1: Perform weighted fusion on the feature matrices, specifically: Normalize the feature matrix X1 of structural magnetic resonance imaging (sMRI) and the feature matrix X2 of PET imaging respectively, and then perform weighted fusion to obtain the fused feature matrix X F , and its weighted fusion formula is:

[0022] X F = δ1X1 + δ2X2 (2)

[0023] In the formula, X1 and X2 are feature matrices, and δ1 and δ2 are weighted fusion coefficients;

[0024] Step 3.2: According to the fused feature matrix X F , calculate the adjacent feature nodes, and then use the cosine distance method to construct the adjacency matrix H, and its calculation method is:

[0025]

[0026] In the formula, h ij is used to measure the similarity between the i-th column feature vector and the j-th column feature vector in the feature matrix X, X ·i and X ·j represent the i-th column and the j-th column feature vectors in the feature matrix X respectively;

[0027] Step 3.3: According to the adjacency matrix H calculated in Step 3.2, calculate the degree matrix S, and further construct the Laplacian matrix L based on weighted fusion, F , and its calculation method is shown in Equation (4):

[0028] L F = S - H (4)

[0029] In the formula, is the degree matrix, and it is a diagonal matrix, and the main diagonal elements are the degrees of each feature in the adjacency matrix;

[0030] Step 3.4: Through the obtained Laplacian matrix L F , further obtain the feature structure regularization that maintains the local spatial geometric structure of the features, and its calculation formula is shown in Equation (5):

[0031] tr(W T L F W) (5)

[0032] In the formula, is the feature weight matrix, and tr(·) represents the trace norm of the matrix.

[0033] Furthermore, the specific content of Step 4 is as follows:

[0034] Use the trace norm to approximate the low-rank constraint, introduce the trace norm on the basis of the multi-task learning model to improve the information sharing between different modality data, and introduce the feature correlation regularization mentioned in Equation (1) and the feature structure regularization mentioned in Equation (5). Finally, a multi-modal brain image feature learning model based on the fusion of feature correlation and feature structure is obtained:

[0035]

[0036] In the formula, Y i represents the label of the i-th modality, X i is the feature matrix of the i-th modality, and each row element in w i represents the corresponding feature weight value in the i-th modality. α, β, and γ are regularization parameters and are all real numbers greater than zero.

[0037] Furthermore, the specific content of Step 5 is as follows:

[0038] Step 5.1: Through the feature learning model of Equation (6), learn the feature weights of the multi-modal brain images to obtain the feature weight matrix W, and screen the features according to the weight matrix;

[0039] Step 5.2: Linearly fuse the features of the two-modal brain images after screening, that is, splice multiple feature matrices together to obtain the fused feature matrix.

[0040] Further, step 6 is specifically as follows:

[0041] Partition the fused feature matrix into a test set and a training set, use a linear support vector machine for model training, use the test set data to verify the model classification performance, and determine whether the specified number of iterations is reached; if not, re-execute step 6; if so, select the feature vector corresponding to the best classification result for output.

[0042] Further, the method further includes the following steps:

[0043] Step 7: Visualize the features corresponding to the selected key brain regions for analyzing the key brain regions affected by the disease.

[0044] The present invention has the following beneficial effects:

[0045] The present invention is a multi-modal brain image feature learning method based on feature correlation and feature structure fusion. Feature correlation regularization and feature structure regularization are embedded into a multi-task model introducing low-rank constraints for feature learning, fully considering the potential relationships between multi-modal brain image features and the local spatial geometric structure of features, screening features associated with diseases, overcoming the problem of insufficient feature information volume, and being able to improve the interpretability of feature learning while improving classification performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 is a flowchart of the implementation of a multi-modal brain image feature learning method of the present invention; DETAILED DESCRIPTION OF THE EMBODIMENTS

[0047] In order to make the objectives, technical solutions and advantages of the present invention clearer and more understandable, the present invention will be further described in detail below with reference to the drawings and specific embodiments.

[0048] Please refer to Figure 1 , the present invention is a multi-modal brain image feature learning method, and its steps include:

[0049] Step 1: Select the average gray matter density of the brain regions of interest from the structural magnetic resonance imaging and positron emission tomography imaging of the same subject in the sample set as the feature of the structural magnetic resonance imaging; extract the glucose metabolism of the brain regions of interest as the feature of the positron emission tomography imaging; in this example, the sample set includes 73 normal subjects, 53 patients with early mild cognitive impairment, 49 patients with late mild cognitive impairment, and 69 Alzheimer's disease patients; step 1 specifically includes:

[0050] Step 1.1: Extract the average gray matter density from the structural magnetic resonance imaging:

[0051] Step 1.1a: Perform spatial normalization on the original images of structural magnetic resonance imaging using the MNI152 standard brain template, making each region of the original images correspond one by one with the template regions, and then segment the images into three brain tissue structures: gray matter, white matter, and cerebrospinal fluid;

[0052] Step 1.1b: Finally, use the AAL (Anatomical Automatic Labeling) template to extract the average gray matter density of the regions of interest as the features of structural magnetic resonance imaging;

[0053] Step 1.2: Extract the average glucose metabolism from positron emission tomography imaging:

[0054] Step 1.2a: Perform head motion correction on the original images of positron emission tomography imaging, register them to the MNI152 standard brain template for standardization and smoothing operations;

[0055] Step 1.2b: Finally, use the AAL template to extract the average glucose metabolism of the brain regions as the features of positron emission tomography imaging.

[0056] Step 2: Calculate the correlation coefficients between the feature data of each modality respectively, and define the matrix where p represents the number of features; record the correlation coefficient matrix of the i-th modality, and obtain the feature correlation regularization through linear fusion. The calculation formula is:

[0057]

[0058] In the formula, tr(·) represents the trace norm of the matrix, represents the correlation matrix after linear fusion, where m represents the number of modalities, then represents the feature weight matrix;

[0059] Step 3: Perform weighted fusion on the feature matrix, calculate the adjacent nodes of the features to obtain the adjacency matrix, and construct the feature graph Laplacian matrix according to the cosine distance method to obtain the feature structure regularization; the specific steps are as follows:

[0060] Step 3.1: Perform weighted fusion on the feature matrix, specifically: standardize the feature matrix X1 of structural magnetic resonance imaging and the feature matrix X2 of positron emission tomography imaging respectively, and then perform weighted fusion to obtain the fused feature matrix X F , and its weighted fusion formula is:

[0061] X F =δ1X1 + δ2X2 (8)

[0062] In the formula, X1 and X2 are feature matrices, and δ1 and δ2 are weighted fusion coefficients;

[0063] Step 3.2: According to the fused feature matrix X F , calculate the adjacent feature nodes, and then construct the adjacency matrix H using the cosine distance method. The calculation method is as follows:

[0064]

[0065] In the formula, h ij is used to measure the similarity between the i-th column feature vector and the j-th column feature vector in the feature matrix X, and X ·i and X ·j represent the i-th column and the j-th column feature vectors in the feature matrix X respectively;

[0066] Step 3.3: According to the adjacency matrix H calculated in Step 3.2, calculate the degree matrix S, and further construct the Laplacian matrix L based on weighted fusion F , and its calculation method is shown in formula (10):

[0067] L F = S - H (10)

[0068] In the formula, is the degree matrix and is a diagonal matrix, and the main diagonal elements are the degrees of each feature in the adjacency matrix;

[0069] Step 3.4: Through the obtained Laplacian matrix L F , further obtain the feature structure regularization that preserves the local spatial geometric structure of the features, and its calculation formula is shown in formula (11):

[0070] tr(W T L F W) (11)

[0071] In the formula, is the feature weight matrix, and tr(·) represents the trace norm of the matrix;

[0072] Step 4: Use the trace norm to approximate the low-rank constraint, introduce the trace norm on the basis of the multi-task learning model to improve the information sharing between different modality data, and introduce the feature correlation regularization mentioned in formula (7) and the feature structure regularization mentioned in formula (11). Finally, obtain a multi-modal brain image feature learning model based on the fusion of feature correlation and feature structure:

[0073]

[0074] In the formula, Y i represents the label of the i-th modality, X i is the feature matrix of the i-th modality, w iEach element in each row represents the corresponding feature weight value in the i-th modality. α, β, and γ are regularization parameters, and all are real numbers greater than zero;

[0075] Step 5: Screen the feature vectors with good representations through the feature learning model, perform standardization respectively, and linearly fuse the features extracted from the multi-modal data to obtain the fused feature matrix, specifically as follows:

[0076] Step 5.1: Through the feature learning model of formula (12), learn the feature weights of the multi-modal brain images to obtain the feature weight matrix W, and screen the features according to the weight matrix;

[0077] Step 5.2: Linearly fuse the features of the two modalities of brain images after screening, that is, splice multiple feature matrices together to obtain the fused feature matrix;

[0078] Step 6: Divide the fused feature matrix into a test set and a training set, and use a linear support vector machine for model training, where the kernel function type is set to a linear kernel. Subsequently, use the test set data to verify the classification performance of the model; in addition, specify the maximum number of iterations as 10 times to balance the classification performance and eliminate accidental factors.

[0079] In some other embodiments, the following steps are further included after Step 6:

[0080] Step 7: Visualize the key brain regions corresponding to the screened features for analyzing the key brain regions affected by the disease.

[0081] The parts not involved in the present invention are the same as the prior art or are implemented by the prior art.

[0082] The above content is a further detailed description of the present invention in combination with specific implementation manners. It cannot be determined that the specific implementation of the present invention is only limited to these descriptions. For those of ordinary skill in the technical field to which the present invention belongs, without departing from the concept of the present invention, several simple deductions or substitutions can be made, which should all be regarded as belonging to the protection scope of the present invention.

Claims

1. A multi-modal brain image feature learning method, characterized in that, The steps include: Step 1: Select the structural magnetic resonance imaging (sMRI) and positron emission tomography (PET) imaging of the same subject from the sample set. Extract the average gray matter density of the brain region of interest from the sMRI as the feature of the sMRI, and extract the glucose metabolism of the brain region of interest from the PET imaging as the feature of the PET imaging. Step 2: Calculate the correlation coefficients between the features of each modality data respectively to obtain the corresponding feature correlation matrix, and perform linear fusion to obtain feature correlation regularization. Specifically: Calculate the correlation coefficients between the feature data of each modality respectively, and define the matrix For \(i = 1, 2, 3, \cdots, m\), record the correlation coefficient matrix of the \(i\)-th modality, where \(p\) represents the number of features; Feature correlation regularization is obtained through linear fusion, and its calculation formula is: where tr(·) represents the trace norm of a matrix, represents the correlation matrix after linear fusion, where m represents the number of modalities, denotes the feature weight matrix; Step 3: Perform weighted fusion on the feature matrix, calculate the adjacent nodes of the features to obtain the adjacency matrix, and construct the feature graph Laplacian matrix according to the cosine distance method to obtain feature structure regularization. Specifically: Step 3.1: Perform weighted fusion on the feature matrices, specifically: standardize the feature matrix X1 of structural magnetic resonance imaging and the feature matrix X2 of positron emission tomography imaging respectively, and then perform weighted fusion to obtain the fused feature matrix X F , and its weighted fusion formula is: X F = δ1X1 + δ2X2 (2) In the formula, X1 and X2 are feature matrices, and δ1 and δ2 are weighted fusion coefficients. Step 3.2: According to the fused feature matrix X F , calculate the adjacent feature nodes, and then construct the adjacency matrix H using the cosine distance method. The calculation method is as follows: where h ij is used to measure the similarity between the F ith column eigenvector and the jth column eigenvector in the feature matrix X, and X i and X j represent the F ith column and the jth column eigenvectors in the feature matrix X respectively; Step 3.3: According to the adjacency matrix H calculated in Step 3.2, calculate the degree matrix S, and further construct the Laplacian matrix L based on weighted fusion F , and its calculation method is shown in formula (4): L F = S - H (4) In the formula, is the degree matrix and is a diagonal matrix, where the elements on the main diagonal are the degrees of each feature in the adjacency matrix; Step 3.4: Through the obtained Laplacian matrix L F , further obtain the feature structure regularization that preserves the local spatial geometric structure, and its calculation formula is shown in Formula (5): Trace of (W T L F W)(5) In the formula, is the feature weight matrix, and tr(·) represents the trace norm of the matrix; Step 4: Embed the feature correlation regularization and feature structure regularization into a multi-task model with low-rank constraint for feature learning to obtain a feature learning model. Specifically: Use the trace norm to approximate the low-rank constraint, introduce the trace norm on the basis of the multi-task learning model, and introduce the feature correlation regularization and feature structure regularization to obtain a multi-modal brain image feature learning model based on the fusion of feature correlation and feature structure. where Y i represents the label of the i-th modality, X i is the feature matrix of the i-th modality, and w i each row element in it represents the corresponding feature weight value under the i-th modality, and α, β, γ are regularization parameters and all are real numbers greater than zero; Step 5: Screen the feature vectors through the feature learning model, and perform linear fusion on the features extracted from the multi-modal data to obtain the fused feature matrix. Step 6: Divide the fused feature matrix of the sample set into a test set and a training set, use the training set to train the support vector machine and generate a model, and use the test set to test the classification performance of the model; classify the feature vectors using the trained model.

2. The multimodal brain image feature learning method according to claim 1, wherein: The specific content of Step 1 is as follows: Step 1.1: Extract the average gray matter density from the sMRI: Perform spatial normalization on the original image of the sMRI using a standard brain template, make the same region of each original image correspond to the template region one by one, and then segment the image into three brain tissue structures: gray matter, white matter, and cerebrospinal fluid; finally, use the brain parcellation template to extract the average gray matter density of the region of interest as the sMRI feature. Step 1.2: Extract the average glucose metabolism from the PET imaging: Perform head motion correction on the original image of the PET imaging, register it to the standard brain template for standardization and smoothing operations, and finally use the brain parcellation template to extract the average glucose metabolism of the brain region as the PET imaging feature.

3. The multi-modal brain image feature learning method according to claim 1, wherein: The specific content of Step 5 is as follows: Step 5.1: Through the feature learning model, learn the feature weights of the multi-modal brain images to obtain the feature weight matrix W, and screen the features according to the weight matrix. Step 5.2: Perform linear fusion on the features of the two modalities of brain images after screening, that is, splice multiple feature matrices together to obtain the fused feature matrix.

4. The multi-modal brain image feature learning method according to claim 1, wherein: The specific content of Step 6 is as follows: Divide the fused feature matrix into a test set and a training set, use a linear support vector machine for model training, use the test set data to verify the classification performance of the model, and judge whether the specified number of iterations is reached. If not, re - execute step 6; if so, select and output the feature vector corresponding to the best classification result.

5. The multimodal brain image feature learning method according to claim 1, wherein: The method further includes the following steps: Step 7: Visualize the key brain regions corresponding to the selected features for analyzing the key brain regions affected by the disease.

Images