A three-dimensional magnetic resonance brain image analysis method and its application

Abstract

The present invention discloses a three-dimensional magnetic resonance brain image analysis method and its application, and relates to the technical field of three-dimensional magnetic resonance brain image analysis. The present invention uses category information to perform feature alignment, and while adjusting the importance of marginal distribution and conditional distribution to domain adaptation, introduces MDD loss based on prototype representation to optimize the adversarial domain adaptation network loss; the cross entropy classification loss of the source domain, the attention consistency loss of the feature extraction network, the global domain adversarial loss for aligning the marginal distribution of the source domain and the target domain, the local domain adversarial loss for aligning the conditional distribution of the source domain and the target domain, and the MDD loss based on prototype representation together constitute the loss function of the network model. The present invention can not only achieve the simultaneous alignment of the marginal distribution and conditional distribution of the data, but also maximize the intra-class density, retaining and strengthening the corresponding extremely subtle intra-class and inter-class feature information.

Description

Technical Field

[0001] The present invention belongs to the technical field of three-dimensional magnetic resonance brain image analysis, and in particular relates to a three-dimensional magnetic resonance brain image analysis method and application thereof. Background Art

[0002] Alzheimer's disease is a progressive neurodegenerative disease with an insidious onset. Clinically, it is characterized by global dementia, including memory impairment, aphasia, apraxia, agnosia, visual-spatial skill impairment, executive dysfunction, and personality and behavioral changes.

[0003] Existing brain images of Alzheimer's disease are obtained by three-dimensional magnetic resonance imaging. When analyzing the data, the existing technical means used are:

[0004] The representation of three-dimensional magnetic resonance data is complex. Considering the spatial complexity of structural magnetic resonance imaging data, one method is to slice the three-dimensional image into two-dimensional images and then extract features. However, this method ignores the inherent spatial advantages of magnetic resonance data, resulting in subjective feature extraction and failure to fully utilize the unique spatial feature information of magnetic resonance data. Although general three-dimensional convolutional networks can realize feature extraction of spatial structural images, Alzheimer's disease magnetic resonance images are mainly composed of extremely subtle intra-class and inter-class features. Direct three-dimensional convolutional feature extraction will bring great challenges to the subsequent high-dimensional abstract feature learning of the network.

[0005] When using machine learning methods for model training, it is usually assumed that the data such as the training set and the test set are independent and identically distributed. However, this assumption is usually difficult to hold in reality. In the process of collecting magnetic resonance images, due to the influence of different instruments, scanning schemes, and image acquisition environments, it is impossible to guarantee that the obtained data have the same distribution, resulting in heterogeneity between data. As a result, when the model trained on the training set is applied to the test set, the model will suffer a significant performance degradation. This is the "domain shift" phenomenon that is common in actual processing;

[0006] The characteristics of magnetic resonance data of Alzheimer's disease are characterized by extremely subtle intra-class and inter-class differences. When existing adversarial domain adaptation methods use domain discriminators and feature extractors for game confrontation, they only align the data distribution of the source domain and the target domain. The important and subtle feature information in the data, especially the intra-class features, is not enhanced, resulting in a decrease in classification performance in the target domain.

[0007] An ideal adversarial domain adaptation model can only be trained when the source and target domain data classes are balanced. However, since neural network training is usually batch-based, data sampling is random, and class mismatches are prone to occur in small batches, it cannot guarantee that the difference between the two domains is minimized. Summary of the invention

[0008] The purpose of the present invention is to provide a three-dimensional magnetic resonance brain image analysis method and its application to solve the existing problems: Alzheimer's disease magnetic resonance images are mainly composed of extremely subtle intra-class and inter-class features, and direct three-dimensional convolution extraction of features is not convenient for subsequent high-dimensional abstract feature learning of the network; when the model trained on the training set is applied to the test set, the model will suffer a significant performance degradation.

[0009] In order to solve the above technical problems, the present invention is achieved through the following technical solutions:

[0010] A three-dimensional magnetic resonance brain image analysis method, comprising:

[0011] S1. A 3D convolutional neural network is used as the backbone network, and a channel attention module is introduced to extract features from the source and target domains of 3D MRI brain images.

[0012] S2. Add the MDD loss based on prototype representation to the cross entropy classification loss of the source domain, the global domain adversarial loss, and the local domain adversarial loss of the adversarial domain adaptation network to quantify the distribution divergence, so as to optimize the adversarial domain adaptation network loss;

[0013] The classifier re-predicts the source domain label based on the features of the source domain and the target domain. Then the classifier uses the source domain predicted label and the true label to derive the classification loss of the source domain, which is recorded as the cross entropy classification loss L of the source domain. cls ;

[0014] The discriminator re-predicts the domain label of the data based on the characteristics of the source domain and the target domain, and then derives the global domain adversarial loss L by comparing the predicted domain label with the real data domain label. g and local domain adversarial loss L l .

[0015] Furthermore, the three-dimensional convolutional neural network includes 9 3×3×3 convolutional layers; the number of channels is 8, 8, 16, 16, 32, 32, 64, 64 and 128 respectively; a BatchNorm layer and an activation function ReLU layer are set after each convolutional layer;

[0016] Downsampling operations are added after Conv2, Conv4, Conv6, Conv8, and Conv9 layers, with a step size of 2×2×2.

[0017] Furthermore, in S1, a channel attention module is introduced, which mainly includes:

[0018] Perform global average pooling and global maximum pooling on the convolution results of the last layer of the source domain and target domain of the backbone network respectively, and merge the pooling results of the two;

[0019] Then a single-channel convolution operation is performed to obtain a spatial attention magnetic resonance feature map;

[0020] The sigmoid activation function is then used to normalize it so that the importance of different brain regions can be reflected in the magnetic resonance feature map. That is, sensitive brain regions will be given greater weights, and less important brain regions will be given smaller weights.

[0021] Furthermore, in S1, the feature extraction model is as follows:

[0022] Assuming that the spatial attention magnetic resonance feature map is A, it is defined as follows:

[0023] A=σ(f 3×3×3 ([M max ,M avg ]))

[0024] Among them, σ represents the sigmoid activation function, f 3×3×3 represents a convolution operation with a convolution kernel of 3×3×3, M max and M avg They represent the result feature maps of global maximum pooling and global average pooling of feature map M respectively;

[0025] The spatial attention magnetic resonance feature map of the source domain is defined as A S , the spatial attention magnetic resonance feature map of the target domain is defined as A T ; The attention consistency loss is:

[0026]

[0027] Among them, N represents the number of samples in a batch, and H, W, and D represent height, width, and dimension respectively.

[0028] Furthermore, in S2, the MDD loss based on prototype representation is added to the cross entropy classification loss of the source domain, the global domain adversarial loss, and the local domain adversarial loss of the adversarial domain adaptation network to quantify the distribution divergence, mainly including:

[0029] S21. Based on the global domain discriminator and the local domain discriminator, the category label information is introduced into the local domain discriminator to align the features of the data. At the same time, the importance of marginal distribution and conditional distribution for domain adaptation is dynamically evaluated, and the marginal distribution of the source domain and the target domain are aligned using the global domain discriminator.

[0030] S22. In the local domain discriminator, category information is introduced to align the conditional distributions of the source domain and the target domain to perform domain adaptation of more subtle features;

[0031] S23. The weights of the local domain discriminator and the global domain discriminator are balanced using the adversarial factor ω;

[0032] S24. Calculate the distance between the prototype and the feature of the class as the MDD loss based on the prototype representation.

[0033] Furthermore, in S2, the global domain adversarial loss L g for:

[0034]

[0035] Among them, L d is the domain discrimination loss, G f is the feature extractor, d i For data x i The domain label, D s and D t are the domain label sets of the source domain and the target domain, n s and n t are the number of data in the source domain and the target domain respectively.

[0036] Furthermore, the local domain adversarial loss L l for:

[0037] The domain discriminator G d Divided into C domain discriminators, each domain discriminator is expressed as Each domain discriminator is responsible for matching the source domain and target domain data related to the cth class. The output of the label classifier G y (x i ) is used to represent data x i Belongs to the cth domain discriminator probability;

[0038]

[0039] in, and are the domain discriminator and cross entropy loss of class c, respectively. is the data x i Predict the probability distribution over class c, d i For data x i The domain label.

[0040] Furthermore, in S2, the MDD loss based on prototype representation is:

[0041]

[0042] Among them, f proto is the prototype representation of the corresponding class in the embedding space, m s and m tThey represent the number of pairs of data with the same label in the source domain and the target domain during each batch processing, which is dynamically calculated for each batch, and n b It is 1 / 2 of the batch size.

[0043] Furthermore, the loss function of the adversarial domain adaptation network is:

[0044] L total =L cls +λL att +αL MDD +ωL g +(1-ω)L l

[0045] Among them, L cls is the cross entropy classification loss of the source domain, L att is the attention consistency loss, L MDD is the MDD loss based on prototype representation, L g is the global adversarial loss, L l is the local domain adversarial loss, and λ, α, and ω are balancing hyperparameters.

[0046] The above-mentioned three-dimensional magnetic resonance brain image analysis method is applied to the automatic diagnosis equipment for Alzheimer's disease.

[0047] The present invention has the following beneficial effects:

[0048] In order to make full use of the spatial feature information of magnetic resonance data, the present invention uses a three-dimensional convolutional neural network for deep feature extraction and adds a channel attention feature module to the network, which helps to highlight the feature differences between brain regions between domains. This is different from the attention mechanism of other methods that can only focus on the feature distribution of a single data domain. At the same time, in order to transfer the feature semantic information of the source domain to the target domain, an attention consistency loss is designed in the feature extraction module, which is beneficial to enhance the performance of the model in the target domain.

[0049] In order to reduce the data distribution differences between related fields, the present invention adopts an adversarial domain adaptation method to train adversarial models. That is, a domain discriminator is used to identify whether the data is from the source domain or the target domain, and a feature extractor attempts to extract domain-invariant features of samples from the two domains, so that the domain discriminator cannot distinguish which domain the information comes from, and the two form an adversarial relationship.

[0050] In order to enhance the extremely subtle intra-class and inter-class feature information in magnetic resonance data as much as possible, the present invention introduces the distance metric loss MDD to optimize the adversarial domain adaptation network loss, thereby minimizing the difference between domains and maximizing the intra-class density. The two complement each other.

[0051] The prototype-based MDD loss of the present invention can alleviate the balance challenge problem in adversarial domain adaptation, and uses the nonlinear mapping of neural networks to map data to the prototype of the embedding space to calculate the class. The feature representation of data of the same category in the embedding space will be clustered around the class prototype to ensure that the statistical difference between the two domains is minimized.

[0052] The cross entropy classification loss of the source domain, the attention consistency loss of the feature extraction network, the global domain adversarial loss for aligning the marginal distribution of the source domain and the target domain, the local domain adversarial loss for aligning the conditional distribution, and the MDD loss based on prototype representation together constitute the loss function of the network model, which can not only align the marginal distribution and conditional distribution of the data at the same time, but also maximize the intra-class density, and retain and strengthen the corresponding extremely subtle intra-class and inter-class feature information. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 A schematic diagram of a three-dimensional magnetic resonance image analysis process of the present invention;

[0054] Figure 2 Schematic diagram of the prototype representation in the embedding space of the present invention. DETAILED DESCRIPTION

[0055] The technical solutions in the embodiments of the present invention are described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0056] See also Figure 1 and Figure 2 ,in, Figure 1 A schematic diagram of a three-dimensional magnetic resonance image analysis process of the present invention; Figure 2 Schematic representation of prototypes in embedding space.

[0057] 1. Feature extraction of three-dimensional magnetic resonance images based on channel attention.

[0058] A three-dimensional convolutional neural network is used as the backbone network, which mainly includes 9 3×3×3 convolutional layers with channel numbers of 8, 8, 16, 16, 32, 32, 64, 64 and 128 respectively. A BatchNorm layer and an activation function ReLU layer are set after each convolutional layer. In order to avoid overfitting and increase the receptive field, downsampling operations are added after Conv2, Conv4, Conv6, Conv8 and Conv9 layers with a step size of 2×2×2.

[0059] Alzheimer's disease is highly correlated with lesions in certain areas of the brain, so locating the disease-related brain areas can improve the robustness of the model accordingly. Therefore, a channel attention module is introduced to automatically identify disease-related brain areas. The main approach is to perform global average pooling and global maximum pooling on the convolution results of the last layer of the source domain and the target domain of the backbone network, respectively, and merge the pooling results of the two, and then perform a single-channel convolution operation to obtain a spatial attention magnetic resonance feature map, and then use the sigmoid activation function to normalize it, so as to reflect the importance of different brain regions in the contribution to automatic diagnosis in the magnetic resonance feature map, that is, brain areas that are sensitive to the disease will be given greater weights, and less important brain areas will be given smaller weights. Assume that the spatial attention magnetic resonance feature map is A, and its definition is as follows:

[0060] A=σ(f 3×3×3 ([M max ,M avg ]))

[0061] Among them, σ represents the sigmoid activation function, f 3×3×3 represents a convolution operation with a convolution kernel of 3×3×3, M max and M avg They respectively represent the result feature maps of global maximum pooling and global average pooling of the feature map M.

[0062] The spatial attention magnetic resonance feature map of the source domain is defined as A S , the spatial attention magnetic resonance feature map of the target domain is defined as A T In order to enhance the consistency of the feature maps of the two domains and facilitate the migration of semantic information from the source domain to the target domain, the attention consistency loss is designed to calculate the mean square error of the feature maps of the two domains:

[0063]

[0064] Where N represents the number of samples in a batch, and H, W, and D represent the height, width, and dimension respectively.

[0065] 2. Adversarial Domain Adaptation Loss Optimization

[0066] The distance metric loss MDD based on prototype representation is added to the cross entropy classification loss of the source domain, the global domain adversarial loss, and the local domain adversarial loss of the adversarial domain adaptation network to quantify the distribution divergence and optimize the adversarial domain adaptation network loss. In metric learning, the main feature of the MDD method is that it not only reduces the marginal distribution between domains by minimizing the inter-domain divergence, but also introduces class information to optimize the conditional distribution between domains and maximize the intra-class density, which can be used to adjust the influence of subtle intra-class features. At the same time, in order to solve the balance challenges and difficult training problems of adversarial learning, the MDD loss based on prototype representation is proposed to strengthen the alignment of extremely subtle intra-class and inter-class features.

[0067] The adversarial domain adaptation network model introduces category information in the domain discriminator to align the data at a very subtle level, and dynamically evaluates the importance of marginal distribution and conditional distribution for domain adaptation. First, a global domain discriminator is used to align the marginal distribution of the source domain and the target domain. The global domain adversarial loss is:

[0068]

[0069] Among them, L d is the domain discrimination loss, G f is the feature extractor, d i For data x i The domain label, D s and D t are the domain label sets of the source domain and the target domain, n s and n t are the number of data in the source domain and the target domain respectively.

[0070] In the local domain discriminator, category information is introduced to align the conditional distributions of the source domain and the target domain, so that domain adaptation of more subtle features can be performed. d Divided into C domain discriminators, each domain discriminator is expressed as Each domain discriminator is responsible for matching the source domain and target domain data related to the cth class. The output of the label classifier G y (x i ) is used to represent data x i Belongs to the cth domain discriminator The probability of the local domain discriminator is calculated as:

[0071]

[0072] in and are the domain discriminator and cross entropy loss of class c, respectively. is the data x i Predict the probability distribution over class c, d i For data xi The domain label.

[0073] The global domain discriminator and the local domain discriminator are responsible for aligning the marginal distribution and conditional distribution respectively. For the characteristics of structural magnetic resonance imaging and local lesions of Alzheimer's disease in brain structure, the data has the characteristics of global similarity and local differences. Therefore, in actual operation, the weights of the local domain discriminator and the global domain discriminator are balanced by the adversarial factor ω.

[0074] See also Figure 2 , the MDD loss based on prototype representation needs to calculate the distance between the prototype and the feature of the class as the loss, which is:

[0075]

[0076] Among them, f proto is the prototype representation of the corresponding class in the embedding space, m s and m t They represent the number of pairs of data with the same label in the source domain and the target domain during each batch processing, which is dynamically calculated for each batch, and n b It is 1 / 2 of the batch size.

[0077] The main part of the model contains a feature extraction backbone network. The source domain data features fs and predicted labels ys are passed through the classifier to obtain the classification loss of the source domain. The classifier consists of three layers of full connections, and the number of neurons in each layer is 256, 128, and n (n is the number of label categories). The features and predicted labels (fs, ys, ft, yt) of the source domain and target domain are simultaneously sent to the domain discriminator for domain confusion. The structure of the domain discriminator is the same as that of the classifier, and it is also composed of three layers of full connections. The number of neurons in each layer is 256, 128, and n (n is the number of label categories). The domain discriminator and the feature extractor compete against each other until a balance point is reached.

[0078] The final network model loss function is:

[0079] L total =L cls +λL att +αL MDD +ωL g +(1-ω)L l

[0080] Where L cls is the cross entropy classification loss of the source domain, L att is the attention consistency loss, L MDD is the MDD loss based on prototype representation, L g is the global adversarial loss, L lis the local domain adversarial loss, and λ, α, and ω are balancing hyperparameters.

[0081] In the embodiments provided by the present invention, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation.

[0082] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A three-dimensional magnetic resonance brain image analysis method, characterized in that: include: S1. A 3D convolutional neural network is used as the backbone network, and a channel attention module is introduced to extract features from the source and target domains of 3D MRI brain images. S2. Add the MDD loss based on prototype representation to the cross entropy classification loss of the source domain, the global domain adversarial loss, and the local domain adversarial loss of the adversarial domain adaptation network to quantify the distribution divergence, so as to optimize the adversarial domain adaptation network loss; The classifier re-predicts the source domain label based on the features of the source domain and the target domain. Then the classifier uses the source domain predicted label and the true label to derive the classification loss of the source domain, which is recorded as the cross entropy classification loss L of the source domain. cls ; The discriminator re-predicts the domain label of the data based on the characteristics of the source domain and the target domain, and then derives the global domain adversarial loss L by comparing the predicted domain label with the real data domain label. g and local domain adversarial loss L l ; In S1, the feature extraction model is as follows: Assuming that the spatial attention magnetic resonance feature map is A, it is defined as follows: A=σ(f 3×3×3 ([M max ,M avg ])) Among them, σ represents the sigmoid activation function, f 3×3×3 represents a convolution operation with a convolution kernel of 3×3×3, M max and M avg They represent the result feature maps of global maximum pooling and global average pooling of feature map M respectively; The spatial attention magnetic resonance feature map of the source domain is defined as A S , the spatial attention magnetic resonance feature map of the target domain is defined as A T ; The attention consistency loss is: Where N represents the number of samples in a batch, H, W, and D represent height, width, and dimension respectively; In S2, the MDD loss based on prototype representation is added to the cross entropy classification loss of the source domain, the global domain adversarial loss, and the local domain adversarial loss of the adversarial domain adaptation network to quantify the distribution divergence, mainly including: S21. Based on the global domain discriminator and the local domain discriminator, the category label information is introduced into the local domain discriminator to align the features of the data. At the same time, the importance of marginal distribution and conditional distribution for domain adaptation is dynamically evaluated, and the marginal distribution of the source domain and the target domain are aligned using the global domain discriminator. S22. In the local domain discriminator, category information is introduced to align the conditional distributions of the source domain and the target domain to perform domain adaptation of more subtle features; S23. The weights of the local domain discriminator and the global domain discriminator are balanced using the adversarial factor ω; S24. Calculate the distance between the prototype and the feature of the class as the MDD loss based on the prototype representation; In S2, the global domain adversarial loss L g for: Among them, L d is the domain discrimination loss, G f is the feature extractor, d i For data x i The domain label, D s and D t are the domain label sets of the source domain and the target domain, n s and n t are the number of data in the source domain and the target domain respectively; Local domain adversarial loss L l for: The domain discriminator G d Divided into C domain discriminators, each domain discriminator is represented as Each domain discriminator is responsible for matching the source domain and target domain data related to the cth class. The output of the label classifier G y (x i ) is used to indicate that the data xi belongs to the cth domain discriminator probability; in, and are the domain discriminator and cross entropy loss of class c, respectively. is the predicted probability distribution of data xi on class c, d i For data x i The domain label of In S2, the MDD loss based on prototype representation is: Among them, f proto is the prototype representation of the corresponding class in the embedding space, m s and m t They represent the number of pairs of data with the same label in the source domain and the target domain during each batch processing, which is dynamically calculated for each batch, n b It is 1 / 2 of the batch size.

2. A three-dimensional magnetic resonance brain image analysis method according to claim 1, characterized in that: The three-dimensional convolutional neural network includes 9 3×3×3 convolutional layers; the number of channels is 8, 8, 16, 16, 32, 32, 64, 64 and 128 respectively; a BatchNorm layer and an activation function ReLU layer are set after each convolutional layer; Downsampling operations are added after Conv2, Conv4, Conv6, Conv8, and Conv9 layers, with a step size of 2×2×2.

3. A three-dimensional magnetic resonance brain image analysis method according to claim 1, characterized in that: In S1, the channel attention module is introduced, which mainly includes: Perform global average pooling and global maximum pooling on the convolution results of the last layer of the source domain and target domain of the backbone network respectively, and merge the pooling results of the two; Then a single-channel convolution operation is performed to obtain a spatial attention magnetic resonance feature map; The sigmoid activation function is then used to normalize it so that the importance of different brain regions can be reflected in the magnetic resonance feature map. That is, sensitive brain regions will be given greater weights, and less important brain regions will be given smaller weights.

4. A three-dimensional magnetic resonance brain image analysis method according to claim 1, characterized in that: The loss function of the adversarial domain adaptation network is: L total =L cls +λL att +αL MDD +ωL g +(1-ω)L l Among them, L cls is the cross entropy classification loss of the source domain, L att is the attention consistency loss, L MDD is the MDD loss based on prototype representation, L g is the global adversarial loss, L l is the local domain adversarial loss, and λ, α, and ω are balancing hyperparameters.

5. Application of a three-dimensional magnetic resonance brain image analysis method as claimed in any one of claims 1 to 4 in an automatic diagnosis device for Alzheimer's disease.

Images