MRI (Magnetic Resonance Imaging) sequence identification method fusing fine granularity and prototype learning

Abstract

The invention discloses an MRI (Magnetic Resonance Imaging) sequence identification method fusing fine granularity and prototype learning. The MRI sequence identification method comprises the steps of acquiring fine granularity image features, constructing a prototype and carrying out sequence classification on the fine granularity image features. According to the method, the fine-grained network model is designed, the medical image features with fine granularity are extracted, and the problems of small inter-class difference and large intra-class difference of the fine-grained MRI images are solved. Meanwhile, a representative prototype is constructed for each sequence, the medical images are classified by calculating the distance between the prototype and the fine-grained image features, and the problem of complex feature distribution caused by complex MRI images is effectively relieved. Through the effective combination of the two proposed methods, the feature extraction quality and the classification precision of the model are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of digital image processing and computer vision, and in particular to an MRI sequence recognition method integrating fine-grained learning and prototype learning. Background Art

[0002] In medical imaging, different MRI scanning parameters correspond to different MRI images, providing doctors with a variety of diagnostic information during the diagnosis process. Therefore, in the process of medical imaging diagnosis, accurate identification of various MRI sequences is very important for subsequent disease diagnosis. At present, in the field of intelligent medical care, MRI sequence recognition has become the basis of various intelligent medical tasks. However, traditional deep learning algorithms have certain difficulties in applying to MRI sequence recognition. First, different MRI sequences have small differences between classes and large differences within classes. Secondly, MRI images are extremely complex, which also makes it difficult for traditional deep learning algorithms to recognize sequences. Therefore, how to more effectively use deep learning-based methods to solve the above-mentioned problems in MRI sequence recognition has become a very challenging task.

[0003] In order to solve the above problems, the present invention adopts a method based on fine-grained and prototype learning to perform MRI sequence recognition. Fine-grained image recognition focuses on processing objects of multiple subordinate categories belonging to the same meta-category, which is quite different from general image classification. Many different algorithms have been proposed in the research field for fine-grained image classification problems. These algorithms can be mainly divided into two categories: classification algorithms based on positioning and classification algorithms based on feature coding. Algorithms based on positioning usually highlight the differences in fine-grained categories by locating and aligning key local areas in the image. These algorithms can improve classification performance by using detailed information of local areas. Algorithms based on feature coding, on the other hand, convert images into discriminative feature vectors by learning robust feature expressions. Fine-grained image classification algorithms based on positioning aim to locate key parts in the image and input the key parts into the classification network for classification. Many early works used additional image annotations to locate the semantic key parts of the object. Branson et al. proposed to use the detected partial key points to calculate multiple distorted image regions and further obtain the corresponding part-level features through posture normalization. Fine-grained image classification algorithms based on feature coding aim to learn more robust features to guide the model for classification. Kong et al. used a low-rank approximation of the covariance matrix and further learned a low-rank bilinear classifier. The resulting classifier can be evaluated without explicitly calculating the bilinear feature matrix, which greatly reduces the model parameters and achieves good results on multiple public datasets.

[0004] In addition, we use the method of building prototypes to classify the extracted image features. Unlike traditional image classification algorithms, the classification algorithm based on prototype learning stores a set of representative prototypes and then uses these prototypes for classification. In the field of natural images, Yang et al. believed that the use of convolutional neural networks for classification tasks lacks robustness, and used the prototype learning method to replace the final softmax layer and store several prototypes for each category. Through metric learning, the distance between the output feature and the corresponding prototype feature in the feature space is reduced. The author experimentally proved that prototype learning can aggregate features of the same category more compactly in the feature space than traditional classification models, and also confirmed that the classification algorithm based on prototype learning can solve the problem of difficult samples in complex data distributions. Summary of the invention

[0005] The purpose of the present invention is to provide an MRI sequence recognition method based on fine-grained and prototype learning. A fine-grained network model is designed to extract fine-grained MRI image features, and a classification prototype is constructed to classify MRI image features through the prototype, which effectively solves the intra-class and inter-class problems and image complexity problems existing in existing algorithms.

[0006] In order to achieve the above purpose, the present invention adopts the following technical scheme: The method of this paper first designs a fine-grained network model and constructs prototypes for different MRI sequence categories. Next, we input the medical image into the fine-grained network model, extract fine-grained image features, and improve the quality of image features through the fine-grained network model. Finally, the distance between the extracted image features and the constructed prototype is measured, and the input image and the image are classified.

[0007] A method for MRI sequence recognition integrating fine-grained and prototype learning comprises the following steps:

[0008] Step 1: Acquire fine-grained features of training data.

[0009] Step 1.1, input training images.

[0010] Step 1.2: The image passes through the backbone model to obtain basic image features.

[0011] Step 1.3: The basic image features are input into the fine-grained module to extract fine-grained image features.

[0012] Step 2: Category prototype construction.

[0013] Step 2.1, obtain the fine-grained image features of all images in the training set.

[0014] In step 2.2, the class center of the fine-grained features of each class is calculated.

[0015] In step 2.3, several fine-grained image features closest to the class center are selected as the prototypes of the class.

[0016] Step 3: Image to be classified.

[0017] Step 3.1: Input the image to be classified into the trained network model to obtain fine-grained image features.

[0018] In step 3.2, the distance between the output image features and the constructed prototype is measured, and the sequence category of the prototype with the closest distance is selected as the sequence category of the image to be classified.

[0019] Compared with the prior art, the present invention has the following obvious advantages:

[0020] The present invention uses the fine-grained learning method to extract the fine-grained image features with decisive factors in medical images, which alleviates the problem of small inter-class differences and large intra-class differences in medical images. On this basis, the prototype classification strategy is fully utilized to classify medical sequence images by calculating the distance between fine-grained features and prototypes, which alleviates the problem of chaotic feature distribution caused by the complexity of medical images. Through the effective combination of the above two methods, the final classification accuracy is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is a flow chart of the method of the present invention; DETAILED DESCRIPTION

[0022] The present invention will be further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

[0023] The flow chart of the method of the present invention is as follows Figure 1 As shown, the specific steps include:

[0024] Step 1: Acquire fine-grained features of training data.

[0025] Step 1.1, input training images.

[0026] Step 1.2: The image passes through the backbone model to obtain basic image features.

[0027] Step 1.3: The basic image features are input into the fine-grained module to extract fine-grained image features.

[0028] The training data is divided from the DUKE Liver Dataset public data set into a training set and a test set. The first step is to extract fine-grained features from the training set.

[0029] Use ResNet and VIT as the backbone network. Then, ResNet starts from the input image I qExtract the basic features of MRI images q Basic MRI features F q It is divided into M equal feature blocks. After that, the obtained feature blocks are added with position information and input into the linear projection layer of VIT to obtain the input of VIT layer 0.

[0030] As the VIT Layer continues to deepen, the input of the lth VIT Layer is expressed as Represents the i-th token in the first layer of VIT. represents the CLS token of VIT at layer l, and M represents the number of tokens. VIT consists of N+1 layers of multi-head self-attention (MSA) and multi-layer perceptron (MLP). Therefore, the input of layer 1 can be expressed as formula 1,2.

[0031]

[0032]

[0033] Where LN represents layer normalization. The attention weight of the l-th layer multi-head self-attention can be expressed as Formula 3.

[0034]

[0035] in Represents the number of heads in the Multi-Head Self-Attention (MSA) of the Vision Transformer.

[0036] An important challenge in MRI sequence classification is to accurately locate the discriminative regions and achieve classification based on the subtle differences between different sequences. Inspired by TransFG, in order to obtain the fine-grained features of MRI, a fine-grained feature extraction module is proposed, which uses the multi-head self-attention of VIT to select important fine-grained features in MRI features. We retain the attention weights of the first N-1 layers, as shown in Formula 4, represents the attention weight of the i-th head in the l-th layer, where w in 0,j represents the attention weight between CLS token and other image patches, w 0,j The larger the value, the more important the corresponding image patch is. Therefore, we filter important fine-grained areas based on the attention weight between the image patch and CLS in the attention weight.

[0037]

[0038] Inspired by Abnar et al., as the number of VIT layers increases, the self-attention weights of the high-level layers can no longer represent the importance between the image patch and the CLS. Therefore, we fuse the first N-1 layers to obtain the final attention weight A used to guide the image patch selection. final , as shown in Formula 5

[0039]

[0040] We send the filtered fine-grained image patches to the last layer of VIT, forcing the last layer of VIT to focus on fine-grained local information. Finally, we obtain the fine-grained image features of all data in the training set. Next, we construct category prototypes based on the extracted fine-grained image features.

[0041] Step 2: Category prototype construction.

[0042] Step 2.1, obtain the fine-grained image features of all images in the training set.

[0043] In step 2.2, the class center of the fine-grained features of each class is calculated.

[0044] In step 2.3, several fine-grained image features closest to the class center are selected as the prototypes of the class.

[0045] To alleviate the problem of small inter-class differences and large intra-class differences in MRI images, we constructed a prototype classification module. The prototype classification module constructs multiple prototypes for classification by measuring the distance to the prototype type instead of class boundary learning. This is beneficial for distinguishing MRI sequences, which have obvious diversity within the category. Specifically, we constructed a prototype pool in which each sequence category contains multiple prototypes, and classified MRI according to the prototype pool.

[0046] The prototype classification module is divided into two stages: the construction of the prototype pool and the classification of the prototype pool. Figure 1 As shown, during the prototype pool construction phase, we will Figure 1 The prototype classification module in is replaced with a fully connected layer, and the model is trained. After training, we extract the features of all sample outputs in the training set and cluster them by category, and select several features closest to the center of each category as the prototype of the category. As shown in Formula 7, we construct a prototype pool Pi for each category, where the prototype pool Pi contains G prototypes, as shown in Formula 6.

[0047] P i =[p i,0 ,p i,1 ,…p i,j …,p i,G] (6)

[0048] R=[P0,P1,…,P i ,…,P c ] (7)

[0049] In the prototype classification module, we replace the above-mentioned fully connected layer with the prototype classification module, and classify the samples by measuring the distance between the features of the output samples and the category prototype. As shown in Formula 8, we calculate the distance between the output sample and each prototype pool, and select the category of the prototype with the smallest distance as the category of the sample. The prototype classification module can better represent the sequence category features and better delinearize the category boundaries, thereby alleviating the difficult sample problem of MRI. In addition, the prototype classification module can make full use of the fine-grained features extracted by the feature selection module to reduce the misclassification caused by irrelevant features.

[0050]

[0051]

[0052] The loss function of the prototype classification module is shown in Formula 10, where αβ is a predefined hyperparameter. In the first loss, we minimize the distance between the sample features and the prototype of the corresponding category, and in the second loss, we maximize the distance between the sample features and the prototypes of different categories to alleviate the problem of small differences between classes and large differences within classes. In addition, as shown in Formula 11, we add a loss between prototypes, in which we prevent the problem of prototypes of different categories getting close to each other during the learning process by reducing the distance between prototypes of the same category and increasing the distance between prototypes of different categories, effectively improving the accuracy of the model.

[0053]

[0054] loss p =-DIs(P i ,P j ) (11)

[0055] Step 3: Image to be classified.

[0056] Step 3.1: Input the image to be classified into the trained network model to obtain fine-grained image features.

[0057] In step 3.2, the distance between the output image features and the constructed prototype is measured, and the sequence category of the prototype with the closest distance is selected as the sequence category of the image to be classified.

[0058] So far, the present invention has completed the construction of the category prototype and the training of the model. Next, the final image to be classified is classified. First, we input the sequence image to be classified into the network model to extract fine-grained image features. Finally, we calculate the distance measurement between the output image features to be classified and each category prototype, and select the category of the prototype with the closest distance as the category of the image to be classified.

[0059] The network performance was verified using the training method shown in the above steps. To ensure that the experimental results have a certain degree of generalization, the experiment adopted the idea of ​​cross-validation, that is, first perform 5-fold cross-validation on each part of the training data, select the optimal training hyperparameters, and then use the optimal hyperparameters to train the entire training data, and test the converged network model on each part of the test data. This process was repeated 5 times, and the performance was evaluated by the average diagnostic index of the 5 tests. The experimental results are shown in the table:

[0060] This study conducted experiments on the DUKE Liver Dataset and the laboratory private dataset. The experimental results are shown in Tables 1 and 2.

[0061]

[0062] Table 1

[0063]

[0064] Table 2

[0065] We conducted comparative experiments on several fine-grained, prototype, and baseline methods. Among the fine-grained methods, we selected tranfg as a comparison method; among the prototype methods, we also selected tesnet. Finally, we compared VIT and ResNet as baseline methods. We used three commonly used indicators, namely accuracy, recall, and precision. The experiments proved that the algorithm of the present invention achieved a great performance improvement.

[0066] So far, the specific implementation process of the present invention has been described.

Claims

1. A method for MRI sequence recognition integrating fine-grained and prototype learning, characterized in that: The following steps are involved: Step 1: Acquire fine-grained features of training data; Step 1.1, input training image; Step 1.2, the image passes through the backbone model to obtain basic image features; For a given MRI image I in the training set q , and extract the basic features through the backbone network to obtain the basic MRI features F q ; Step 1.3, the basic image features are input into the fine-grained module to extract fine-grained image features; Basic MRI featuresF q It is divided into M equal feature blocks; the obtained feature blocks are added with position information and then input into the linear projection layer of VIT; a fine-grained module is designed to extract the basic features F q Extract fine-grained image features. The fine-grained module uses the self-attention mechanism in Vision Transformer to achieve fine-grained feature screening; By retaining the attention weights of the first N-1 layers: A i l represents the attention weight of the i-th head of the l-th layer, where A i l w in 0,j represents the attention weight between CLS token and other image patches, w 0,j The larger the size, the more important fine-grained areas are filtered according to the attention weight between the image patch and the CLS in the attention weight; Step 2: Category prototype construction; Step 2.1, obtaining fine-grained image features of all images in the training set; Step 2.2, calculate the class center of the fine-grained features of each category; Step 2.3, select several fine-grained image features closest to the class center as the prototype of the class; The output features of all samples in the training set are clustered according to the corresponding labels, and several prototype samples closest to the class center are selected as representatives of the category; the training model obtained based on the classification strategy is fine-tuned through prototype learning to reduce the distance between the sample features and the corresponding prototype features in the feature space, while increasing the distance between the sample features and the prototype features of non-corresponding categories; Step 3, image to be classified; Step 3.1, input the image to be classified into the trained network model to obtain fine-grained image features; For the image to be classified, it is input into the backbone model to obtain the basic features of the classified image, and the basic features are sent to the Vision Transformer with the fine-grained module added. Through the self-attention mechanism of the Vision Transformer, the fine-grained semantic features of the classified image are obtained, and the fine-grained features are input into the Vision Transformer of the final layer, forcing the final layer to focus on the detail area to capture the final fine-grained overall features; Step 3.2, measure the distance between the output image features and the constructed prototype, and select the sequence category of the prototype with the closest distance as the sequence category of the image to be classified; For the image to be tested, the fine-grained features are obtained, and the distance between the fine-grained features of the image to be tested and the category prototype is used to classify the sample. The distance between the output sample and each prototype pool is calculated, and the category of the prototype with the smallest distance is selected as the category of the sample; the prototype classification module represents the sequence category features and delinearizes the category boundaries to alleviate the difficult samples of MRI; the prototype classification module uses the fine-grained features extracted by the feature selection module to reduce the misclassification caused by irrelevant features.

2. The MRI sequence recognition method integrating fine-grained and prototype learning according to claim 1, characterized in that: In steps 1.2 to 1.3, the fine-grained MRI image features of the image are obtained as follows: for an image to be classified, the basic features of the image are first obtained, and then the basic features are input into the Vision Transformer, the self-attention of the first N layers in the Vision Transformer is fused, and the fine-grained image features are filtered according to the fused attention.

3. The MRI sequence recognition method integrating fine-grained and prototype learning according to claim 1, characterized in that: In step 2.2-step 2.3, the MRI sequence category prototype is constructed as follows: fine-grained features of all images in the training set are obtained, and the class center of each category is obtained according to the category, the prototype of each sequence category is obtained according to the class center, and the image to be classified is classified according to the constructed prototype.

Images