Multimodal medical image classification system based on convolutional neural network and curriculum learning

Abstract

The present invention discloses a multimodal medical image classification system based on convolutional neural networks and curriculum learning, which includes: an image preprocessing module and a multi-scale convolutional neural network classifier, and the multi-scale convolutional neural network classifier includes a Resnet50 network, a feature pyramid network, a maskROI module and an attention module. The Resnet50 network extracts the features of the input image and outputs it to the feature pyramid network for processing to obtain a complete feature map and outputs it to the attention module; the maskROI module is used to input a specific image containing a region of interest into the classifier; the feature map obtained by the attention module is multiplied by the image input by the maskROI module and output as a result. The present invention uses curriculum learning as prior knowledge for network training to reasonably utilize limited data. During the training process, the learning difficulty of the network gradually increases, making the training process more reasonable and obtaining a more accurate classifier.

Description

Technical Field

[0001] The present invention relates to the field of medical image processing, and in particular to a multimodal medical image classification system based on convolutional neural networks and curriculum learning. Background Art

[0002] Prostate cancer is one of the most common cancers in middle-aged and elderly men and ranks fifth among the leading causes of cancer death in men. Early detection and timely treatment of prostate cancer can effectively improve the five-year survival rate of prostate cancer patients. The challenge in diagnosing prostate cancer lies in distinguishing insignificant from clinically significant prostate cancer. Currently, the best method for assessing lesion aggressiveness is the histopathological Gleason score, which is classified into five different Gleason grades. However, the Gleason score is obtained through a biopsy, which can overdiagnose insignificant cancers and is insensitive for diagnosing clinically significant cancers. Recent studies have demonstrated the potential of MRI for the diagnosis and grading of prostate cancer. Furthermore, computer-assisted diagnosis (CAD) techniques can help radiologists estimate the grade of prostate cancer from MRI images, whereas manual classification alone is time-consuming and stressful. Therefore, research on automated MRI methods for determining Gleason grade is of great significance.

[0003] In recent years, image classification algorithms based on deep learning have been widely used to predict prostate cancer and have achieved good results. In 2019, Cao et al. (Cao, R., et al., Joint prostate cancer detection and gleason score prediction in mp-MRI via FocalNet. 2019. 38(11): p. 2496-2506.) proposed a new multi-classification convolutional neural network, FocalNet, which uses the Gleason score (GS) to simultaneously detect prostate cancer lesions and predict their invasiveness. For Gleason score classification, through ROC curve analysis, FocalNet's AUC for predicting GS ≥ 7 vs. GS < 7 was 0.81, and for predicting GS ≥ 4 + 3 vs. GS ≤ 3 + 4, the AUC was 0.79. However, most existing methods only distinguish between high-grade and low-grade prostate cancer as their main task.

[0004] In general, there are at least two common problems in existing methods. (a) Traditional methods usually crop a patch of a specific size for training based on the known coordinates of the lesion center. However, if the cropping size is too large, the image will contain some irrelevant areas, and if the size is too small, the regional information of some large lesions will be missing. (b) On the other hand, there is no significant difference in visual appearance between different Gleason grades. The expression of Gleason grade in MRI is diverse and uncertain. For example, a Gleason grade 5 lesion may appear similar to a Gleason grade 1 lesion on MRI. Therefore, we need to allocate and apply limited data more reasonably. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a multimodal medical image classification system based on convolutional neural networks and curriculum learning in response to the deficiencies in the above-mentioned prior art.

[0006] To solve the above technical problems, the technical solution adopted by the present invention is: a multimodal medical image classification system based on convolutional neural networks and curriculum learning, which includes: an image preprocessing module and a multi-scale convolutional neural network classifier, and the multi-scale convolutional neural network classifier includes a Resnet50 network, a feature pyramid network, a maskROI module and an attention module.

[0007] Preferably, the Resnet50 network extracts features of the input image and outputs them to the feature pyramid network, the feature pyramid network is processed to obtain a complete feature map and outputs it to the attention module, the attention module is used to redistribute the weights of each channel and space in the feature map; the maskROI module is used to input a specific image containing a region of interest into the classifier so that the classifier pays more attention to the region of interest; the feature map obtained by the attention module is multiplied by the image input by the maskROI module and output as a result.

[0008] Preferably, the image preprocessing module includes a registration unit, a patch extraction unit and a normalization unit.

[0009] Preferably, the registration unit uses one modality of the input multimodal medical image as a reference image, and the images of the remaining modalities as floating images to perform registration calculations with the reference image respectively, and applies the registration transformation matrix obtained by the registration calculations to all floating images to achieve image registration;

[0010] The patch extraction unit uses an automatic segmentation network to obtain a region of interest in the registered image, and extracts a 2D patch image of the region of interest containing the lesion based on the region of interest;

[0011] The normalization unit performs normalization processing on the 2D patch image and outputs a preprocessed image. The normalization processing method is: for any image, the grayscale value of any pixel of the image is subtracted from the mean of the grayscale values ​​of all pixels in the image and then divided by the standard deviation of the grayscale values ​​of all pixels in the image.

[0012] Preferably, the feature pyramid network includes bottleneck layer 1, bottleneck layer 2, bottleneck layer 3, bottleneck layer 4, convolution layer C2, convolution layer C3, convolution layer C4, convolution layer C5, smoothing layer and Concat layer;

[0013] The features of the image output by the Resnet50 network are used to generate feature maps of four different resolutions through the feature pyramid network. The feature maps from bottleneck layers 1-4 are enhanced using lateral connections. Each lateral connection merges feature maps of the same spatial size from bottleneck layers 1-4 and C5-C2 layers to obtain feature maps of three scales. The feature maps of the three scales are then processed using a smoothing layer to reduce the aliasing effect of upsampling. Finally, the features of these different scales are upsampled to the same scale and then merged using a concat layer to obtain a complete feature map.

[0014] Preferably, the attention module is a BAM module, which combines spatial attention and channel attention in parallel. For a given input feature map F∈R C×H×W , the BAM module infers a 3D attention map M(F)∈R C ×H×W , the calculation method of the feature map F′ after the BAM module redistributes the weights of each channel and space is:

[0015]

[0016] in, Represents element-wise multiplication; the calculation method of the 3D attention map M(F) is: first calculate the channel attention map M on two independent branches c (F)∈R C and spatial attention map M s (F)∈R C , and then calculate the BAM attention map M(F) according to the following formula:

[0017] M(F)=σ(M c (F)+M s (F));

[0018] Where σ is a sigmoid function so that the two independent branch outputs are rescaled to R before being added C×H×W .

[0019] Preferably, the maskROI module convolves the input specific image containing the region of interest to generate an attention map, and the attention map obtains a feature map through a convolution layer with a convolution kernel size of 3×3 and a step size of 4 and outputs it; the feature map output by the maskROI module is element-wise multiplied with the feature map output by the attention module and output as the result.

[0020] Preferably, the method for classifying multimodal medical images using the multimodal medical image classification system based on convolutional neural network and curriculum learning comprises the following steps:

[0021] 1) Training image preparation:

[0022] Acquire multimodal medical images, mark the target area in each modality of the multimodal medical images, obtain the region of interest mask corresponding to each modality, and manually perform subjective scoring on the target lesion area;

[0023] 2) Preprocessing the image obtained in step 1) by the image preprocessing module:

[0024] The registration unit registers the input multimodal medical image, and the patch extraction unit extracts a 2D patch image of a region of interest containing a lesion from the registered image; the normalization unit normalizes the 2D patch image and outputs a pre-processed image;

[0025] 3) using the image obtained in step 2) to train the multi-scale convolutional neural network classifier using a curriculum-based learning method to obtain a trained classifier;

[0026] 4) For the multimodal medical images that need to be classified, they are first input into the image preprocessing module for preprocessing, and then input into the trained classifier obtained in step 3) for classification processing to obtain a classification result.

[0027] Preferably, the step 3) is specifically as follows:

[0028] 3-1) The training images are manually scored objectively in advance, and then the absolute value Gap between the objective score and the subjective score is used as the evaluation criterion for the training data:

[0029] 3-2) Inputting the preprocessed image into the multi-scale convolutional neural network classifier for training:

[0030] First, the training data is divided into N parts from easy to difficult according to the size of the Gap value, where N is not less than 2, the easiest data has the smallest Gap value, and the most difficult data has the largest Gap value;

[0031] During the first training, the easiest first data set is input. When the loss converges, the second data set is input together with the first data set for the second training. In subsequent training, more difficult data is added each time. That is, during the i-th training, the input data is the sum of the first to i-th data sets, where i = 1, 2, ..., N; until the loss converges during the N-th training, a trained classifier is obtained.

[0032] The beneficial effects of the present invention are: the multimodal medical image classification system based on convolutional neural network and curriculum learning provided by the present invention uses curriculum learning as prior knowledge for network training to more reasonably utilize limited data. During the training process, the learning difficulty of the network gradually increases, making the training process more reasonable and obtaining a more accurate classifier; when the present invention is applied to predict the Gleason level of prostate cancer in multi-parameter MRI images, the Kappa value for predicting the Gleason level of prostate cancer in multi-parameter MRI can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 Schematic diagram of the structure of the multimodal medical image classification system based on convolutional neural network and curriculum learning of the present invention;

[0034] Figure 2 The structure of the BAM module of the present invention;

[0035] Figure 3 is the training process of the classifier in the present invention;

[0036] Figure 4 This is a flowchart of image classification in the multimodal medical image classification system based on convolutional neural networks and curriculum learning of the present invention. DETAILED DESCRIPTION

[0037] The present invention is further described in detail below with reference to the embodiments so that those skilled in the art can implement the invention with reference to the description.

[0038] It should be understood that terms such as “having”, “including” and “comprising” used herein do not preclude the existence or addition of one or more other elements or combinations thereof.

[0039] Example 1

[0040] The present embodiment is a multimodal medical image classification system based on convolutional neural network and curriculum learning, which includes: an image preprocessing module and a multi-scale convolutional neural network classifier, the multi-scale convolutional neural network classifier includes a Resnet50 network, a feature pyramid network (FPN), a maskROI module and an attention module, referring to Figure 1 Schematic diagram of the network framework of the classifier.

[0041] In this embodiment, the image preprocessing module includes a registration unit, a patch extraction unit, and a normalization unit. The registration unit uses one modality of the input multimodal medical image as a reference image and the images of the remaining modalities as floating images to perform registration calculations with the reference image. The registration transformation matrix obtained by the registration calculation is applied to all floating images to achieve image registration.

[0042] The patch extraction unit uses an automatic segmentation network to obtain the region of interest in the registered image, and extracts the 2D patch image of the region of interest containing the lesion based on the region of interest;

[0043] The normalization unit normalizes the 2D patch image and outputs the preprocessed image. The normalization method is: for any image, the grayscale value of any pixel in the image is subtracted from the mean of the grayscale values ​​of all pixels in the image and then divided by the standard deviation of the grayscale values ​​of all pixels in the image.

[0044] In this embodiment, the Resnet50 network is used as the backbone network to extract the features of the input image and output it to the feature pyramid network. The feature pyramid network is processed to obtain a complete feature map and output it to the attention module. The attention module is used to redistribute the weights of each channel and space in the feature map; the maskROI module is used to input a specific image containing a region of interest into the classifier so that the classifier pays more attention to the region of interest; the feature map obtained by the attention module is multiplied by the image input by the maskROI module and output as a result.

[0045] Among them, the feature pyramid network includes bottleneck 1 layer, bottleneck 2 layer, bottleneck 3 layer, bottleneck 4 layer (4 bottleneck layers), convolution layer C2, convolution layer C3, convolution layer C4, convolution layer C5 (C5-C2 layer from bottom to top), smoothing layer and Concat layer;

[0046] The features of the image output by the Resnet50 network are generated through a feature pyramid network to produce feature maps of four different resolutions. The feature maps from bottleneck layers 1-4 are enhanced using lateral connections. Each lateral connection merges feature maps of the same spatial size from bottleneck layers 1-4 and C5-C2, resulting in feature maps of three scales. The three scale feature maps are then processed using a smoothing layer to reduce the aliasing effect of upsampling. Finally, the features of these different scales are upsampled to the same scale and then merged using a concat layer to obtain a complete feature map. Because the weights of the feature maps after direct merging are not optimal and some redundant features are not conducive to prediction, this embodiment adds an attention module and a maskROI module after the feature maps are merged.

[0047] In this embodiment, the attention module is a BAM module, which essentially combines spatial attention and channel attention in parallel. The detailed structure of the BAM module is as follows: Figure 2 As shown, for a given input feature map F∈R C×H×W , the BAM module can infer a 3D attention map M(F)∈R C×H×W , the feature map F after the BAM module redistributes the weights of each channel and space ′ The calculation method is:

[0048]

[0049] in, Indicates element-wise multiplication; this embodiment adopts a residual learning scheme with an attention mechanism to promote gradient descent. The calculation method of the 3D attention map M(F) is as follows: First, the channel attention map M on two independent branches is calculated. c (F)∈R C and spatial attention map M s (F)∈R C , and then calculate the BAM attention map M(F) according to the following formula:

[0050] M(F)=σ(M c (F)+M s (F));

[0051] Where σ is a sigmoid function so that the two independent branch outputs are rescaled to R before being added C×H×W .

[0052] In this embodiment, the maskROI module convolves the input image containing a specific region of interest to generate an attention map. The attention map is then passed through a convolution layer with a kernel size of 3×3 and a stride of 4 to obtain a feature map, which is then output. The feature map output by the maskROI module is element-wise multiplied with the feature map output by the attention module and output as the result. The purpose of adding the maskROI module is to ensure that the distribution of elements in the matrix of the ROI image after convolution is not just 0 and 1. This allows the network to focus on the diseased area of ​​the image while still extracting contextual information from the background area.

[0053] Example 2

[0054] This embodiment provides a method for classifying multimodal medical images of the prostate using the multimodal medical image classification system based on convolutional neural networks and curriculum learning as in Example 1, specifically comprising the following steps:

[0055] 1) Training image preparation:

[0056] Acquire multimodal medical images of the prostate, mark the target area in each modality of the multimodal medical images, and obtain a region of interest mask corresponding to each modality. Simultaneously, a radiologist assigns a subjective score to the target lesion area, specifically the PI-RADS score in this embodiment; and pre-check the image to obtain an objective score, specifically the Gleason grade in this embodiment.

[0057] 2) Preprocess the image obtained in step 1) using an image preprocessing module:

[0058] The registration unit registers the input multimodal medical images. The patch extraction unit uses the automatic segmentation network nnUNet to obtain the region of interest (ROI) of the prostate region in the registered image. Based on the ROI of the prostate region, the 2D patch image of the prostate region containing the lesion is extracted.

[0059] The normalization unit normalizes the 2D patch image and outputs a preprocessed image. The normalization method is as follows: for any image, the grayscale value of any pixel in the image is subtracted from the mean grayscale value of all pixels in the image, and then divided by the standard deviation of the grayscale values ​​of all pixels in the image. In this example, 112 lesion images were preprocessed to obtain 497 slices for network training.

[0060] 3) using the image obtained in step 2) to train a multi-scale convolutional neural network classifier using a curriculum-based learning method to obtain a trained classifier;

[0061] PI-RADS aims to standardize the interpretation and reporting of MRI sequences and improve the sensitivity of prostate cancer identification by introducing a 5-point strategy to represent the probability of clinically significant cancer: extremely unlikely (score 1), unlikely (score 2), equivocal (score 3), possible (score 4), and very likely (score 5) (Weinreb, JC, et al., PI-RADS prostate imaging–reporting and data system: 2015, version 2.2016.69(1): p.16-40.). Gleason grade is obtained by histopathological examination and has been used as the basis for clinical treatment. Therefore, it is used as the gold standard in this invention. The greater the difference between the PI-RADS score and the Gleason score, the more difficult it is to distinguish the Gleason grade of the lesion in the image, and vice versa. Therefore, for training data, the present invention provides MRI images of prostate cancer to radiologists in advance to obtain PI-RADS scores, and obtains Gleason grades through histopathological examination. Then, the difficulty of distinguishing images is judged based on the absolute value of the difference between the Gleason grade and the PI-RADS score, specifically:

[0062] 3-1) The training images are manually graded for Gleason level in advance, and the absolute value of the difference between the Gleason level and the PI-RADS score, Gap, is used as the evaluation criterion for the training data:

[0063] Gap=|Gleason Grade-PIRADS|;

[0064] 3-2) Reference Figure 3 The preprocessed images are fed into a multi-scale convolutional neural network classifier for training: First, training images with Gap values ​​of 0 and 1 are fed into the classifier as the first-stage training data. When the loss converges, training images with a Gap value of 2 are fed into the classifier along with the first-stage training data as the second-stage training data. When the loss converges, training images with Gap values ​​of 3 and 4 are fed into the classifier along with the second-stage training data as the third-stage training data. When the loss converges, a trained classifier is obtained. This gradually increases the learning difficulty of the network, making the training process more rational and resulting in a more accurate classifier.

[0065] 4) For multimodal medical images that need to be classified, first input them into the image preprocessing module for preprocessing, and then input them into the trained classifier obtained in step 3) for classification processing to obtain the predicted classification results of Gleason level. The whole process is referred to Figure 4 .

[0066] In this embodiment, the classifier of the present invention was used to predict the Gleason grade of prostate cancer in multi-parameter MRI images with reference to the method of Example 2, and the prediction results were compared with those of the existing algorithm based on the combination of convolutional neural network feature extraction and traditional classifier and the original Resnet method. The comparison results are shown in Table 1 below. It can be seen that the present invention can improve the Kappa value for predicting the Gleason grade of prostate cancer in multi-parameter MRI and achieve more accurate prediction and classification.

[0067] Table 1 Quantitative comparison of the results of different network models for predicting prostate cancer Gleason grade

[0068]

[0069]

[0070] Although the embodiments of the present invention have been disclosed above, they are not limited to the applications listed in the description and implementation methods. They can be fully applied to various fields suitable for the present invention. For those familiar with the art, additional modifications can be easily implemented. Therefore, without departing from the general concept defined by the claims and the scope of equivalents, the present invention is not limited to specific details.

Claims

1. A multimodal medical image classification system based on convolutional neural networks and curriculum learning, characterized by: It includes: An image preprocessing module and a multi-scale convolutional neural network classifier, wherein the multi-scale convolutional neural network classifier includes a Resnet50 network, a feature pyramid network, a maskROI module, and an attention module; The Resnet50 network extracts features of the input image and outputs them to the feature pyramid network. The feature pyramid network processes the feature map and outputs it to the attention module. The attention module is used to redistribute the weights of each channel and space in the feature map. The maskROI module is used to input a specific image containing a region of interest into the classifier so that the classifier pays more attention to the region of interest. The feature map obtained by the attention module is multiplied by the image input to the maskROI module and output as the result. The feature pyramid network includes bottleneck 1 layer, bottleneck 2 layer, bottleneck 3 layer, bottleneck 4 layer, convolution layer C2, convolution layer C3, convolution layer C4, convolution layer C5, smoothing layer and Concat layer; The features of the image output by the Resnet50 network are generated into feature maps of four different resolutions through the feature pyramid network. The feature maps from bottleneck layers 1-4 are enhanced using lateral connections. Each lateral connection merges feature maps of the same spatial size from bottleneck layers 1-4 and C5-C2 layers to obtain feature maps of three scales. The feature maps of the three scales are then processed using a smoothing layer to reduce the aliasing effect of upsampling. Finally, the features of these different scales are first upsampled to the same scale and then merged using a concat layer to obtain a complete feature map. The attention module is a BAM module, which combines spatial attention and channel attention in parallel. For a given input feature map F∈R C×H×W , the BAM module infers a 3D attention map M(F)∈R C×H×W , the calculation method of the feature map F' after the BAM module redistributes the weights of each channel and space is: in, Represents element-wise multiplication; the calculation method of the 3D attention map M(F) is: first calculate the channel attention map M on two independent branches c (F)∈R C and spatial attention map M s (F)∈R C , and then calculate the BAM attention map M(F) according to the following formula: M(F)=σ(M c (F)+M s (F)); Where σ is a sigmoid function so that the two independent branch outputs are rescaled to R before being added C ×H×W .

2. The multimodal medical image classification system based on convolutional neural network and curriculum learning according to claim 1, characterized in that: The image preprocessing module includes a registration unit, a patch extraction unit and a normalization unit.

3. The multimodal medical image classification system based on convolutional neural network and curriculum learning according to claim 2, characterized in that: The registration unit uses one modality of the input multimodal medical image as a reference image, and the images of the remaining modalities as floating images to perform registration calculations with the reference image respectively, and applies the registration transformation matrix obtained by the registration calculations to all floating images to achieve image registration; The patch extraction unit uses an automatic segmentation network to obtain a region of interest in the registered image, and extracts a 2D patch image of the region of interest containing the lesion according to the region of interest; The normalization unit performs normalization processing on the 2dpatch image and outputs a preprocessed image. The normalization processing method is: for any image, the grayscale value of any pixel of the image is subtracted from the mean of the grayscale values ​​of all pixels in the image and then divided by the standard deviation of the grayscale values ​​of all pixels in the image.

4. The multimodal medical image classification system based on convolutional neural network and curriculum learning according to claim 3, characterized in that: The maskROI module convolves the input specific image containing the region of interest to generate an attention map. The attention map obtains a feature map through a convolution layer with a convolution kernel size of 3×3 and a stride of 4 and outputs it; the feature map output by the maskROI module is element-wise multiplied with the feature map output by the attention module and output as the result.

5. The multimodal medical image classification system based on convolutional neural network and curriculum learning according to claim 4, characterized in that: The method of the system for classifying multimodal medical images includes the following steps: 1) Training image preparation: Collect multimodal medical images, mark the target area in each modality of the multimodal medical images, obtain the region of interest mask corresponding to each modality, and manually perform subjective scoring on the target lesion area; 2) Preprocessing the image obtained in step 1) by the image preprocessing module: The registration unit registers the input multimodal medical image, and the patch extraction unit extracts a 2D patch image of a region of interest containing a lesion from the registered image; the normalization unit normalizes the 2D patch image and outputs a pre-processed image; 3) using the image obtained in step 2) to train the multi-scale convolutional neural network classifier using a curriculum-based learning method to obtain a trained classifier; 4) For the multimodal medical images that need to be classified, they are first input into the image preprocessing module for preprocessing, and then input into the trained classifier obtained in step 3) for classification processing to obtain a classification result.

6. The multimodal medical image classification system based on convolutional neural network and curriculum learning according to claim 5, characterized in that: The step 3) is specifically as follows: 3-1) The training images are manually scored objectively in advance, and then the absolute value Gap between the objective score and the subjective score is used as the evaluation criterion for the training data: 3-2) Inputting the preprocessed image into the multi-scale convolutional neural network classifier for training: First, the training data is divided into N parts from easy to difficult according to the size of the Gap value, where N is not less than 2, the easiest data has the smallest Gap value, and the most difficult data has the largest Gap value; During the first training, the easiest first data set is input. When the loss converges, the second data set is input together with the first data set for the second training. Subsequent trainings gradually increase the difficulty of data. During the i-th training, the input data is the sum of the first to i-th data sets, where i = 1, 2, ..., N. When the loss converges after the N-th training, a trained classifier is obtained.

Images