A glioma image classification method and system based on multi-task learning

Abstract

The application relates to a glioma image classification method and system based on multi-task learning, which comprises the following steps: step S1, collecting multi-modal magnetic resonance imaging (MRI) image data of brain glioma, and dividing the data into a training set and a test set; step S2, constructing a multi-modal MRI image classification and segmentation joint network based on multi-task learning, wherein a loss function of the network is a weighted sum of a segmentation network loss function and a classification network loss function; the training set labeled with a tumor mask gold standard is used to train the multi-modal MRI image classification and segmentation joint network; step S3, using the trained multi-modal MRI image classification and segmentation joint network to perform image segmentation and classification on the test set, and outputting an automatic segmentation result of the multi-modal MRI image of glioma and a classification result. Compared with the prior art, the application has the advantage of high classification accuracy.

Description

Technical Field

[0001] This invention relates to the field of medical image recognition technology, and in particular to a method and system for glioma segmentation and classification based on multi-task learning. Background Technology

[0002] Gliomas are tumors originating from glial cells in the brain. They are the most common primary intracranial tumors, characterized by high incidence and mortality. The differences in survival rates and treatment responses of gliomas are primarily attributed to their genetic and histological characteristics, particularly isocitrate dehydrogenase (IDH) mutation status, 1p / 19q co-deletion status, and tumor grade. In 2016, the World Health Organization (WHO) updated its classification criteria for brain tumors, classifying gliomas based on these genetic and histological characteristics.

[0003] Currently, accurate grading and classification of gliomas can only be obtained through pathological examination, which primarily involves obtaining tissue samples via surgical biopsy. Non-invasive examinations producing multimodal MRI images (T1WI, CE-T1WI, T2WI, and T2Flair modalities, etc.) offer advantages such as rich diagnostic information and high soft tissue contrast. The method of extracting quantified features from images by applying numerous image feature algorithms to regions of interest (ROIs) and then using machine learning methods to predict histological features is known as radiomics. Applying radiomics to the accurate grading and genetic classification prediction of gliomas using multimodal MRI images can provide greater diagnostic support for preoperative treatment.

[0004] However, this approach still has some limitations. On the one hand, some radiomics work is based on manually delineated ROIs, which is time-consuming and laborious for radiologists. On the other hand, all image features are extracted through computer algorithms, which may not capture all useful features. Convolutional Neural Network (CNN)-based methods have achieved more accurate segmentation results than traditional computer vision methods, and some of these methods have been successfully used to extract deep features from images for further analysis. However, this incremental approach still has a problem: because CNN networks are trained only for segmentation, the features they extract are not optimized for classification tasks, and the final classification accuracy needs to be improved. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the existing technology by providing a glioma image classification method and system based on multi-task learning. While extracting deep features using CNN neural network, feature fusion is performed using radiomics features, and glioma segmentation and classification are performed simultaneously. This can improve the accuracy of glioma classification and generate multi-label automatic segmentation results for the segmentation task.

[0006] The objective of this invention can be achieved through the following technical solutions:

[0007] According to a first aspect of the present invention, a glioma image classification method based on multi-task learning is provided, the method comprising the following steps:

[0008] Step S1: Collect multimodal magnetic resonance imaging image data of gliomas and divide them into training set and test set;

[0009] Step S2: Construct a multimodal magnetic resonance imaging image segmentation and classification joint network based on multi-task learning. The multimodal magnetic resonance imaging image segmentation and classification joint network is trained using a training set labeled with the gold standard of tumor masks. The loss function is a weighted sum of the segmentation network loss function and the classification network loss function.

[0010] Step S3: Use the trained multimodal magnetic resonance imaging image classification and segmentation joint network to segment and classify the test set, and output the automatic segmentation results and classification results of the multimodal magnetic resonance imaging images of gliomas.

[0011] Preferably, the multimodal magnetic resonance imaging image data in step S1 includes T1WI, CE-T1WI, T2WI, and T2Flair modal magnetic resonance imaging image data.

[0012] Preferably, step S1 further includes preprocessing the multimodal magnetic resonance imaging image data of glioma, including data desensitization, resampling and skull dissection, as well as annotation processing of the tumor mask gold standard.

[0013] Preferably, the multimodal magnetic resonance imaging image classification and segmentation joint network in step S2 includes a segmentation network based on convolutional neural networks and a classification network that fuses radiomics features. The segmentation network based on convolutional neural networks includes a segmentation network encoder and a segmentation network decoder.

[0014] The segmentation and classification tasks are implemented as follows:

[0015] Multimodal magnetic resonance imaging images and their corresponding annotations are input into a segmentation network based on a convolutional neural network, which outputs multi-label segmentation results for gliomas.

[0016] The classification network fuses the shared features of the segmentation network decoder with the features extracted by radiomics methods to output glioma classification results.

[0017] Preferably, the segmentation network encoder has a U-shaped network structure, including three-dimensional convolutional layers and three-dimensional max pooling layers, and uses ReLU as the activation function.

[0018] Preferably, the segmentation network decoder specifically comprises: starting from the last layer of the segmentation network encoder, including three decoding blocks, each decoding block having three three-dimensional transposed convolutional layers, which are skip-connected to the corresponding convolutional layers output by the segmentation network decoder, and two three-dimensional convolutional layers, and finally generating segmentation prediction results by using the softmax function as the activation function.

[0019] Preferably, step S2 includes the following sub-steps:

[0020] Step S2.1: Extract radiomics features F from the multimodal magnetic resonance imaging images at the corresponding segmented regions of interest (ROIs) using radiomics methods. rad ;

[0021] Step S2.2: The classification network shares the segmentation network encoder, and performs average pooling on the output of each convolutional layer to form the one-dimensional deep feature F obtained from the segmentation. C The image omics features obtained in step S2.1 are used as a one-dimensional input vector. After applying a ReLU layer, they are combined with one-dimensional deep features F. C The layers are connected, and then a fully connected layer with a softmax function is used to obtain the classification prediction result;

[0022] Step S2.3: The weighted sum of the segmentation network loss function and the classification network loss function is used as the joint network for multimodal magnetic resonance imaging image classification and segmentation, and the network is trained accordingly.

[0023] L gross =w1L dice +w2L ce

[0024] Among them, L dice L ce , respectively, are the loss functions for the segmentation network and the classification network, and w1 and w2 are adjustable weight coefficients.

[0025] Preferably, the loss function of the segmentation network based on the convolutional neural network is the Dice loss function, expressed as:

[0026]

[0027] In the formula, u is the segmentation prediction result output by the segmentation network, v is the encoded tumor mask gold standard M label, subscript i is the voxel index, superscript k is the target sample, and K is the total number of target samples.

[0028] Preferably, the classification network uses cross-entropy as the loss function, expressed as:

[0029]

[0030] In the formula, u is the classification prediction result output by the classification network, v is the encoded gold standard label, k is the target sample, and K is the total number of target samples.

[0031] According to a second aspect of the present invention, a glioma image classification system based on multi-task learning is provided, characterized in that the system employs the method described above, the system comprising:

[0032] The image data acquisition module is used to acquire multimodal magnetic resonance imaging image data of gliomas and divide them into training set and test set;

[0033] The image segmentation and classification joint network construction and training module is used to construct a multimodal magnetic resonance imaging image segmentation and classification joint network based on multi-task learning. The multimodal magnetic resonance imaging image segmentation and classification joint network is trained using a training set labeled with the gold standard of tumor masks. The loss function is a weighted sum of the segmentation network loss function and the classification network loss function.

[0034] The image segmentation and classification module is used to segment and classify the test set using a pre-trained multimodal magnetic resonance imaging image classification and segmentation joint network, and outputs the automatic segmentation results and classification results of glioma multimodal magnetic resonance imaging images.

[0035] This invention utilizes multi-task learning combined with CNN networks to extract radiomics features from multimodal MRI images of gliomas. Simultaneously, it designs segmentation and classification networks with shared network parameters, combining radiomics features and deep network features to complete glioma segmentation and classification tasks. Compared to other multimodal MRI image segmentation and classification models, this multi-task learning method changes the step-by-step, staged single-task learning approach. It automatically segments the glioma while simultaneously optimizing the classification network using radiomics and deep network features to achieve higher classification accuracy. This provides greater assistance to doctors in clinical diagnosis and the development of personalized treatment plans for patients, demonstrating better clinical applicability.

[0036] Compared with the prior art, the present invention has the following advantages:

[0037] 1. Radiomics-based non-destructive classification and prediction of gliomas was carried out. Deep learning was used to build a classification model and extract image features, which enabled the prediction model to adapt to image noise and improve accuracy compared with traditional methods.

[0038] 2. A multi-task glioma segmentation and classification method and system based on CNN architecture is constructed. Compared with single-task learning networks, it can improve glioma segmentation and classification because multi-task learning can achieve more accurate localization and precise classification by sharing features. The segmentation task is dedicated to the localization of glioma regions, while the classification task is dedicated to the classification from glioma representation. Therefore, the localization and representation of gliomas are closely related to the classification task.

[0039] 3. The method and system of this invention can continuously iterate and optimize themselves, thereby continuously improving the prediction accuracy and calculation speed, and can also effectively help clinicians diagnose gliomas. Attached Figure Description

[0040] Figure 1 This is a flowchart of the method of the present invention;

[0041] Figure 2 This is a detailed flowchart of the method in the embodiment. Detailed Implementation

[0042] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0043] like Figure 1 As shown, this invention presents a glioma image classification method based on multi-task learning, which includes the following steps:

[0044] Step S1: Collect multimodal magnetic resonance imaging image data of gliomas and divide them into training set and test set;

[0045] Step S2: Construct a multimodal magnetic resonance imaging image segmentation and classification joint network based on multi-task learning. The multimodal magnetic resonance imaging image segmentation and classification joint network is trained using a training set labeled with the gold standard of tumor masks. The loss function is a weighted sum of the segmentation network loss function and the classification network loss function.

[0046] Step S3: Use the trained multimodal magnetic resonance imaging image classification and segmentation joint network to segment and classify the test set, and output the automatic segmentation results and classification results of the multimodal magnetic resonance imaging images of gliomas.

[0047] Next, combined Figure 2 The specific embodiments of the present invention will be described in detail below.

[0048] A glioma image segmentation and classification method based on multi-task learning, comprising the following steps:

[0049] Step M1: Collect and acquire multimodal MRI image data of gliomas (including: T1WI, CE-T1WI, T2WI, T2Flair, etc.) and perform data preprocessing. Through data desensitization, resampling, and skull dissection, obtain multimodal MRI data of gliomas with uniform resolution and approximately the same grayscale distribution, as I. Glioma Furthermore, the size of the .nii.gz data for each modality is 256×256×16.

[0050] In addition, each case includes a gold standard tumor mask corresponding to the MRI image, manually annotated by a radiologist, denoted as M; the size of each modality's .nii.gz data is 256×256×16.

[0051] In addition, each case also includes the corresponding glioma IDH gene typing results, including both wild-type and mutant IDH types.

[0052] Pretreatment includes the marking of the brain tumor's ROI by a radiologist.

[0053] Step M2: Construct a joint network for multimodal MRI brain tumor segmentation and classification to complete the segmentation and classification tasks. Multimodal MRI image I Glioma The corresponding M and IDH genotyping results were fed into the multimodal MRI segmentation and classification network constructed in this invention for feature extraction and feature fusion.

[0054] A CNN encoder for multimodal MRI glioma segmentation. The network consists of a U-shaped structure, including 3×3×3 three-dimensional convolutional layers and 3-dimensional max-pooling layers, using ReLU as the activation function.

[0055] Step M3: The segmentation network uses the gold standard M obtained by the CNN encoder to construct the relationship between the region of interest (ROI) and the background, and optimizes the network through a loss function.

[0056] The Dice loss function of the segmentation network is constructed using the gold standard M for gliomas obtained from the segmentation network, as shown in the following equation:

[0057]

[0058] Where u is the segmentation prediction result output by the segmentation network, v is the encoded gold standard label, i is the voxel index, and k is the target sample.

[0059] Step M4: The segmentation network decoder outputs the segmentation results of the glioma multi-label segmentation, specifically:

[0060] The segmentation network decoder, starting from the last layer of the encoder, includes three decoding blocks. Each decoding block has three 3D transposed convolutional layers, which are skipped to the corresponding convolutional layers at the decoder output, as well as a 3×3×3 3D convolutional layer and a final 1×1×1 3D convolutional layer. Finally, the segmentation prediction result is generated by using the softmax function as the activation function.

[0061] Step M5: The segmentation network decoder shares feature parameters and fuses them with features extracted from radiomics to generate the results of the glioma IDH binary classification network. Specifically, a multimodal MRI image glioma segmentation network using a CNN architecture and a glioma IDH binary classification network incorporating radiomics features are used to generate automatic segmentation results for multimodal MRI gliomas while simultaneously performing IDH gene binary classification of multimodal glioma images.

[0062] Step M5.1: Extract radiomics features F from multimodal MRI images at corresponding segmented ROIs using radiomics methods. rad The radiomics feature parameters that need to be calculated include: basic image grayscale features, image shape and volume features, image grayscale features, and co-occurrence matrix texture features. Basic image grayscale features include: Maximum, Minimum, Median, Range, and Mean. Image shape and volume features include: Maximum 3D diameter, tumor surface area, Volume, Compactness, Spherical disproportion, Sphericity, and Surface to volume ratio. Image grayscale features and co-occurrence matrix texture features include: Entropy, Homogeneity, and Sum Entropy.

[0063] Step M5.2: The CNN segmentation encoder in step M2 is shared by the glioma classification network. The output of each convolutional layer is averaged and pooled to form the one-dimensional deep feature F obtained from the segmentation. C The image omics features obtained in step M5.1 are used as a one-dimensional input vector. After applying a ReLU layer, they are combined with the deep features F. C The layers are connected, and then a fully connected layer with a softmax function is used to obtain the IDH classification prediction results.

[0064] Step M5.3: The glioma classification network uses cross-entropy as the loss function, as shown in the following equation:

[0065]

[0066] Where u is the glioma IDH gene prediction result output by the classification network, v is the encoded patient's actual IDH gene typing result, and k is the target sample.

[0067] Step M5.4: The loss function used when training the multi-task network is the weighted sum of the segmentation network loss function and the glioma IDH binary classification network loss function, as shown in the equation:

[0068] L gross =w1L dice +w2L ce

[0069] Here, e1 and w2 are adjustable weighting coefficients.

[0070] Next, a system embodiment of the present invention is given: a glioma image classification system based on multi-task learning, employing the above-described method, the system comprising:

[0071] The image data acquisition module is used to acquire multimodal magnetic resonance imaging image data of gliomas and divide them into training set and test set;

[0072] The image segmentation and classification joint network construction and training module is used to construct a multimodal magnetic resonance imaging image segmentation and classification joint network based on multi-task learning. The multimodal magnetic resonance imaging image segmentation and classification joint network is trained using a training set labeled with the gold standard of tumor masks. The loss function is a weighted sum of the segmentation network loss function and the classification network loss function.

[0073] The image segmentation and classification module is used to segment and classify the test set using a pre-trained multimodal magnetic resonance imaging image classification and segmentation joint network, and outputs the automatic segmentation results and classification results of glioma multimodal magnetic resonance imaging images.

[0074] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A glioma image classification method based on multi-task learning, characterized in that, The method comprises the following steps: Step S1, collecting multi-modal magnetic resonance imaging image data of brain glioma, and dividing the data into a training set and a test set; Step S2, constructing a multi-modal magnetic resonance imaging image segmentation and classification joint network based on multi-task learning, and training the multi-modal magnetic resonance imaging image classification and segmentation joint network by using the training set labeled with tumor mask gold standard; wherein the loss function is a weighted sum of the segmentation network loss function and the classification network loss function; The segmentation network encoder is a U-shaped network structure, comprising a three-dimensional convolution layer and a three-dimensional maximum pooling layer, and using ReLu as an activation function; The segmentation network decoder specifically comprises: starting from the last layer of the segmentation network encoder, comprising three decoding blocks, each decoding block having three three-dimensional transpose convolution layers, performing skip connection with the corresponding convolution layer output by the segmentation network decoder, and two three-dimensional convolution layers, and finally generating a segmentation prediction result through a softmax function as an activation function; The step S2 comprises the following sub-steps: Step S2.1, extracting radiomics features on the corresponding segmented region of interest (ROI) on the multi-modal magnetic resonance imaging images using radiomics method ; Step S2.2, the classification network shared segmentation network encoder, average pooling is performed on the output of each convolutional layer to form a one-dimensional deep feature obtained by segmentation The radiomics features obtained in step S2.1 are applied to the ReLu layer as a one-dimensional input vector, and then combined with the one-dimensional deep feature are connected, and then a full connection layer with a softmax function is used to obtain a classification prediction result; Step S2.3, using the weighted sum of the segmentation network loss function and the classification network loss function as the multi-modal magnetic resonance imaging image classification and segmentation joint network, and training the network: wherein, are a segmentation network loss function and a classification network loss function, respectively, and are adjustable weight coefficients; Step S3, using the trained multi-modal magnetic resonance imaging image classification and segmentation joint network to perform image segmentation and classification on the test set, and outputting the automatic segmentation result and the classification result of the multi-modal magnetic resonance imaging image of brain glioma.

2. The glioma image classification method based on multi-task learning according to claim 1, characterized in that, The multi-modal magnetic resonance imaging image data in the step S1 comprises T1WI, CE-T1WI, T2WI, and T2Flair modal magnetic resonance imaging image data.

3. The glioma image classification method based on multi-task learning according to claim 1, characterized in that, The step S1 further comprises preprocessing of the multi-modal magnetic resonance imaging image data of brain glioma, including data desensitization, resampling, and skull stripping, and labeling processing of the tumor mask gold standard.

4. The method of claim 3, wherein the method is based on multi-task learning. The multi-modal magnetic resonance imaging image classification and segmentation joint network in the step S2 comprises a segmentation network based on a convolutional neural network, and a classification network fusing imageomic features, wherein the segmentation network based on the convolutional neural network comprises a segmentation network encoder and a segmentation network decoder; The segmentation and classification tasks are implemented as follows: The multi-modal magnetic resonance imaging image and the corresponding label are input into the segmentation network based on the convolutional neural network, and a multi-label segmentation result of brain glioma is output; The classification network fuses the shared features of the segmentation network decoder and the features extracted by the imageomic method, and outputs a glioma classification result.

5. The glioma image classification method based on multi-task learning according to claim 1, characterized in that, The loss function of the segmentation network based on the convolutional neural network is a Dice loss function, and the expression is: wherein, is the segmentation prediction output by the segmentation network, is the encoded tumor mask ground truth label, subscript is the index of the voxel, superscript is the target sample, is the total number of target samples.

6. The glioma image classification method based on multi-task learning according to claim 1, characterized in that, The classification network uses cross-entropy as the loss function, and the expression is: In the formula, is a classification prediction result of the classification network output, is an encoded gold standard label, is a target sample, is the total number of target samples. 7.A glioma image classification system based on multi-task learning, characterized in that, The method of claim 1, the system comprises: An image data acquisition module for acquiring multi-modal magnetic resonance imaging image data of brain glioma, and dividing the data into a training set and a test set; The image segmentation and classification combined network construction and training module is used for constructing a multi-modal magnetic resonance imaging image segmentation and classification combined network based on multi-task learning, and training the multi-modal magnetic resonance imaging image classification and segmentation combined network by using a training set labeled with a tumor mask gold standard; wherein, the loss function is a weighted sum of a segmentation network loss function and a classification network loss function; The image segmentation and classification module is used for performing image segmentation and classification on a test set by using the trained multi-modal magnetic resonance imaging image classification and segmentation combined network, and outputting a glioma multi-modal magnetic resonance imaging image automatic segmentation result and a classification result.

Images