A method for constructing a tumor classification system based on deep learning

Abstract

The present invention relates to a method for constructing a tumor classification system based on deep learning, which belongs to the fields of computer vision and deep learning. The method includes selecting confirmed PCNSL and GBM radiographic images and dividing them into training and test sets, performing data augmentation on the training set, improving the Swin-Transformer model, training the improved Swin-Transformer model with data from the augmented training set, and then inputting the images from the test set into the trained model to verify the recognition effect. This method, based on a non-invasive medical imaging diagnostic method, can alleviate patient suffering while providing doctors with accurate imaging diagnostic evidence, and has important scientific significance and medical value.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and deep learning, and specifically relates to a method for constructing a tumor classification system based on deep learning. Background Art

[0002] The greatest advances in medicine in the 20th century were the rise of organ transplantation and minimally invasive surgery. The 21st century will be marked by the maturation of precision medicine based on breakthroughs in molecular biology and the penetration of artificial intelligence (AI) into all areas of medicine.

[0003] Primary central nervous system lymphoma (PCNSL) is a type of extranodal non-Hodgkin's lymphoma that originates within the central nervous system. It is a rare clinical disease with nonspecific clinical manifestations, making it prone to misdiagnosis. PCNSL patients typically undergo stereotactic biopsy followed by high-dose methotrexate-based chemotherapy. Glioblastoma (GBM) is the most malignant brain glioma, with a median survival of only approximately one year, making it one of the most morbidly fatal tumors in the human body. GBM patients are typically treated with complete resection followed by radiotherapy and temozolomide chemotherapy. With current advances in medical imaging, these two tumors exhibit many similar imaging features on conventional magnetic resonance imaging (MRI) modalities, such as T1W, T2W, and T2 FLAIR. This makes it difficult for the human eye to accurately distinguish between the two tumors, and incorrect imaging diagnoses can result in significant treatment costs. Therefore, current diagnosis of these two tumors relies on multiple lumbar punctures to obtain free tumor cells for identification, which not only causes great pain to the patient but also delays the optimal treatment time. Furthermore, lumbar punctures are difficult for doctors to perform, and repeated lumbar punctures can easily lead to misdiagnosis.

[0004] If an accurate diagnosis could be made in advance through non-invasive medical imaging, it would greatly alleviate the patient's suffering. Studies have also shown that PCNSL and GBM can be differentiated using multi-parameter magnetic resonance imaging (MRI). However, this differentiation method has limitations: Firstly, advanced MRI equipment is not widely available, and these imaging modalities are primarily used in experimental research, while clinical examinations still rely primarily on conventional MRI (T1W, T2W, and FLAIR). Secondly, multi-parameter MRI methods also rely on the experience of clinicians, with previous studies reporting clinician accuracy ranging from 62.3% to 86.9%.

[0005] This patent introduces machine learning into the problem of differentiating between two malignant tumors. By constructing a Balance-Transformer network, the differences in the imaging features of the two tumors are calculated to classify the two tumors. Doctors only need to input the patient's MRI images to obtain auxiliary diagnostic results for differentiating between the two tumors before treatment, providing doctors with accurate and reliable diagnostic basis. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide a method for constructing a tumor classification system based on deep learning, using a Transformer-type network to extract tumor category features, and making corresponding improvements to the Transformer network structure and loss function based on the specific problems of this tumor in computer vision. This application proposes a new Transformer-type network, namely Balance-Transformer. The network model of this application achieved an accuracy of 99.86% on the test set, which is 5-10% higher than other networks, and can fully reach the level of clinical application, realizing fast and accurate tumor classification.

[0007] The technical solutions of the present invention are as follows:

[0008] A method for constructing a tumor classification system based on deep learning, the method comprising the following steps:

[0009] S1: Confirmed PCNSL radiographic images and GBM radiographic images were selected, with no less than 800 images of each type. Each type of radiographic image was divided into a training set and a test set in a ratio of 7:3;

[0010] S2: Perform data augmentation on the training set in S1, including image scaling, random image flipping, random image cropping, image data normalization, Mixup, Cutout, and CutMix.

[0011] S3: Based on the Swin-Transformer model, the three modules of data sampling, feature extraction, and loss function in the Swin-Transformer network are improved to achieve higher accuracy and precision. The balancing process is as follows: Patch-Balance replaces the feature extraction module; Sample-Balance replaces the data sampling module; and Balanced Loss replaces the loss function module. Patch-Balance changes the extraction method of patches in the Swin-Transformer model from uniform sampling to interval sampling, Sample-Balance weights the data types in the Swin-Transformer model, and Balanced Loss uses Label Smoothing to regularize the loss function.

[0012] S4: Use the data in the training set after data augmentation in S2 to train the improved Swin-Transformer model in S3, and then input the images of the test set into the trained model to verify the recognition effect.

[0013] In the method for constructing a tumor classification system based on deep learning of the present invention, step S3 specifically includes:

[0014] S31: Build a basic Swin-Transformer network model;

[0015] S32: In the data sampling module of the Swin-Transformer network model, Sample-Balance is used instead of the uniform reading method by category, and the number of pictures read is distributed according to the number of pictures in each data category; in the feature extraction part after reading the picture, the Patch-Balance module is used to replace the original data processing layer PatchPartiton Block and the data aggregation layer PatchMerging Block of the Swin-Transformer architecture. The Patch-Balance module extracts the patch by interval sampling.

[0016] Furthermore, in step S32, Swin-Transformer is used as the backbone network, and the balanced loss function is used instead of the traditional cross entropy loss function, that is, the cross entropy function after label smoothing is used as the loss function of the output layer to construct a deep learning tumor classification system; the deep learning tumor classification system includes a data processing layer Patch Partiton Block, a linear encoding layer Linear Enbedding Block, a shift window-based Transformer layer Swin Transformer Block, and finally a fully connected layer that outputs the classification result. Its operation steps are as follows:

[0017] S321: Select an image from the training set and input it into the data processing layer. The selected image is denoted as P0 and the image size is denoted as H×W×3. After the data processing layer, the output size is The image is recorded as P1, and P1 is sent to the linear coding layer;

[0018] S322: P1 is encoded using the Patch-Balance method, fed into the Transformer layer based on the shift window, and output through the fully connected layer , denoted as P2, where C is the number of heads in the multi-head attention mechanism;

[0019] S323: Repeat step S322 three times to obtain the sizes The image is denoted as P3 and the size is The image is denoted as P4 and the size is The obtained P5 is sent to the fully connected layer, and the SoftMax function is operated on it to obtain the classification confidence rate of the tumor.

[0020] Furthermore, the last layer of the image classification network is a fully connected layer, and its output is a C-dimensional vector, where C is the number of categories. In this paper, two types of tumors are classified, so C = 2; the data in the vector is in the form of z i To express, z i Considered as a random variable, the probability of its output is q i By z i Calculated by the SoftMax function:

[0021]

[0022] The Label Smooth loss function is used for classification in the output layer. The Label Smooth Loss loss function is improved from the cross entropy function. The specific expression is shown in formulas (1) to (3).

[0023]

[0024]

[0025]

[0026] Where var is a hyperparameter, p i is the confidence probability of each type of tumor, q i is the probability of label output, z i is the predicted probability distribution, a is an arbitrary integer; formula (2) is the cross entropy loss function formula Loss before improvement, the original label is replaced by the smoothed label, (3) is the improved loss function formula L LS .

[0027] Compared with the prior art, the present invention has the following beneficial effects:

[0028] The proposed method for intelligent GBM and PCNSL tumor classification combines medical imaging with machine learning methods to construct a Balance-Transformer network for automated tumor lesion detection. This patented method, based on non-invasive medical imaging, can alleviate patient suffering while providing doctors with accurate imaging diagnostic evidence, possessing significant scientific and medical value. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is a network structure diagram of the method for constructing a tumor classification system based on deep learning described in the present invention.

[0030] Figure 2 This is a flowchart of the method for constructing a tumor classification system based on deep learning described in the present invention.

[0031] Figure 3 This is a block diagram of the layer model of Patch-Balance described in the present invention.

[0032] Figure 4 This is the ROC diagram of the tumor classification method of the present invention.

[0033] Figure 5 This is a heat map of the tumor classification method described in the present invention.

[0034] Figure 6 This is a comparison chart of the data enhancement method described in the present invention.

[0035] Figure 7 This is a result diagram of the tumor classification method described in the present invention. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0037] Example 1

[0038] See also Figure 1 、 Figure 2 The network structure diagram and flow chart of the method for constructing a tumor classification system based on deep learning are shown, which specifically include the following steps:

[0039] S1: Several confirmed PCNSL radiographs and GBM radiographs were selected as training and test data. These images were divided into training and test sets using a 7:3 data split. The specific number of images in each dataset is shown in Table 1.

[0040] Table 1 The number of dataset types constructed and the number of images in each dataset

[0041]

[0042] S2: Optimize the Swin-Transformer network by balancing data sampling, loss function, and data augmentation to achieve higher accuracy and precision. Then perform data augmentation on the data obtained in S1, including traditional data augmentation methods such as image resizing, random image flipping, random image cropping, and image data normalization, as well as new data augmentation methods such as Mixup, Cutout, and CutMix. We also improve the situation of data imbalance and balance the data of different categories.

[0043] S3: After training the model in S2, the test set images are input into the model designed by the present invention to perform automatic recognition to verify the recognition effect.

[0044] In S2, the constructed model includes:

[0045] S21: Build a basic Swin-Transformer network model;

[0046] S22: In the data sampling part, Sample-Balance is used instead of the uniform reading method by category. The number of images read is distributed according to the number of images in each data category. The specific operation is to select a parameter, denoted as sp(sample_persent), and weight the probability of occurrence of different categories. Since the amount of information in a uniform distribution is the largest, the parameter sp is set to the ratio of the sum of the two image data, where P sumis the number of pictures in the entire training set, P i is the number of pictures in the i-th category, and the specific formula is as follows:

[0047]

[0048] After reading the image, the feature extraction part uses the Patch-Balance module to replace the original data processing layer (Patch Partiton Block) and data aggregation layer (Patch Merging Block) of the Swin-Transformer architecture. The Patch-Balance module extracts the Patch interval by sampling. The specific operation can be seen in Figure 1 .

[0049] In S22, a deep learning tumor classification system is constructed using Balance-Transformer as the backbone network and the cross entropy function after label smoothing as the loss function of the output layer, including: building a deep learning tumor semantic segmentation model in a Linux system environment, and performing training and verification. The deep learning tumor classification system includes a data processing layer (Patch Partiton Block), a linear encoding layer (Linear Enbedding Block), a shift window-based Transformer layer (Swin Transformer Block), and finally a fully connected layer that outputs the classification results. The operation steps are as follows:

[0050] S221: Select an image from the training set and input it into the data processing layer. The selected image is denoted as P0 and the image size is denoted as H×W×3. After the data processing layer, the output size is The image is recorded as P1, and P1 is sent to the linear coding layer;

[0051] S222: Encode P1 using the Patch-Balance method, such as Figure 3 , sent to the Transformer layer based on the displacement window, and output through the fully connected layer , denoted as P2, where C is the number of heads in the multi-head attention mechanism;

[0052] S223: Repeat the method in step S222, and repeatedly perform the operation in step S222 on P2 three times, and the sizes can be obtained respectively. The image is denoted as P3 and the size is The image is denoted as P4 and the size is The obtained P5 is sent to the fully connected layer, and the SoftMax function is performed on it to obtain the classification confidence rate of the tumor.

[0053] The last layer of the image classification network is the fully connected layer, and its output size is a C-dimensional vector, where C is the number of categories. In this paper, two types of tumors are classified, so C = 2. The data in the vector is in the form of z i To express. So z i It can be regarded as a random variable. The probability of output q i Then z i Calculated by the SoftMax function:

[0054]

[0055] Ideal z i As shown below:

[0056]

[0057] After calculation, we know that when i∈class, then q i =1, otherwise q i = 0. However, since +∞ cannot be obtained, the ideal prediction probability distribution can never be obtained, so the present invention chooses to regularize the labels.

[0058] For small sample data, the present invention uses the Label Smooth Loss loss function for classification in the output layer. The Label Smooth Loss loss function is improved from the cross entropy function, and its specific expression is shown in formulas (1) to (4). var is a hyperparameter, p i is the confidence probability of each type of tumor, q i is the probability of the expected output, z i is the optimal prediction probability distribution, and a is an arbitrary integer. Formula (2) is the cross entropy loss function formula Loss before improvement. Replacing the original label with the smoothed label can achieve the purpose of regularization, further increasing the intra-class compactness and inter-class distance. (3) is the improved loss function formula L LS . Among them (4) is the ideal z after improvement i Distribution.

[0059]

[0060]

[0061]

[0062]

[0063] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for constructing a tumor classification system based on deep learning, comprising the following steps: S1: Confirmed PCNSL radiographic images and GBM radiographic images were selected, with no less than 800 images of each type. Each type of radiographic image was divided into a training set and a test set in a ratio of 7:3; S2: Perform data augmentation on the training set in S1, including image scaling, random image flipping, random image cropping, image data normalization, Mixup, Cutout, and CutMix. S3: Improves on the Swin-Transformer model by balancing the data sampling, feature extraction, and loss function modules in the Swin-Transformer network to achieve higher accuracy and precision. The balancing process is as follows: replacing the feature extraction module with Patch-Balance; The data sampling module is replaced by Sample-Balance, and the loss function module is replaced by Balanced Loss. Patch-Balance changes the patch extraction method in the Swin-Transformer model from uniform sampling to interval sampling. Sample-Balance weights the data types in the Swin-Transformer model. Balanced Loss uses Label Smoothing to regularize the loss function. S4: Use the data in the training set after data augmentation in S2 to train the improved Swin-Transformer model in S3, and then input the images of the test set into the trained model to verify the recognition effect.

2. The method for constructing a tumor classification system based on deep learning according to claim 1, characterized in that: Step S3 specifically includes: S31: Build a basic Swin-Transformer network model; S32: In the data sampling module of the Swin-Transformer network model, Sample-Balance is used instead of the uniform reading method by category, and the number of pictures read is distributed according to the number of pictures in each data category; in the feature extraction part after reading the picture, the Patch-Balance module is used to replace the original data processing layer PatchPartiton Block and the data aggregation layer PatchMerging Block of the Swin-Transformer architecture. The Patch-Balance module extracts the patch by interval sampling.

3. The method for constructing a tumor classification system based on deep learning according to claim 2, characterized in that: In step S32, a deep learning tumor classification system is constructed using Swin-Transformer as the backbone network and using the Balanced Loss loss function instead of the traditional cross entropy loss function. That is, the cross entropy function after label smoothing is used as the loss function of the output layer. The deep learning tumor classification system includes a data processing layer Patch PartitonBlock, a linear encoding layer Linear Enbedding Block, a displacement window-based Transformer layer SwinTransformer Block, and finally a fully connected layer that outputs the classification results. The operation steps are as follows: S321: Select an image from the training set and input it into the data processing layer. The selected image is denoted as P0 and the image size is denoted as H×W×3. After the data processing layer, the output size is The image is recorded as P1, and P1 is sent to the linear coding layer; S322: P1 is encoded using the Patch-Balance method, fed into the Transformer layer based on the shift window, and output through the fully connected layer , denoted as P2, where C is the number of heads in the multi-head attention mechanism; S323: Repeat step S322 three times to obtain the sizes The image is denoted as P3 and the size is The image is denoted as P4 and the size is The obtained P5 is sent to the fully connected layer, and the SoftMax function is operated on it to obtain the classification confidence rate of the tumor.

4. The method for constructing a tumor classification system based on deep learning according to claim 3, characterized in that: The last layer of the image classification network is the fully connected layer, and its output is a C-dimensional vector, where C is the number of categories. In this paper, two types of tumors are classified, so C = 2; the data in the vector is in the form of z i To express, z i Considered as a random variable, the probability of its output is q i By z i Calculated by the SoftMax function: The Label Smooth loss function is used for classification in the output layer. The Label Smooth Loss loss function is improved from the cross entropy function. The specific expression is shown in formulas (1) to (3). Where var is a hyperparameter, p i is the confidence probability of each type of tumor, q i is the probability of label output, z i is the predicted probability distribution, a is an arbitrary integer; formula (2) is the cross entropy loss function formula Loss before improvement, the original label is replaced by the smoothed label, (3) is the improved loss function formula L LS .

Images