A method and system for classifying intracranial tumor cells based on deep learning

Abstract

The application discloses a kind of based on deep learning's intracranial tumor cell classification method and system, including obtaining the whole slice scanning pathology image of section, by three-step image preprocessing method obtains the small block of whole slice scanning pathology image, utilize the feature of small block that is extracted using the neural network model based on deep learning of preestablished, and according to the feature of small block Prediction block classification result, according to block classification result, predict the type of intracranial tumor cell, by based on whole slice scanning pathology image, the type of intracranial tumor cell is predicted, it is advantageous to assist physician to predict intracranial tumor cell and carry out prognosis scheme formulation, improve the classification accuracy and work efficiency of physician to intracranial tumor cell.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of picture recognition, and in particular to a method and system for classifying intracranial tumor cells based on deep learning. BACKGROUND

[0002] Currently, more than 300,000 patients worldwide are newly diagnosed with central nervous system tumors, of which about 250,000 patients die of the disease. Oligodendroglioma and astrocytoma are the most common pathological types of glioma. Central nervous system germ cell tumor is the third most common central nervous system tumor, which is mainly common in children, and intracranial germ cell tumor (IG) is the most common histological subtype. Although IG, oligodendroglioma and astrocytoma all belong to the category of central nervous system tumors and have similar cell morphology, their treatment regimens and prognosis differ greatly. IG has high radiotherapy sensitivity, and the treatment regimen is standard treatment with whole brain radiotherapy plus local boost. Oligodendroglioma and astrocytoma are first choice for surgery, and then choose radiotherapy or / and chemotherapy regimen according to the grade of tumor. IG patients have a good prognosis, with a 5-year overall survival rate (0S) of more than 90%, while the 5-year 0S rate of astrocytoma and oligodendroglioma is between 48%-66%. However, unnecessary IG resection due to misdiagnosis may damage the brain's neurocognitive function, and insufficient lesion resection due to misdiagnosis may increase the risk of recurrence.

[0003] Accurate diagnosis of IG, oligodendroglioma and astrocytoma is very important, which is mainly based on three examinations: intraoperative frozen section (frozen section) examination, tumor section examination based on hematoxylin-eosin (HE) staining and molecular detection. Molecular detection can identify diagnosis by immunohistochemistry (IHC) and gene sequencing, and biopsy specimens with pathological molecular detection are considered the gold standard for diagnosing IG, oligodendroglioma and astrocytoma. However, molecular detection is both complex and time-consuming (usually more than a week), so it cannot be used for classification diagnosis intraoperatively or early postoperatively. Clinically, pathological examination based on frozen section and HE-stained section is a better diagnostic method. Frozen section can provide a benign or malignant diagnostic tendency within a few minutes during surgery, while HE-stained section has better diagnostic value than frozen section, with clearer cell structure, and can obtain a diagnostic conclusion within 3-5 working days after surgery combined with immunohistochemistry, so as to follow-up regimen. Therefore, intraoperative and postoperative decisions for IG, oligodendroglioma and astrocytoma mainly rely on pathological diagnosis of frozen section and HE-stained section. However, due to similar cell morphology, even experienced pathologists cannot accurately distinguish the morphological characteristics of IG, oligodendroglioma and astrocytoma based on frozen section and HE-stained section alone. SUMMARY

[0004] The present application aims to at least solve the technical problems existing in the prior art. To this end, the present application provides a deep learning-based classification method and system for intracranial tumor cells, which is beneficial to assisting physicians in predicting intracranial tumor cells and making prognosis plans, and improves the classification accuracy and work efficiency of physicians for intracranial tumor cells.

[0005] In a first aspect, the present application provides a deep learning-based classification method for intracranial tumor cells, comprising the following steps:

[0006] Obtaining a full-scan pathology image of a slice;

[0007] Obtaining small patches of the full-scan pathology image through a three-step image preprocessing method;

[0008] Extracting features of the small patches using a preset deep learning-based neural network model, and predicting patch classification results according to the features of the small patches;

[0009] According to the patch classification results, predicting the type of the intracranial tumor cells.

[0010] According to the embodiments of the present application, at least the following technical effects are achieved:

[0011] The present method obtains a full-scan pathology image of a slice, obtains small patches of the full-scan pathology image through a three-step image preprocessing method, extracts features of the small patches using a preset deep learning-based neural network model, and predicts patch classification results according to the features of the small patches. According to the patch classification results, the type of the intracranial tumor cells is predicted. By predicting the type of the intracranial tumor cells based on the full-scan pathology image, it is beneficial to assisting physicians in predicting intracranial tumor cells and making prognosis plans, and improves the classification accuracy and work efficiency of physicians for intracranial tumor cells.

[0012] According to some embodiments of the present application, the slice includes a frozen slice and an HE-stained slice.

[0013] According to some embodiments of the present application, the small patches of the full-scan pathology image are obtained through a three-step image preprocessing method, comprising:

[0014] Converting the full-scan pathology image into a grayscale image, and segmenting the tissue region on the grayscale image with 220 as the threshold value;

[0015] Dividing the tissue region into 224x224 patches with a resolution of 0.5 microns per pixel and without overlap;

[0016] inputting the 224*224 patch into a preset deep learning-based tissue slice selection model to obtain a small patch of the whole slide scanning pathology image output by the deep learning-based tissue slice selection model.

[0017] According to some embodiments of the present application, after obtaining the small patch of the whole slide scanning pathology image by the three-step image preprocessing method, the method further comprises the steps of:

[0018] randomly performing one or a combination of horizontal flipping, vertical flipping, modifying brightness, and horizontal flipping according to a probability P1 on the small patch.

[0019] According to some embodiments of the present application, the step of extracting the features of the small patch using the preset deep learning-based neural network model and predicting the patch classification result according to the features of the small patch comprises:

[0020] inputting the small patch into an improved resnet18 network model to generate a feature map A, wherein the improved resnet18 network model is obtained by deleting the network part after the average pooling layer of the resnet18 network model;

[0021] inputting the feature map A into an ARPY network model to output the position information of the key features, wherein the ARPY network model is composed of two convolution layers with a convolution kernel of 3*3 connected to each other and two fully connected layers connected in sequence;

[0022] cropping the tissue region corresponding to the key features and resetting the size to 224*224, and then inputting the cropped tissue region into the improved resnet18 network model after random occlusion to obtain a feature map B;

[0023] performing a splicing operation on the feature map A and the feature map B, and then sequentially passing through a fully connected layer FC1, a Dropout layer with a probability of P2, a fully connected layer FC2, a BN layer, a fully connected layer FC3, a relu activation function layer, a fully connected layer FC4, and a softmax function to predict the class probability value of the feature region of the small patch;

[0024] obtaining the patch classification result according to the class probability value.

[0025] According to some embodiments of the present application, the patch classification result comprises an intracranial germinoma patch, an oligodendroglioma patch, and an astrocytoma patch.

[0026] According to some embodiments of the present application, the step of predicting the type of the intracranial tumor cells according to the patch classification result comprises:

[0027] predict the type of the intracranial tumor cells from the tile classification results by majority voting.

[0028] In a second aspect, the present application provides a deep learning-based classification system for intracranial tumor cells, comprising:

[0029] a data acquisition module configured to acquire a whole-slide scanning pathology image of a section;

[0030] a data preprocessing module configured to obtain small tiles of the whole-slide scanning pathology image by a three-step image preprocessing method;

[0031] a tile classification module configured to extract features of the small tiles by using a preset deep learning-based neural network model, and predict tile classification results according to the features of the small tiles;

[0032] a type prediction module configured to predict the type of the intracranial tumor cells according to the tile classification results.

[0033] The system acquires a whole-slide scanning pathology image of a section, obtains small tiles of the whole-slide scanning pathology image by a three-step image preprocessing method, extracts features of the small tiles by using a preset deep learning-based neural network model, predicts tile classification results according to the features of the small tiles, and predicts the type of the intracranial tumor cells according to the tile classification results. The type prediction of the intracranial tumor cells based on the whole-slide scanning pathology image is conducive to assisting doctors in predicting the intracranial tumor cells and making prognosis plans, and improves the classification accuracy and work efficiency of the doctors for the intracranial tumor cells.

[0034] In a third aspect, the present application provides a deep learning-based classification electronic device for intracranial tumor cells, comprising at least one control processor and a memory in communication connection with the at least one control processor; the memory stores instructions executable by the at least one control processor, and the instructions are executed by the at least one control processor to enable the at least one control processor to perform the above-mentioned deep learning-based classification method for intracranial tumor cells.

[0035] In a fourth aspect, the present application provides a computer-readable storage medium storing computer-executable instructions for causing a computer to perform the above-mentioned deep learning-based classification method for intracranial tumor cells.

[0036] It should be noted that the beneficial effects of the second aspect to the fourth aspect of the present application and the prior art are the same as those of the above-mentioned deep learning-based classification system for intracranial tumor cells and the prior art, which will not be described here.

[0037] Additional aspects and advantages of the present application will be apparent from the following description of the application and as defined in the appended claims. BRIEF DESCRIPTION OF DRAWINGS

[0038] The above and / or additional aspects and advantages of the present application will become apparent and be readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings.

[0039] Figure 1 is a flowchart of a classification method of intracranial tumor cells based on deep learning according to an embodiment of the present application;

[0040] Figure 2 is a flowchart of step S102 in Figure 1

[0041] Figure 3 is a flowchart of step S103 in Figure 1

[0042] Figure 4 is a flowchart of step S104 in Figure 1

[0043] Figure 5 is a flowchart of a classification system of intracranial tumor cells based on deep learning according to an embodiment of the present application. DETAILED DESCRIPTION

[0044] Embodiments of the present application are described in detail below with reference to the attached drawings, which are presented as examples and are not intended to limit the present application.

[0045] In the description of the present application, if there is a description to first, second, etc., it is only for the purpose of distinguishing technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features or the order of the indicated technical features.

[0046] In the description of the present application, it is to be understood that the orientation description, such as up, down, etc., indicates the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, only for the purpose of facilitating the description of the present application and simplifying the description, and is not to indicate or imply that the device or element indicated must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation of the present application.

[0047] ​​​In the description of this invention, it should be noted that, unless otherwise explicitly defined, terms such as "setting," "installation," and "connection" should be interpreted broadly, and those skilled in the art can reasonably determine the specific meaning of the above terms in this invention in conjunction with the specific content of the technical solution.

[0048] Currently, intraoperative and postoperative decisions for igstroma, oligodendroglioma, and astrocytoma rely primarily on pathological diagnosis using frozen sections and HE-stained sections. However, due to the similarity in cell morphology, even experienced pathologists cannot accurately distinguish the morphological characteristics of igstroma, oligodendroglioma, and astrocytoma solely based on frozen sections and HE-stained sections.

[0049] To address the aforementioned technical deficiencies, referring to... Figure 1 One embodiment of the present invention provides a deep learning-based method for classifying intracranial tumor cells, comprising:

[0050] S101. Obtain full-slice pathological images of the slide.

[0051] S102. Small patches of the full-slice pathological image are obtained through a three-step image preprocessing method.

[0052] S103. Use a pre-set deep learning-based neural network model to extract features of small patches and predict the patch classification result based on the features of the small patches.

[0053] S104. Based on the patch classification results, predict the type of intracranial tumor cells.

[0054] This method acquires whole-slice scan pathological images, obtains small patches from the whole-slice scan pathological images through a three-step image preprocessing method, extracts features of the small patches using a pre-defined deep learning-based neural network model, predicts the patch classification results based on the features of the small patches, and predicts the type of intracranial tumor cells based on the patch classification results. By predicting the type of intracranial tumor cells based on whole-slice scan pathological images, this method helps physicians predict intracranial tumor cells and formulate prognostic plans, improving the accuracy and efficiency of physicians in classifying intracranial tumor cells.

[0055] In some embodiments, the sections include frozen sections and HE-stained sections.

[0056] Reference Figure 2 In some embodiments, step S102 may include, but is not limited to, steps S201 to S203:

[0057] Step S201: Convert the full-slice pathological image into a grayscale image, and segment the tissue region on the grayscale image using 220 as the threshold.

[0058] Step S202: Divide the tissue region into 224×224 non-overlapping patches with a resolution of 0.5 micrometers per pixel.

[0059] Step S203: Input the 224×224 image patch into the preset deep learning-based tissue slice selection model to obtain small image patches of the full-slice scan pathological image output by the deep learning-based tissue slice selection model.

[0060] In some embodiments, after obtaining small patches of the whole-slice pathological image through a three-step image preprocessing method, the deep learning-based classification method for intracranial tumor cells may include, but is not limited to, step S301:

[0061] Step S301: Randomly perform one or more of the following operations on the small image tiles: horizontal flip, vertical flip, brightness modification, and horizontal flip based on probability P1.

[0062] Reference Figure 3 In some embodiments, step S103 may include, but is not limited to, steps S401 to S405:

[0063] Step S401: Input the small image patch into the improved ResNet18 network model to generate feature map A; wherein, the improved ResNet18 network model is the network part after removing the average pooling layer based on the ResNet18 network model.

[0064] Step S402: Input feature map A into the ARPY network model and output the location information of key features. The ARPY network model consists of two convolutional layers with 3*3 kernels connected together and then connected to two fully connected layers.

[0065] Step S403: After cropping the tissue regions corresponding to the key features, resize them to 224*224, randomly occlude them, and input them into the improved ResNet18 network model to obtain feature map B.

[0066] Step S404: After concatenating feature map A and feature map B, the layers are sequentially passed through fully connected layer FC1, Dropout layer with probability P2, fully connected layer FC2, BN layer, fully connected layer FC3, ReLU activation function layer, fully connected layer FC4, and softmax function to predict the category probability value of the feature region of the small image patch.

[0067] Step S405: Obtain the tile classification result based on the category probability value.

[0068] In some embodiments, the tile classification results include intracranial germ cell tumor tiles, oligodendroglioma tiles, and astrocytoma tiles.

[0069] Reference Figure 4 In some embodiments, step S104 may include, but is not limited to, step S501:

[0070] Step S501: Predict the type of intracranial tumor cells from the tile classification results using the majority voting method.

[0071] To facilitate understanding by those skilled in the art, the following set of experimental data is provided:

[0072] During surgery, pathologists can only distinguish between benign and malignant areas at the frozen section level, and cannot immediately differentiate tumor types, which may lead to inappropriate treatment plans. Our research shows that the diagnostic accuracy of pathologists based on frozen sections for IG, oligodendroglioma, and astrocytoma is only 54.6%-69.7%. In identifying IG cases, the accuracy rates of the three pathologists were 9.1%, 54.6%, and 90.9%, respectively, which may lead to misdiagnosis and unnecessary surgical resection. Our frozen section model can independently distinguish IG from the other two tumors with 100.0% accuracy, which also brings the accuracy rate of all pathologists with AI assistance to 100.0%. Therefore, the deep learning-based neural network model of frozen sections in this invention can effectively assist neurosurgeons in selecting appropriate treatment plans during surgery for all three tumors.

[0073] Early postoperative treatment (chemotherapy and / or radiotherapy) is determined by pathological diagnosis of HE-stained sections, with preliminary results available approximately 24 hours post-surgery. However, we found that pathologists' accuracy in diagnosing IG, oligodendroglioma, and astrocytoma based on HE-stained whole-slide pathological images ranged from 53.5% to 83.7%. The deep learning-based neural network model for HE-stained sections of this invention can assist in the diagnosis of these three tumors with internal and external accuracies of 95.4% and 83.3%, respectively, thus aiding pathologists in diagnosis.

[0074] Additionally, refer to Figure 5 The present invention also provides a deep learning-based classification system for intracranial tumor cells, comprising a data acquisition module 1100, a data preprocessing module 1200, a tile classification module 1300, and an auditing and type prediction module 1400, wherein:

[0075] The data acquisition module 1100 is used to acquire full-slice scan pathological images of the slides.

[0076] The data preprocessing module 1200 is used to obtain small patches of full-slice pathological images through a three-step image preprocessing method.

[0077] The tile classification module 1300 is used to extract features of small tiles using a preset deep learning-based neural network model, and predict the tile classification result based on the features of the small tiles.

[0078] The type prediction module 1400 is used to predict the type of intracranial tumor cells based on the tile classification results.

[0079] This system acquires full-slice pathological images of tissue sections, obtains small patches from the full-slice pathological images through a three-step image preprocessing method, extracts features from the small patches using a pre-defined deep learning-based neural network model, predicts the patch classification result based on the features, and predicts the type of intracranial tumor cells based on the patch classification result. By predicting the type of intracranial tumor cells based on full-slice pathological images, this system helps physicians predict intracranial tumor cells and formulate prognostic plans, improving the accuracy and efficiency of physicians in classifying intracranial tumor cells.

[0080] It should be noted that the system embodiments described above are based on the same inventive concept as the method embodiments described above. Therefore, the relevant content of the method embodiments described above is also applicable to the system embodiments described above, and will not be repeated here.

[0081] The processor and memory can be connected via a bus or other means.

[0082] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network model. Examples of such network models include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0083] The non-transitory software program and instructions required to implement the deep learning-based intracranial tumor cell classification method of the above embodiments are stored in memory. When executed by a processor, the deep learning-based intracranial tumor cell classification method of the above embodiments is executed, for example, the method described above is executed. Figure 1 The method steps S101 to S104.

[0084] This application also provides a computer-readable storage medium storing computer-executable instructions for performing: the deep learning-based classification method for intracranial tumor cells as described above.

[0085] The computer-readable storage medium stores computer-executable instructions that are executed by a processor or controller, for example, by a processor in the above-described electronic device embodiment, causing the processor to perform the deep learning-based intracranial tumor cell classification method described above, for example, performing the above-described... Figure 1 The method steps S101 to S104.

[0086] It will be understood by those skilled in the art that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program units, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically contain computer-readable instructions, data structures, program units, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0087] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.

Claims

1. A method of classifying intracranial tumor cells based on deep learning, characterized by, The method for classifying intracranial tumor cells based on deep learning comprises: acquiring a whole-slide scanning pathology image of a slice; obtaining small patches of the whole-slide scanning pathology image through a three-step image preprocessing method, specifically: converting the whole-slide scanning pathology image into a grayscale image, and segmenting the tissue region on the grayscale image with 220 as the threshold value; dividing the tissue region into 224*224 patches with a resolution of 0.5 microns per pixel and without overlap; inputting the 224*224 patches into a pre-set deep learning-based tissue slice selection model to obtain small patches of the whole-slide scanning pathology image output by the deep learning-based tissue slice selection model; extracting features of the small patches using a pre-set deep learning-based neural network model, and predicting patch classification results according to the features of the small patches, specifically: inputting the small patches into an improved resnet18 network model to generate a feature map A; wherein the improved resnet18 network model is obtained by deleting the network part after the average pooling layer of the resnet18 network model; inputting the feature map A into an ARPY network model to output the position information of key features, wherein the ARPY network model is composed of two convolution layers with a convolution kernel of 3*3 connected to two fully connected layers; cropping the tissue region corresponding to the key features and resetting the size to 224*224, and then inputting the randomly occluded tissue region into the improved resnet18 network model to obtain a feature map B; performing a splicing operation on the feature map A and the feature map B, and then sequentially passing through a fully connected layer FC1, a Dropout layer with a probability of P2, a fully connected layer FC2, a BN layer, a fully connected layer FC3, a relu activation function layer, a fully connected layer FC4, and a softmax function to predict the class probability value of the feature region of the small patch; obtaining a patch classification result according to the class probability value; predicting the type of the intracranial tumor cells according to the patch classification result. 2.The method of claim 1, wherein, The slice includes a frozen slice and an HE staining slice. 3.The method of claim 2, wherein the method comprises: After obtaining the small patches of the whole-slide scanning pathology image through the three-step image preprocessing method, the method further comprises the step of: randomly performing one or a combination of horizontal flipping, vertical flipping, brightness modification, and horizontal flipping according to a probability P1 on the small patches. 4.The method of claim 3, wherein the method comprises: The patch classification result includes an intracranial germinoma patch, an oligodendroglioma patch, and an astrocytoma patch. 5.The method of classifying intracranial tumor cells based on deep learning according to claim 4, wherein, The method for predicting the type of the intracranial tumor cells according to the patch classification result comprises: predicting the type of the intracranial tumor cells from the patch classification result by majority voting. 6.A deep learning-based classification system of intracranial tumor cells, characterized by, The system for classifying intracranial tumor cells based on deep learning comprises: a data acquisition module for acquiring a whole-slide scanning pathology image of a slice; a data preprocessing module for obtaining small patches of the whole-slide scanning pathology image through a three-step image preprocessing method, specifically: converting the whole-slide scanning pathology image into a grayscale image, and segmenting the tissue region on the grayscale image with 220 as the threshold value; divide the tissue region into 224*224 patches with a resolution of 0.5 microns per pixel and without overlapping; input the 224*224 patches into a preset deep learning-based tissue slice selection model to obtain small patches of a whole-slide pathology image output by the deep learning-based tissue slice selection model; a patch classification module configured to extract features of the small patches by using a preset deep learning-based neural network model and predict patch classification results according to the features of the small patches, specifically: input the small patches into an improved resnet18 network model to generate a feature map A, wherein the improved resnet18 network model is obtained by deleting a network part after an average pooling layer based on a resnet18 network model; input the feature map A into an ARPY network model to output position information of key features, the ARPY network model is composed of two convolution layers with a convolution kernel of 3*3 connected to each other and then connected to two fully connected layers; cut the tissue region corresponding to the key features and reset the size to 224*224, then input the cut tissue region into the improved resnet18 network model after random occlusion to obtain a feature map B; perform a splicing operation on the feature map A and the feature map B, and then sequentially pass through a fully connected layer FC1, a Dropout layer with a probability of P2, a fully connected layer FC2, a BN layer, a fully connected layer FC3, a relu activation function layer, a fully connected layer FC4 and a softmax function to predict a class probability value of a feature region of a small patch; obtain patch classification results according to the class probability value; a type prediction module configured to predict a type of the intracranial tumor cells according to the patch classification results. 7.A device for classifying intracranial tumor cells based on deep learning, characterized by, The memory is connected in communication with the at least one control processor and stores instructions executable by the at least one control processor, and the instructions are executed by the at least one control processor to enable the at least one control processor to perform the deep learning-based classification method of the intracranial tumor cells according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: The computer readable storage medium stores computer executable instructions for causing a computer to perform the deep learning-based classification method of the intracranial tumor cells according to any one of claims 1 to 5.

Images