Glioma grading method, device, equipment and medium based on tissue pathology image

By extracting and predicting histopathological images, a glioma grading model is generated, which solves the problem of strong subjectivity of glioma diagnosis results in the prior art, and improves the accuracy and objectivity of the diagnosis.

CN115082718BActive Publication Date: 2024-11-22TSINGHUA UNIVERSITY
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202210488276.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-05-06
Publication Date
2024-11-22
Estimated Expiration
2042-05-06

AI Technical Summary

Technical Problem

The diagnosis results of glioma in the prior art are highly subjective, resulting in low diagnostic accuracy of glioma.

Method used

By obtaining histopathological full-scan WSI images, input them into the pre-trained glioma grading model, data preprocessing and feature extraction are performed, feature representations of ROI subpackages are generated, and glue is predicted using multi-example feature aggregation operator and MLP modules The malignant level of tumors is assisted in diagnosis.

Benefits of technology

It improves the objectivity and accuracy of glioblastoma diagnosis results, especially for the diagnosis of glioblastoma with the highest degree of malignancy.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115082718B_ABST
    Figure CN115082718B_ABST
Patent Text Reader

Abstract

The present application provides a method, device, equipment and medium for glioma grading based on tissue pathology images. The method obtains the glioma malignancy level corresponding to the WSI image to be detected by inputting the acquired WSI image to be detected into a pre-trained glioma grading model, and then saves the glioma malignancy level to achieve intelligent diagnosis, assist doctors in diagnosis, enhance the objectivity of glioma diagnosis results, and improve diagnostic accuracy.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of medical technology, and in particular to a method, device, equipment and medium for grading glioma based on tissue pathology images. Background Art

[0002] Gliomas account for about 30% of all brain and central nervous system tumors and 80% of all malignant brain tumors. The annual incidence of gliomas in my country is 50,000 to 100,000, and the 5-year mortality rate ranks third among systemic tumors, second only to pancreatic cancer and lung cancer. Clinically, the grading of gliomas is based on pathological diagnosis. Gliomas of different grades show different characteristics in histopathological sections. For example, the presence of mitosis represents that the glioma is at least grade II, while microvascular hyperplasia and increased cell density correspond to the characteristics of grade III.

[0003] In the prior art, for the diagnosis of glioma, pathologists usually make a preliminary judgment on the degree of malignancy by observing tissue pathology sections. For some difficult-to-judge section images, it is necessary to combine molecular characteristics such as cell proliferation score Ki-67 to determine the level of malignancy of the glioma.

[0004] However, in the above-mentioned diagnostic methods, the diagnostic results of pathologists are highly subjective, which makes the accuracy of the diagnostic results of gliomas low. Summary of the invention

[0005] The present application provides a glioma grading method, device, equipment and medium based on tissue pathology images, which is used to solve the problem in the prior art that the diagnosis results of pathologists are highly subjective, resulting in low accuracy of the diagnosis results of gliomas.

[0006] In a first aspect, the present application provides a glioma grading method based on histopathological images, comprising:

[0007] Acquire a full scan WSI image of the tissue pathology to be detected;

[0008] Inputting the WSI image to be detected into a pre-trained glioma grading model to obtain the glioma malignancy grade of the WSI image to be detected;

[0009] The glioma malignancy grade of the WSI image to be detected is saved.

[0010] Optionally, the method further includes:

[0011] Acquire training data, wherein the training data includes a WSI image and a level label, wherein the level label is used to indicate the malignancy level of the glioma;

[0012] The glioma grading model is obtained by training the preset model with the training data.

[0013] Optionally, the step of inputting the WSI image to be detected into a pre-trained glioma grading model to obtain the glioma malignancy grade of the WSI image to be detected includes:

[0014] Performing data preprocessing on the WSI image to be detected to obtain image blocks at different magnifications, wherein the image blocks are sub-images of the WSI image to be detected;

[0015] The image blocks at each magnification are input into the feature extraction network for feature extraction to obtain the feature vector corresponding to each image block;

[0016] Clustering the feature vectors corresponding to each image block respectively to obtain classifications of the feature vectors at each magnification, and obtaining a first preset number of region of interest ROI sub-packages according to the classifications of the feature vectors at each magnification, wherein the ROI sub-packages include a second preset number of feature vectors, and the ROI sub-packages include feature vectors at different magnifications;

[0017] Aggregating the feature vectors in the ROI sub-package by a multi-instance feature aggregation operator to generate a feature representation of the ROI sub-package;

[0018] Classifying the feature representation of the ROI sub-package through the MLP module to predict the malignancy grade of the glioma of the ROI sub-package;

[0019] The glioma malignancy level of the WSI image to be detected is determined according to the glioma malignancy level of each ROI sub-package.

[0020] Optionally, the performing data preprocessing on the WSI image to be detected to obtain image blocks at different magnifications includes:

[0021] Dividing the WSI image to be detected into a plurality of image blocks of a first preset size in the form of a sliding window;

[0022] Calculating the tissue ratio of each image block, and determining a target image block according to the tissue ratio, wherein the target image block is an image block whose tissue ratio is greater than a preset threshold;

[0023] Cropping the target image block to obtain an image block of a second preset size and an image block of a third preset size;

[0024] The image block of the first preset size, the image block of the second preset size, and the image block of the third preset size are scaled to a fixed size to obtain image blocks at different magnifications.

[0025] Optionally, aggregating the feature vectors in the ROI sub-package by using a multi-instance feature aggregation operator to generate a feature representation of the ROI sub-package includes:

[0026] Inputting n feature vectors and an initialization vector of the ROI sub-package into the multi-instance feature aggregation operator, performing information fusion between the n feature vectors and the initialization vector through an information fusion formula to obtain a fusion result corresponding to the initialization vector, and generating a new feature vector according to the fusion result, and using the new feature vector as the feature vector of the ROI sub-package;

[0027] The information fusion formula is:

[0028]

[0029] Where Q is the vector of the initialization vector after the first linear transformation, K and V are the vectors of the target feature vector in the ROI sub-package after the second linear transformation and the third linear transformation, U is a hyperparameter threshold, S is the similarity between the initialization vector and the target feature vector, QK T is the attention matrix, Is a real number used to normalize the attention matrix.

[0030] Optionally, determining the glioma malignancy level corresponding to the WSI image to be detected according to the glioma malignancy level of the ROI sub-package includes:

[0031] Through a voting mechanism, the glioma malignancy level with the largest number is determined as the glioma malignancy level corresponding to the WSI image to be detected.

[0032] In a second aspect, the present application provides a glioma grading device based on histopathological images, comprising:

[0033] A first acquisition module is used to acquire a full scan WSI image of the tissue pathology to be detected;

[0034] A detection module, used for inputting the WSI image to be detected into a pre-trained glioma grading model to obtain the glioma malignancy grade of the WSI image to be detected;

[0035] The saving module is used to save the glioma malignancy grade of the WSI image to be detected.

[0036] Optionally, the device further comprises:

[0037] A second acquisition module is used to acquire training data, wherein the training data includes a WSI image and a level label, and the level label is used to indicate the malignancy level of the glioma;

[0038] A training module is used to train a preset model using the training data to obtain the glioma grading model.

[0039] In a third aspect, the present application provides an electronic device, comprising: a processor, and a memory communicatively connected to the processor;

[0040] The memory stores computer-executable instructions;

[0041] The processor executes the computer-executable instructions stored in the memory to implement a glioma grading method based on tissue pathology images as described in the first aspect.

[0042] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement a glioma grading method based on tissue pathology images as described in the first aspect.

[0043] In a fifth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, is a method for grading gliomas based on tissue pathology images as described in the first aspect.

[0044] The present application provides a method, device, equipment and medium for glioma grading based on tissue pathology images. The method obtains the glioma malignancy level corresponding to the WSI image to be detected by inputting the acquired WSI image to be detected into a pre-trained glioma grading model, and then saves the glioma malignancy level to achieve intelligent diagnosis, assist doctors in diagnosis, enhance the objectivity of glioma diagnosis results, and improve diagnostic accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0046] Figure 1 A schematic diagram of a flow chart of a glioma grading method based on histopathological images provided in Example 1 of the present application;

[0047] Figure 2 A schematic diagram of a flow chart of a glioma grading method based on histopathological images provided in Example 2 of the present application;

[0048] Figure 3 A schematic diagram of a sliding window provided in Embodiment 2 of the present application;

[0049] Figure 4 A schematic diagram of image clipping provided in Embodiment 2 of the present application;

[0050] Figure 5 A schematic diagram of an image block in an ROI sub-package provided in Embodiment 2 of the present application;

[0051] Figure 6 A schematic diagram of a multi-example feature aggregation operator provided in Embodiment 2 of the present application;

[0052] Figure 7 A schematic diagram of the processing process of a glioma grading method based on histopathological images provided in Example 2 of the present application;

[0053] Figure 8 A schematic diagram of the structure of a glioma grading device based on histopathological images provided in Example 3 of the present application;

[0054] Fig. 9 A schematic diagram of the structure of a glioma grading device based on tissue pathology images provided in Embodiment 4 of the present invention.

[0055] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0056] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0057] Clinically, the grading of gliomas is based on pathological diagnosis, which is the final diagnosis and is also known as the "gold standard" in clinical diagnosis. WSI images contain rich phenotypic information. WSI uses a digital scanner to scan traditional pathological sections, collect high-resolution digital images, and then use a computer to seamlessly stitch and integrate the obtained fragmented images to produce a visual digital image. WSI images can be used to monitor the potential mechanisms that lead to disease progression and patient survival outcomes.

[0058] The grading of gliomas is of great significance for the clinical treatment and prognosis of patients. The World Health Organization divides gliomas into grades I-IV according to their malignancy. In the prior art, for the diagnosis of gliomas, pathologists usually make a preliminary judgment on the malignancy by observing the tissue pathology sections. For some difficult-to-judge slice images, it is necessary to combine molecular features such as cell proliferation score Ki-67 to determine the malignancy level of the glioma.

[0059] However, in the above-mentioned diagnostic methods, the diagnostic results of pathologists are highly subjective, which makes the accuracy of the diagnostic results of gliomas low.

[0060] Therefore, the present application provides a glioma grading method, device, equipment and medium based on tissue pathology images. By inputting the WSI image to be detected into a pre-trained glioma grading model to obtain the diagnostic result of the WSI image, that is, the malignancy level of the glioma, intelligent diagnosis is achieved, which assists doctors in diagnosis, enhances the objectivity of the glioma diagnosis results, and improves the diagnostic accuracy, especially for the diagnosis of glioblastoma (grade IV). Since the incidence of glioblastoma is the highest among all gliomas, and it has the highest malignancy and the worst prognosis, accurate judgment of its level is of great significance for clinical treatment and prognosis.

[0061] The application scenario of the present application can be the diagnosis of oligodendroglioma, or the diagnosis of glioblastoma (astrocytoma) or ependymoma and other cell tumors. It is understandable that the glioma grading method based on histopathological images provided by the present application includes but is not limited to the above application scenarios.

[0062] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments may exist independently or in combination with each other, and the same or similar concepts or processes may not be described in detail in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0063] refer to Figure 1 , Figure 1 A flow chart of a glioma grading method based on tissue pathology images is provided in Example 1 of the present application. The method can be executed by a glioma grading device based on tissue pathology images, and the device can be a server. The server is used as an example for explanation below. The method includes the following steps.

[0064] S101: Acquire a WSI image to be detected.

[0065] The server obtains a WSI image to be detected. Specifically, the WSI image is a digital scanner that scans a traditional pathological section to collect a digital image with high resolution, and then the server seamlessly stitches and integrates the obtained fragmented images to produce the obtained WSI image.

[0066] S102 , inputting the WSI image to be detected into a pre-trained glioma grading model to obtain the glioma malignancy grade of the WSI image to be detected.

[0067] The pre-trained glioma grading model is obtained by training a preset model with acquired training data by the server. The training data includes WSI images and level labels. Exemplarily, the level labels may include level I, level II, level III and level IV. The level labels are used to distinguish the malignancy level of glioma.

[0068] After the server obtains the WSI image to be detected, it inputs the image into the pre-trained glioma grading model to obtain the glioma malignancy level of the image to achieve intelligent diagnosis. The specific implementation is described in Example 2, please refer to Example 2.

[0069] S103 . Save the glioma malignancy level of the WSI image to be detected.

[0070] After the server determines the malignancy level of the glioma of the WSI image to be detected, it saves the malignancy level of the glioma so that the pathologist can obtain the diagnosis result and carry out targeted treatment for the patient.

[0071] In this embodiment, the acquired WSI image to be detected is input into the pre-trained glioma grading model to obtain the glioma malignancy level corresponding to the WSI image to be detected, and then the glioma malignancy level is saved to achieve intelligent diagnosis, assist doctors in diagnosis, enhance the objectivity of glioma diagnosis results, and improve diagnostic accuracy.

[0072] The following describes the prediction of the malignancy grade of glioma by inputting the WSI image to be detected into the pre-trained glioma grading model through Example 2.

[0073] refer to Figure 2 , Figure 2 A flow chart of a glioma grading method based on tissue pathology images provided in Example 2 of the present application is provided. The method can be executed by a glioma grading device based on tissue pathology images, and the device can be a server. The server is used as an example for explanation below. The method includes the following steps.

[0074] S201 , performing data preprocessing on the WSI image to be detected to obtain image blocks at different magnifications, where the image blocks are sub-images of the WSI image to be detected.

[0075] First, the server divides the WSI image to be detected into a plurality of image blocks of a first preset size in the form of a sliding window. The sliding window size is the first preset size. For example, the first preset size is a pixel size of 2048×2048. Figure 3 .

[0076] Then calculate the tissue ratio of each image block, that is, the ratio of the tissue area. Specifically, the tissue ratio is obtained by calculating the ratio of the number of pixels in the foreground area to the total number of pixels. Taking the size of the sliding window example above as an example, the total number of pixels is 2048. 2 The target image block is determined according to the tissue ratio, and the target image block is an image block where the tissue area ratio is greater than a preset threshold, and the preset threshold can be set to 0.5. By discarding image blocks where the background area ratio is too high, the accuracy of the detection result can be improved.

[0077] Then the target image block is cropped to obtain an image block of a second preset size and an image block of a third preset size. For example, for a target image block of a size of 2048×2048, the target image block is cropped to obtain four image blocks of a size of 1024×1024, and then the 1024×1024 image block is cropped again to obtain four image blocks of a size of 512×512. Figure 4 .

[0078] Then, the image blocks of the first preset size, the image blocks of the second preset size, and the image blocks of the third preset size are scaled to a fixed size, and image blocks at different magnifications can be obtained. For example, the fixed size can be 256×256, and the magnifications can be 5 times, 10 times, and 20 times. Of course, a magnification of 40 times can also be included, and the present application is not limited to this. Different magnifications can be determined according to the size of the WSI image. When the size of the WSI image reaches 40000×40000 and above, the magnification can be 5 times, 10 times, 20 times, 40 times, etc.

[0079] Through the above data preprocessing, the WSI image to be detected is divided into image blocks of different sizes, so that the subsequent step S203 can be clustered according to the image blocks of different scales, thereby effectively combining multi-scale features, so that the model has better accuracy in the classification task.

[0080] S202: Input the image blocks at each magnification factor into a feature extraction network for feature extraction to obtain a feature vector corresponding to each image block.

[0081] The extractable network may be a ResNet50 network or other neural networks for feature extraction.

[0082] S203 , clustering the feature vectors corresponding to the image blocks respectively to obtain classifications of the feature vectors at various magnifications, and obtaining a first preset number of region-of-interest ROI sub-packages according to the classifications of the feature vectors at various magnifications.

[0083] Specifically, the following examples are used to illustrate:

[0084] The server clusters the feature vectors corresponding to the image blocks with a magnification of 5 times (2048×2048) through a clustering module, and the number of categories can be set to 4. The clustering module can be a K-means clustering algorithm Kmeans. Similarly, the feature vectors of the image blocks with a magnification of 10 times (1024×1024) and the feature vectors of the image blocks with a magnification of 20 times (512×512) are also divided into 4 categories.

[0085] Then, m feature vectors are selected from the feature vectors of the four categories corresponding to the image blocks of each magnification. For the feature vectors corresponding to the image blocks of different magnifications, m is set to 1, 4, and 4 in sequence. That is, 1 feature vector is selected from each of the four categories of feature vectors corresponding to the image blocks of 5 times magnification, 4 feature vectors are selected from each of the four categories of feature vectors corresponding to the image blocks of 10 times magnification, and 4 feature vectors are selected from each of the four categories of feature vectors corresponding to the image blocks of 20 times magnification to form an ROI sub-package, which includes 4 feature vectors corresponding to the image blocks of 5 times magnification, 16 feature vectors corresponding to the image blocks of 10 times magnification, and 16 feature vectors corresponding to the image blocks of 20 times magnification, that is, 36 feature vectors form an ROI sub-package, and 36 is the first preset number. Exemplarily, a schematic diagram of the image blocks in an ROI sub-package is as follows: Figure 5 shown.

[0086] The server generates a second preset number of ROI sub-packages, which may be 20. This indicates that 20 ROI sub-packages are generated. It should be noted that the feature vectors included in each generated ROI sub-package are different, that is, there are no repeated feature vectors in each ROI sub-package, which effectively combines multi-scale features, so that the model achieves better accuracy in hierarchical prediction.

[0087] S204 , aggregating feature vectors in the ROI sub-package by using a multi-instance feature aggregation operator to generate a feature representation of the ROI sub-package.

[0088] The multi-instance feature aggregation operator can be called an encoder. Since the feature vector Embedding in the ROI sub-package is a highly concentrated high-level semantic feature, the information it contains cannot be changed by subsequent back propagation. The essential function of the multi-instance feature aggregation operator is to effectively combine the feature vectors in the ROI sub-package. When too many layers of encoders are added, it will result in non-convergence. This is because the amount of data in the ROI sub-package has not reached the millions. Therefore, the embodiment of the present application has built a layer in the selection of encoders. Experiments have shown that as the number of encoder layers increases, the effect of the hierarchical model will be greatly reduced. For example, the structural diagram of the encoder can be referred to. Figure 6 .

[0089] Specifically, the process of aggregating the feature vectors in the ROI sub-package by the multi-instance feature aggregation operator is as follows:

[0090] The server inputs the n feature vectors of the ROI sub-package and an initialization vector into the multi-instance feature aggregation operator, and first normalizes the n+1 feature vectors to obtain the regularized feature vectors, and then performs the first linear transformation, the second linear transformation, and the third linear transformation on the n+1 feature vectors to obtain the Query, Key, and Value vectors, referred to as Q, K, and V, respectively.

[0091] The above linear transformation refers to multiplying the eigenvector with three different linear transformation matrices to generate linearly transformed vectors Q, K and V.

[0092] Then, the information fusion formula (attention mechanism Multi-head Limited Attention) is used to fuse the n feature vectors and the initialization vector to obtain the fusion result corresponding to the initialization vector. Specifically, the information fusion formula is:

[0093]

[0094] Among them, Q is the vector of the initialization vector after the first linear transformation, K and V are the vectors of the target feature vector in the ROI sub-package after the second and third linear transformations, U is a hyperparameter threshold that can be taken as 0.8, S is the similarity between the initialization vector and the target feature vector, QK T is the attention matrix, is a real number, where d kIt can be 64, which is used to normalize the attention matrix. The target feature vector is any feature vector in the ROI sub-package.

[0095] After the above calculation, the server obtains the fusion result corresponding to the initialization vector, which is a feature vector. The fusion result is then subjected to residual calculation with the initialization vector, and the result of the residual calculation is regularized. The regularized result is passed through a multilayer perceptron (MLP) to obtain an optimized feature vector. The optimized feature vector is subjected to residual calculation with the residual calculation result to generate a new feature vector. The new feature vector is used as the feature representation of the ROI sub-package, so that the model obtains the target area that needs to be focused on and obtains the focus of attention. Then, more attention is paid to this area to obtain more detailed information of the target that needs to be focused on, thereby suppressing other useless information and improving the accuracy of the model.

[0096] It should be noted that among the above n+1 feature vectors, every two feature vectors need to be calculated using formula (1), and the number of new feature vectors obtained is n+1. Since only the new feature vector corresponding to the initialization vector is required as the feature representation of the ROI sub-package, the above description does not explain the calculation between every two feature vectors among the n feature vectors in the ROI sub-package.

[0097] S205 , classifying the feature representation of the ROI sub-package through the MLP module to predict the malignancy grade of the glioma of the ROI sub-package.

[0098] After obtaining the feature representation of the ROI sub-package, the server classifies the feature representation of the ROI sub-package through the MLP module to predict the malignancy grade of the glioma in the ROI sub-package.

[0099] S206 . Determine the malignancy level of the glioma of the WSI image to be detected according to the malignancy level of the glioma of each ROI sub-package.

[0100] Through the above operations, the server can determine the glioma malignancy level of each ROI sub-package, and then can determine the glioma malignancy level of the WSI image to be detected through a voting mechanism according to the glioma malignancy level of each ROI sub-package. Specifically, the server determines the glioma malignancy level with the largest number as the glioma malignancy level corresponding to the WSI image to be detected. Exemplarily, among 20 ROI sub-packages, 15 ROI sub-packages have a glioma malignancy level of level III, then the glioma malignancy level of the WSI image to be detected is determined to be level III.

[0101] The glioma malignancy level of the WSI image to be detected is determined by the above voting mechanism, so that the accuracy of the model prediction is higher.

[0102] For example, the above processing can refer to Figure 7 .

[0103] In this embodiment, the server inputs the image blocks of the WSI image to be detected at different magnifications into the feature extraction network for feature extraction, and obtains the feature vectors corresponding to each image block. Then the server clusters the feature vectors at each magnification respectively, divides the feature vectors at each magnification into four categories, and obtains the first preset number of ROI sub-packages of interest based on the four categories of feature vectors at each magnification, effectively combines multi-scale features, so that the model achieves good accuracy in graded prediction. Then the server aggregates the feature vectors in the ROI sub-package through a multi-instance feature aggregation operator, and then classifies the feature representations of each ROI sub-package obtained after aggregation through an MLP module, predicts the malignant grade of glioma of each ROI sub-package, and then determines the malignant grade of glioma of the WSI image to be detected through a voting mechanism. In order to realize intelligent diagnosis, assist doctors in diagnosis, enhance the objectivity of glioma diagnosis results, and improve diagnostic accuracy.

[0104] It should be noted that the server trains the preset model through training data to obtain a glioma grading model, and also needs to go through the operations of steps S501-S504 in the above-mentioned embodiment 2. The difference is that in step S503, after obtaining the ROI sub-package, it is necessary to assign the same level label of the WSI image to the ROI sub-package, and then train the multi-example feature aggregation operator and MLP module in the preset model through each ROI sub-package and its corresponding level label to obtain the glioma grading model.

[0105] refer to Figure 8 , Figure 8 The schematic diagram of the structure of a glioma grading device based on histopathological images provided in the third embodiment of the present application is as follows. The device 80 comprises: a first acquisition module 801 , a detection module 802 and a storage module 803 .

[0106] The first acquisition module 801 is used to acquire a full scan WSI image of the tissue pathology to be detected.

[0107] The detection module 802 is used to input the WSI image to be detected into the pre-trained glioma grading model to obtain the glioma malignancy grade of the WSI image to be detected.

[0108] The saving module 803 is used to save the glioma malignancy level of the WSI image to be detected.

[0109] Optionally, the device 80 further includes:

[0110] The second acquisition module is used to acquire training data, the training data includes WSI images and level labels, and the level labels are used to indicate the malignancy level of glioma.

[0111] The training module is used to train the preset model through training data to obtain a glioma grading model.

[0112] Optionally, the detection module 802 is specifically used for:

[0113] Data preprocessing is performed on the WSI image to be detected to obtain image blocks at different magnifications, where the image blocks are sub-images of the WSI image to be detected.

[0114] The image blocks at each magnification are input into the feature extraction network for feature extraction to obtain the feature vectors corresponding to each image block.

[0115] The feature vectors corresponding to each image block are clustered respectively to obtain the classification of the feature vectors at each magnification, and based on the classification of the feature vectors at each magnification, a first preset number of region of interest ROI sub-packages are obtained, the ROI sub-package includes a second preset number of feature vectors, and the ROI sub-package includes feature vectors at different magnifications.

[0116] The feature vectors in the ROI sub-package are aggregated through the multi-instance feature aggregation operator to generate the feature representation of the ROI sub-package.

[0117] The feature representation of the ROI sub-package is classified through the MLP module to predict the malignancy grade of glioma in the ROI sub-package.

[0118] According to the glioma malignancy level of each ROI sub-package, the glioma malignancy level of the WSI image to be detected is determined.

[0119] Optionally, the detection module 802 is specifically used for:

[0120] The WSI image to be detected is divided into a plurality of image blocks of a first preset size in the form of a sliding window.

[0121] The tissue ratio of each image block is calculated, and a target image block is determined according to the tissue ratio, where the target image block is an image block whose tissue ratio is greater than a preset threshold.

[0122] The target image block is cropped to obtain an image block of a second preset size and an image block of a third preset size.

[0123] The image blocks of the first preset size, the image blocks of the second preset size, and the image blocks of the third preset size are scaled to a fixed size to obtain image blocks at different magnifications.

[0124] Optionally, the detection module 802 is specifically used for:

[0125] The n feature vectors and an initialization vector of the ROI sub-package are input into the multi-instance feature aggregation operator, and the information fusion formula is used to fuse the n feature vectors and the initialization vector to obtain the fusion result corresponding to the initialization vector. A new feature vector is generated according to the fusion result, and the new feature vector is used as the feature representation of the ROI sub-package.

[0126] The information fusion formula is:

[0127]

[0128] Among them, Q is the vector of the initialization vector after the first linear transformation, K and V are the vectors of the target feature vector in the ROI sub-package after the second and third linear transformations, U is a hyperparameter threshold, S is the similarity between the initialization vector and the target feature vector, QK T is the attention matrix, Is a real number used to normalize the attention matrix.

[0129] Optionally, the detection module 802 is specifically used for:

[0130] Through the voting mechanism, the glioma malignancy level with the largest number is determined as the glioma malignancy level corresponding to the WSI image to be detected.

[0131] The device of this embodiment can be used to execute the steps of a glioma grading method based on tissue pathology images in Example 1 or Example 2. The specific implementation method and technical effect are similar and will not be repeated here.

[0132] Fig. 9 A schematic diagram of a glioma grading device based on histopathological images provided in Embodiment 4 of the present invention is shown in FIG. Fig. 9 As shown, the device 90 includes: a processor 901, a memory 902, and a transceiver 903. The memory 902 is used to store instructions, and the transceiver 903 is used to communicate with other devices. The processor 901 is used to execute the instructions stored in the memory so that the device 90 performs the steps of a glioma grading method based on tissue pathology images as in Example 1 or Example 2. The specific implementation method and technical effect are similar and will not be repeated here.

[0133] Embodiment 5 of the present invention provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, it is used to implement the steps of a glioma grading method based on tissue pathology images as in the above-mentioned embodiment 1 or embodiment 2. The specific implementation method and technical effect are similar and will not be repeated here.

[0134] Embodiment 6 of the present invention provides a computer program product, including a computer program. When the computer program is executed by a processor, it implements the steps of a glioma grading method based on tissue pathology images as in the above-mentioned embodiment 1 or embodiment 2. The specific implementation method and technical effect are similar and will not be repeated here.

[0135] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0136] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A glioma grading method based on histopathological images, characterized in that: The method comprises: Acquire a full scan WSI image of the tissue pathology to be detected; Inputting the WSI image to be detected into a pre-trained glioma grading model to obtain the glioma malignancy grade of the WSI image to be detected; Saving the glioma malignancy grade of the WSI image to be detected; The step of inputting the WSI image to be detected into a pre-trained glioma grading model to obtain the glioma malignancy grade of the WSI image to be detected comprises: Performing data preprocessing on the WSI image to be detected to obtain image blocks at different magnifications, wherein the image blocks are sub-images of the WSI image to be detected; The image blocks at each magnification are input into the feature extraction network for feature extraction to obtain the feature vector corresponding to each image block; Clustering the feature vectors corresponding to each image block respectively to obtain classifications of the feature vectors at each magnification, and obtaining a first preset number of region of interest ROI sub-packages according to the classifications of the feature vectors at each magnification, wherein the ROI sub-packages include a second preset number of feature vectors, and the ROI sub-packages include feature vectors at different magnifications; Aggregating the feature vectors in the ROI sub-package by a multi-instance feature aggregation operator to generate a feature representation of the ROI sub-package; Classifying the feature representation of the ROI sub-package through the MLP module to predict the malignancy grade of the glioma of the ROI sub-package; The glioma malignancy level of the WSI image to be detected is determined according to the glioma malignancy level of each ROI sub-package.

2. The method according to claim 1, characterized in that The method further comprises: Acquire training data, wherein the training data includes a WSI image and a level label, wherein the level label is used to indicate the malignancy level of the glioma; The glioma grading model is obtained by training the preset model with the training data.

3. The method according to claim 1, characterized in that The performing data preprocessing on the WSI image to be detected to obtain image blocks at different magnifications includes: Dividing the WSI image to be detected into a plurality of image blocks of a first preset size in the form of a sliding window; Calculating the tissue ratio of each image block, and determining a target image block according to the tissue ratio, wherein the target image block is an image block whose tissue ratio is greater than a preset threshold; Cropping the target image block to obtain an image block of a second preset size and an image block of a third preset size; The image block of the first preset size, the image block of the second preset size, and the image block of the third preset size are scaled to a fixed size to obtain image blocks at different magnifications.

4. The method according to claim 3, characterized in that The step of aggregating the feature vectors in the ROI sub-package by using a multi-instance feature aggregation operator to generate a feature representation of the ROI sub-package includes: Inputting n feature vectors and an initialization vector of the ROI sub-package into the multi-instance feature aggregation operator, performing information fusion between the n feature vectors and the initialization vector through an information fusion formula to obtain a fusion result corresponding to the initialization vector, and generating a new feature vector according to the fusion result, and using the new feature vector as a feature representation of the ROI sub-package; The information fusion formula is: Where Q is the vector of the initialization vector after the first linear transformation, K and V are the vectors of the target feature vector in the ROI sub-package after the second linear transformation and the third linear transformation, U is a hyperparameter threshold, S is the similarity between the initialization vector and the target feature vector, QK T is the attention matrix, is a real number used to normalize the attention matrix.

5. The method according to claim 4, characterized in that Determining the glioma malignancy level corresponding to the WSI image to be detected according to the glioma malignancy level of the ROI sub-package includes: Through a voting mechanism, the glioma malignancy level with the largest number is determined as the glioma malignancy level corresponding to the WSI image to be detected.

6. A glioma grading device based on histopathological images, characterized in that: The device comprises: A first acquisition module is used to acquire a full scan WSI image of the tissue pathology to be detected; A detection module, used for inputting the WSI image to be detected into a pre-trained glioma grading model to obtain the glioma malignancy grade of the WSI image to be detected; A saving module, used for saving the glioma malignancy grade of the WSI image to be detected; The detection module is specifically used for: Performing data preprocessing on the WSI image to be detected to obtain image blocks at different magnifications, wherein the image blocks are sub-images of the WSI image to be detected; The image blocks at each magnification are input into the feature extraction network for feature extraction to obtain the feature vector corresponding to each image block; Clustering the feature vectors corresponding to each image block respectively to obtain classifications of the feature vectors at each magnification, and obtaining a first preset number of region of interest ROI sub-packages according to the classifications of the feature vectors at each magnification, wherein the ROI sub-packages include a second preset number of feature vectors, and the ROI sub-packages include feature vectors at different magnifications; Aggregating the feature vectors in the ROI sub-package by a multi-instance feature aggregation operator to generate a feature representation of the ROI sub-package; Classifying the feature representation of the ROI sub-package through the MLP module to predict the malignancy grade of the glioma of the ROI sub-package; The glioma malignancy level of the WSI image to be detected is determined according to the glioma malignancy level of each ROI sub-package.

7. The device according to claim 6, characterized in that The device also includes: A second acquisition module is used to acquire training data, wherein the training data includes a WSI image and a level label, and the level label is used to indicate the malignancy level of the glioma; A training module is used to train a preset model using the training data to obtain the glioma grading model.

8. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the glioma grading method based on tissue pathology images as described in any one of claims 1-5.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the glioma grading method based on tissue pathology images as described in any one of claims 1 to 5.

Citation Information

Patent Citations

  • A method and device for grading evaluation of brain glioma

    CN109271969A

  • Lung cancer computer-aided testing method based on deep learning and system thereof

    CN109754879A

  • Image classification method and device, computer equipment and storage medium

    CN110427970A

  • Intelligent judgment method and system for lung cancer histopathological typing

    CN112884724A

  • Computer supported review of tumors in histology images and post operative tumor margin assessment

    CN113366530A