Tumor retrieval library construction method and system based on WSI image global features

Abstract

The invention belongs to the field of image data information mining, and particularly relates to a tumor retrieval library construction method and system based on WSI image global features. According to the method, various types of frozen sections in different pathological states are obtained and converted into panoramic pathological scanning images, and quality control is carried out; performing pathology state labeling on the panoramic pathology scanning image and endowing a global pathology label; cutting the panoramic pathology scanning image into a plurality of color blocks at equal intervals, performing quality control to obtain a color block set, and performing internal parameter training on a pre-trained ViT model based on a DINO v2 self-supervised learning method; inputting the color block set into the trained ViT model for processing, and obtaining a descriptor matrix on a full connection layer; a geometric mean value of descriptors of all color blocks of a complete panoramic pathology scanning image and a global pathology label are taken according to the color blocks and are regarded as an instance, and all instances with known labels form a retrieval library. According to the invention, tumors can be found and traced quickly and accurately.

Description

Technical Field

[0001] The present invention belongs to the field of image data information mining, and particularly relates to a method and system for constructing a tumor retrieval library based on the global features of WSI images. Background Art

[0002] Promoting tumor prevention and treatment is an important topic in the current national health promotion plan. In tumor diagnosis and treatment, early detection is an important means affecting the progression of the disease; in addition, clarifying whether the tumor is in situ carcinoma or metastatic has a profound impact on the patient's treatment choice, prognosis evaluation, psychological state, and quality of life.

[0003] Pathology, as a key bridge between basic medicine and clinical medicine, provides objective criteria for judging the in situ and metastasis of tumors, the histological origin of tumor tissues, subsequent treatment decisions, and prognosis evaluation through histological morphological analysis. However, the shortcomings of conventional pathology are very obvious. When making a diagnosis, it is extremely dependent on manual analysis. This mode has an experience dependence, is time-consuming and laborious, and there may be problems such as artificial neglect of local key features due to tumor heterogeneity and possible omission of early micrometastasis or low-proportion gene mutations in complex cases. In addition, there are also problems such as high labor costs and difficult data management.

[0004] In recent years, with the popularization of whole-slide digital scanning technology (WSIs) and the rise of computational pathology, medical institutions and research platforms have gradually transformed traditional pathological sections into high-resolution panoramic pathological scanning images (usually reaching several GB for a single image). Due to the ultra-high resolution characteristics of panoramic pathological scanning images, most existing pathological artificial intelligence analyses are carried out at the field of view level, lacking an architecture for carrying out various analyses at the WSI level. Therefore, the existing solutions have a huge amount of computation, and it is difficult to directly obtain the results at the WSI level, resulting in difficulty in being applied in practice. At the same time, the existing methods mainly adopt classification schemes and are difficult to solve open-ended problems. Applying color block features to slice feature description to reduce the amount of computation and retain necessary information so that retrieval can be quickly realized is a necessary solution.

[0005] The panoramic pathological scanning image (WSI) data has a large volume, and its contained morphological features, spatial topological information, and multi-omics correlation value are becoming the core data sources for precision medicine and AI-assisted diagnosis. Therefore, it becomes possible to construct a retrieval library based on the pathological status and histological origin information of known panoramic pathological scanning image data for the discovery and traceability of tumor targets in unknown panoramic pathological scanning images. Summary of the Invention

[0006] The objective of the present invention is to provide a method and system for constructing a tumor retrieval library based on the global features of WSI images, which combines a computer vision model with image matching degree calculation, so as to meet the establishment of a retrieval library and matching search for panoramic pathological scanning images (WSI) from different sources and in different formats, and provide a method basis for subsequent pathological diagnosis and text report tasks.

[0007] The first aspect of the present invention provides a method for constructing a tumor retrieval library based on the global features of WSI images, including the following steps: Step 1: Obtain frozen sections of various categories with different pathological states, convert them into panoramic pathological scanning images (WSI) and perform quality control; Step 2: Label the pathological states of the panoramic pathological scanning images and assign global pathological labels; Step 3: Cut the panoramic pathological scanning images into a number of color patches at equal intervals, perform quality control to obtain a set of color patches; and use the set of color patches to train the internal parameters of a pre-trained ViT (Vision Transformer) model based on the DINO v2 self-supervised learning method; Step 4: Input the set of color patches of each panoramic pathological scanning image into the trained ViT model for processing, and obtain a descriptor matrix of the color patches at the fully connected layer; Step 5: Regard the geometric mean of all the descriptors of the color patches of a complete panoramic pathological scanning image and its global pathological label as an instance, and all the instances with known labels constitute the retrieval library.

[0008] Further, in Step 3, perform background removal processing on the panoramic pathological scanning images, retain the tissue area images in the panoramic pathological scanning images, and equally divide them into a number of color patches with the same physical distance according to the highest resolution, then perform quality control, and at the same time assign spatial position information to each color patch to obtain a set of color patches.

[0009] Further, in Step 4, use the trained ViT model to extract the features of the color patches, and obtain the global features of the panoramic pathological scanning image by taking the geometric mean of each color patch feature of all the color patches of a single panoramic pathological scanning image.

[0010] Further, in Step 5, for the 1024 geometric means of the color patch set of a single panoramic pathological scanning image, which are the global features of the panoramic pathological scanning image, and using the clinical diagnosis information as the label, establish the retrieval library.

[0011] Further, obtain panoramic pathological scanning images with unknown labels, perform Steps 1 to 4 to obtain the global features; calculate the matching degree between the panoramic pathological scanning images with unknown labels and the panoramic pathological scanning images in the retrieval library, and infer the tissue origin and pathological state based on the top ten data with the highest matching degree in the retrieval library.

[0012] Further, after obtaining the panoramic pathological scan image of the unknown label, it is equally spaced cut and quality controlled to obtain a set of color patch features, and the average value of the color patch features of each panoramic pathological scan image is used as the global feature of the panoramic pathological scan image of the unknown label.

[0013] The second aspect of the present invention provides a tumor retrieval library construction system based on the global features of WSI images, including: A panoramic pathological scan image scanning module for obtaining frozen sections of various categories with different pathological states, converting them into panoramic pathological scan images (WSI) and performing quality control; A labeling module for labeling the pathological state of the panoramic pathological scan image and assigning a global pathological label; An image segmentation module for equally spaced cutting the panoramic pathological scan image into several color patches and obtaining a color patch set after quality control; A feature extraction module for inputting the color patch set into a trained ViT model for processing and obtaining a descriptor matrix of the color patches in the fully connected layer; A retrieval library module for taking the geometric mean of the descriptors of all color patches of a complete panoramic pathological scan image and its global pathological label as an instance, and all instances with known labels constitute the retrieval library.

[0014] The third aspect of the present invention provides a terminal, which includes a memory, a processor, and a computer program stored in the memory. When the computer program is executed by the processor, it implements each step of the above-mentioned method for constructing a tumor retrieval library based on the global features of WSI images.

[0015] The fourth aspect of the present invention provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements each step of the above-mentioned method for constructing a tumor retrieval library based on the global features of WSI images.

[0016] The self-supervised learning method based on DINO v2 uses two main loss objectives: self-distillation loss and masked image modeling loss. The self-supervised learning method based on DINO v2 can generate visual features that are accurate and feasible in any task at the image level and pixel level. The present invention combines the ViT model and the image matching degree calculation method, and performs consistency analysis on the feature space projection output by the ViT model trained by the self-supervised learning method based on DINO v2 combined with pathological images, effectively overcoming the defect that the traditional matching degree calculation method is sensitive to data amplitude, and realizing the extraction of image descriptors from panoramic pathological scan images of different data sources to generate a retrieval library and be used for clinical retrieval. This strategy can meet various pathological tissue diagnosis tasks and provide a basis for related technologies such as intelligent pathological medical report generation. Brief Description of the Drawings

[0017] Figure 1 This is the flowchart of the present invention.

[0018] Figure 2 This is the tumor sample feature clustering diagram.

[0019] Figure 3 This is the normal sample feature clustering diagram.

[0020] Figure 4 This is the performance comparison between the ViT model trained by the present invention and the pre-trained ViT model. Detailed Embodiment

[0021] In order to be able to more clearly understand the above objects, retrieval process and advantages of the present invention, the following further details the present invention in combination with actual examples. The specific examples described here are only used to explain the present invention and are not used to limit the present invention.

[0022] Reference Figure 1 , a method for constructing a tumor retrieval library based on the global features of WSI images, includes the following steps: Step 1: Obtain frozen sections of various categories with different pathological states, convert them into panoramic pathological scan images (WSI) and perform quality control; Step 2: Label the pathological state of the panoramic pathological scan image and assign a global pathological label; Step 3: Cut the panoramic pathological scan image into several color blocks at equal intervals, perform quality control to obtain a color block set, and use the color block set to train the internal parameters of the pre-trained ViT (Vision Transformer) model based on the self-supervised learning method of DINO v2; Step 4: Input the color block set of each panoramic pathological scan image into the trained ViT model for processing, and obtain a descriptor matrix of the color blocks at the fully connected layer; Step 5: Regarding the descriptors of all color patches in a complete panoramic pathological scan image and their global pathological labels obtained by taking the geometric mean of the color patches as an instance, all instances with known labels form a retrieval library.

[0023] Furthermore, the frozen sections of various pathological states collected in Step 1 should depend on the specific goal. The frozen sections of different pathological states of each category should come from different individuals and the number should not be less than 30. The pathological tissue sources are clear, without damage, folding, the images are clear, without staining contamination and impurities, and are digitized by a slide scanner and saved in image formats such as svs, tif, png, etc. In this embodiment, the goal is set to establish a retrieval library containing descriptors of sections from various tissue sources and conduct a retrieval. Frozen pathological tissue sections without tissue, with damaged or overlapping tissue, or with severe staining contamination are excluded, and panoramic pathological scan images with clear and good image quality are retained. A total of 5509 frozen sections are collected, among which there are 200 cases of bladder urothelial carcinoma, 37 cases of non-cancerous bladder urothelium; 198 cases of breast invasive carcinoma, 130 cases of non-cancerous breast; 51 cases of cholangiocarcinoma, 20 cases of non-cancerous bile duct; 195 cases of colon cancer, 79 cases of non-cancerous colon; 173 cases of esophageal cancer, 64 cases of non-cancerous esophagus; 200 cases of head and neck squamous cell carcinoma, 38 cases of non-cancerous head and neck; 146 cases of renal chromophobe cell carcinoma, 199 cases of renal clear cell carcinoma, 198 cases of renal papillary cell carcinoma, 59 + 141 + 91 cases of non-cancerous kidney; 200 cases of hepatocellular carcinoma, 89 cases of non-cancerous liver; 200 cases of lung adenocarcinoma, 91 cases of non-cancerous lung adenocarcinoma; 195 cases of lung squamous cell carcinoma, 95 cases of non-cancerous lung; 197 cases of ovarian serous cystadenocarcinoma, 155 cases of non-cancerous ovary; 198 cases of pancreatic cancer, 33 cases of non-cancerous pancreas; 199 cases of prostate cancer, 114 cases of non-cancerous prostate; 199 cases of rectal adenocarcinoma, 25 cases of non-cancerous rectum; 198 cases of sarcoma, 20 cases of non-cancer; 62 cases of gastric cancer, 242 cases of non-cancerous stomach; 200 cases of thyroid cancer, 71 cases of non-cancerous thyroid; 248 cases of thymoma, 10 cases of non-cancerous thymus; 197 cases of endometrial cancer, 51 cases of non-cancerous endometrium; and they are digitized into panoramic pathological scan images by a pathological slide scanner.

[0024] Further, in step three, background removal processing is performed on the panoramic pathological scan image, and the tissue area image in the panoramic pathological scan image is retained. After being equally divided into several color blocks with the same physical distance according to the highest resolution, quality control is carried out, and at the same time, spatial position information is assigned to each color block to obtain a color block set. In this embodiment, referring to the maximum resolution of the panoramic pathological scan image, the panoramic pathological scan image is equally divided into color blocks with an actual physical distance of 128μm * 128μm, and quality control is performed on the cut color blocks. The quality control content includes: 1. Background removal, setting the threshold of the three channels of the image, and removing the color blocks with pure white, pure black or colors close to pure white or pure black; 2. Size control, determining the cutting size with reference to the maximum resolution of the panoramic pathological scan image. It is considered that a resolution of "40×" uses 512 pixels * 512 pixels, and "20×" uses 256 pixels * 256 pixels, and the color blocks with inconsistent sizes after segmentation are removed; 3. Sharpness control. Generally, when the sharpness threshold is set at 150, the texture and color of the color block are still meaningful, and the color blocks with sharpness lower than 150 are removed. A total of 8,263,740 color blocks are obtained using 5,509 frozen pathological sections from the TCGA dataset, and finally a color block set is obtained.

[0025] DINO v2 is a self-supervised learning method based on student-teacher knowledge distillation, using two main loss objectives: self-distillation loss and masked image modeling loss. The pre-trained ViT model refers to the ViT model trained with a public dataset. The present invention uses the self-supervised learning method based on DINO v2 to train the internal parameters of the pre-trained ViT (Vision Transformer) model. The binary classification effect of the ViT model trained in the present invention and the pre-trained ViT model on tumors and normal tissues from different pathological tissue sources can be seen Figure 4 , where the AUC-ROC values of breast cancer, colon cancer, lung adenocarcinoma, lung squamous cell carcinoma, prostate cancer, rectal adenocarcinoma, sarcoma, thymic carcinoma, endometrial cancer, bladder cancer, esophageal cancer, renal chromophobe cell carcinoma, ovarian cancer, pancreatic cancer, gastric cancer, thyroid cancer, cholangiocarcinoma, head and neck squamous cell carcinoma, renal clear cell carcinoma, renal papillary cell carcinoma, and liver cancer before and after training are 0.98, 1.0, 0.98, 0.96, 0.94, 1.0, 0.99, 1.0, 0.99, 1.0, 1.0, 0.99, 0.98, 0.99, 0.96, 1.0, 0.97, 0.96, 0.99, 0.97, 1.0; 0.67, 0.72, 0.85, 0.56, 0.68, 0.59, 0.63, 0.5, 0.63, 0.71, 0.55, 0.72, 0.68, 0.72, 0.57, 0.68, 0.62, 0.83, 0.71, 0.71, 0.7.

[0026] DINO v2 is a method for training a self-supervised learning computer vision model. Compared with general model methods, its advantage lies in being able to train images without any metadata. Specifically, DINO v2 uses a cross-entropy loss function to compare the classification tokens of the ViT model for two features extracted from different crops of the same image; DINO v2 randomly masks some input color patches, predicts the features of the masked areas based on two situations and compares them using the cross-entropy loss function, and separates the weights to avoid overfitting; DINO v2 uses a regularization method to calculate the differences between feature vectors to ensure their uniform distribution within the batch.

[0027] In step four, 1024 color patch features are extracted through the trained ViT model, and the global feature of the panoramic pathological scan image is obtained by taking the geometric mean of each feature of all color patches of a single panoramic pathological scan image.

[0028] In step five, for the 1024 geometric means of the color patch set of a single panoramic pathological scan image, which are the global features of the panoramic pathological scan image, a retrieval library is established with the clinical diagnosis information as the label.

[0029] Furthermore, in step five, a panoramic pathological scan image with an unknown label in practical applications is obtained, and its image descriptor is obtained, such as Figure 2 and Figure 3The image descriptors in UMAP feature clustering are shown. UMAP (Uniform Manifold Approximation and Projection) is a non-linear dimensionality reduction technique commonly used for the visualization of high-dimensional data. UMAP is used to visualize high-dimensional average features. Among them, groups 0, 1, 2, 3, 4, 6, 8, 11, 14, 16, 17, 18, 20, 21, 22, 23, 25, 27, 29, 30, 32, 34, 35, 39, 41 only include tumor samples, groups 7, 24, 40 only include normal samples of each tissue, group 9 is the tumor and normal of prostate cancer, groups 10, 12, 15, 31, 33, 40 include samples from the kidney, groups 5, 19, 26, 28, 36, 37 include samples from the lung, and group 38 is ovarian serous cystadenocarcinoma. This indicates that the global features extract important features related to diseases and tissue sources. It can be seen that the points representing diseased and normal sections in the figure can be clearly distinguished, which indicates that the relevant features extracted by the image descriptors are accurate and feasible in the classification task. The global features of panoramic pathological scan images with known labels and the global features of panoramic pathological scan images with unknown labels are amplitude-normalized to eliminate the interference of data scales; the projection relationship between the above feature vectors is calculated, and the consistency of their spatial directions is quantified to generate a matching score matrix. Each score in this matrix corresponds to a panoramic pathological scan image with a known label, and the most matching image label in the retrieval library is used as the discriminant label for the unknown image.

[0030] Obtain a panoramic pathological scan image with an unknown label, perform steps one to four to obtain global features; calculate the matching degree between the panoramic pathological scan image with an unknown label and the panoramic pathological scan images in the retrieval library, and infer the tissue source and pathological status based on the top ten data with the highest matching degrees in the retrieval library.

[0031] After obtaining a panoramic pathological scan image with an unknown label, perform equidistant cutting on it and perform quality control to obtain a set of color patch features, and use the average value of the color patch features of each panoramic pathological scan image as the global feature of the panoramic pathological scan image with an unknown label.

[0032] The retrieval library described in this embodiment is established based on 5509 panoramic pathological scan images of 19 cancer types in the TCGA public database. 200 cases are randomly selected from the retrieval library and the corresponding descriptors are masked from the retrieval library. This method is used for 50-fold cross-validation. The correct recognition rate for the search and recognition of normal tissues is 99%, and the correct recognition rate for the search and recognition of tumor tissues is 99%. Table 1 shows the average value and standard deviation of the error probability in the retrieval of frozen sections of different tissues for disease status, indicating that this method has good feasibility in image retrieval.

[0033] Table 1

[0034] The second embodiment of the present invention provides a tumor retrieval library construction system based on the global features of WSI images, including: A panoramic pathological scanning image scanning module, which is used to obtain frozen sections of various categories with different pathological states, convert them into panoramic pathological scanning images (WSI) and perform quality control; A labeling module, which is used to label the pathological state of the panoramic pathological scanning image and assign a global pathological label; An image segmentation module, which is used to equally cut the panoramic pathological scanning image into several color patches, and obtain a color patch set after quality control; A feature extraction module, which is used to input the color patch set of the panoramic pathological scanning image to be recognized into a trained ViT model for processing, and obtain a descriptor matrix of the color patches in the fully connected layer; A retrieval library module, which is used to regard the geometric mean value of the color patches and its global pathological label of all color patches of a complete panoramic pathological scanning image as an instance, and all instances with known labels constitute a retrieval library.

[0035] The third embodiment of the present invention provides a terminal, which includes a memory, a processor, and a computer program stored in the memory. When the computer program is executed by the processor, it realizes each step of the above-mentioned method for constructing a tumor retrieval library based on the global features of WSI images.

[0036] The fourth embodiment of the present invention provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it realizes each step of the method for constructing a tumor retrieval library based on the global features of WSI images as described above.

[0037] The above embodiments of the present invention are merely examples for clearly illustrating the present invention, rather than limitations on the implementation manners of the present invention. For those of ordinary skill in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to enumerate all the implementation manners here. Any modifications, equivalent replacements, and improvements made within the spirit and principle of the present invention shall be included in the protection scope of the claims of the present invention.

Claims

1. A method for constructing a tumor retrieval library based on the global features of WSI images, characterized in that: The following steps are involved: Step 1: Obtain frozen sections of various types and pathological conditions, convert them into panoramic pathology scan images and perform quality control; Step 2: Annotate the pathological status of the panoramic pathological scan image and assign a global pathological label; Step 3: Cut the panoramic pathology scan image into several color blocks at equal distances, and obtain a color block set after quality control; and use the color block set to train the internal parameters of the pre-trained ViT model based on the DINO v2 self-supervised learning method; Step 4: Input the color block set of each panoramic pathology scan image into the trained ViT model for processing, and obtain the descriptor matrix of the color block in the fully connected layer; Step 5: The descriptors of all color blocks of a complete panoramic pathology scan image are regarded as an instance according to the geometric mean of the color blocks and their global pathology labels, and all instances of known labels constitute a retrieval library.

2. The method for constructing a tumor retrieval library based on global features of WSI images according to claim 1, characterized in that: In step three, the panoramic pathology scan image is subjected to background removal processing, the tissue area image in the panoramic pathology scan image is retained, and it is equidistantly divided into several color blocks with the same physical distance according to the highest resolution, and then quality control is performed. At the same time, spatial position information is given to each color block to obtain a color block set.

3. The method for constructing a tumor retrieval library based on global features of WSI images according to claim 1, characterized in that: In the step 4, the trained ViT model is used to extract color block features, and each color block feature of all color blocks of a single panoramic pathology scan image is geometrically averaged to obtain the global features of the panoramic pathology scan image.

4. The method for constructing a tumor retrieval library based on global features of WSI images according to claim 1, characterized in that: In the step 5, the 1024 geometric mean values ​​of the color block set of a single panoramic pathology scan image are used as the global features of the panoramic pathology scan image, and the clinical diagnosis information is used as a label to establish a retrieval library.

5. The method for constructing a tumor retrieval library based on WSI image global features according to claim 1, characterized in that: Obtain a panoramic pathology scan image with unknown labels, perform steps 1 to 4, and obtain global features; The matching degree between the panoramic pathology scan image with unknown label and the panoramic pathology scan image in the retrieval library is calculated, and the tissue origin and pathological status are inferred based on the ten data with the highest matching degree in the retrieval library.

6. The method for constructing a tumor retrieval library based on WSI image global features according to claim 1, characterized in that: After obtaining the panoramic pathology scan image with unknown label, it is cut into equal distances and quality controlled to obtain a set of color block features. The average value of the color block features of each panoramic pathology scan image is taken as the global feature of the panoramic pathology scan image with unknown label.

7. A tumor retrieval library construction system based on WSI image global features, characterized in that: include: Panoramic pathology scanning image scanning module, used to obtain frozen sections of various types and different pathological conditions, convert them into panoramic pathology scanning images and perform quality control; An annotation module, used to annotate the pathological status of the panoramic pathological scan image and assign a global pathological label; An image segmentation module is used to equidistantly cut the panoramic pathology scan image into a number of color blocks, and obtain a color block set after quality control; The feature extraction module is used to input the color block set into the trained ViT model for processing and obtain the descriptor matrix of the color block in the fully connected layer; The retrieval library module is used to regard the descriptors of all color blocks of a complete panoramic pathology scan image as an instance according to the geometric mean of the color blocks and their global pathology labels, and all instances of known labels constitute the retrieval library.

8. A terminal comprising a memory, a processor, and a computer program stored in the memory, characterized in that: When the computer program is executed by the processor, each step of the method for constructing a tumor retrieval library based on global features of WSI images according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, each step of the method for constructing a tumor retrieval library based on WSI image global features as described in any one of claims 1 to 6 is implemented.

Images