Method and device for classifying tissues in pathological image based on adaptive clustering

Abstract

The invention discloses a method and equipment for classifying tissues in a pathological image based on adaptive clustering. The method comprises the following steps: acquiring a to-be-classified pathological image; processing the pathological image according to a preset zoom ratio to obtain a thumbnail; segmenting the thumbnail to obtain a tissue mask image; dividing the tissue mask image into a plurality of image blocks, and recording the coordinate position of each image block; extracting image semantic features and statistical features of each image block; fusing the semantic features and the statistical features of the image; clustering is carried out based on fused features, and all image blocks are divided into k class clusters; predicting an organization category of each class cluster according to a clustering result; the prediction result is optimized and corrected; and according to the coordinate positions of the image blocks, splicing the image blocks of different organization categories in different colors to form a new thumbnail, and carrying out visual display. According to the method, classification tasks of different pathological images can be adaptively processed, the prediction time is shortened, and the classification efficiency is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of pathological image processing, and in particular relates to a method and device for classifying tissue in pathological images based on adaptive clustering. Background Art

[0002] In the medical field, pathological image analysis is the cornerstone of disease diagnosis and research.

[0003] Traditional pathological image classification methods usually use manual annotation. The number of slice image blocks of a (WSI) full digital pathological image ranges from tens of thousands to hundreds of thousands, which is both time-consuming and laborious.

[0004] With the development of artificial intelligence and deep learning technology, intelligent pathological image classification has gradually replaced manual annotation and has become an important means of medical diagnosis and research. A series of existing pathological image classification methods based on deep learning attempt to improve classification accuracy by optimizing model structure or adjusting training strategies. Although these methods have improved the generalization performance of the model to a certain extent, they have put forward higher requirements on the hardware equipment for deploying and running the model, greatly increased the computational complexity and amount of calculation, and are inefficient, making it even more difficult to meet the actual needs of large-scale pathological image classification.

[0005] Different pathological images have different staining styles, different acquisition sites, different image sizes, etc., which leads to the problem of poor generalization of deep learning methods and the inability to correctly classify pathological image blocks.

[0006] Due to the differences in image quality, brightness, contrast, and color parameters of different pathological images, directly using deep learning models to classify images may lead to inaccurate classification results. Therefore, the current mainstream practice is to perform color standardization and data normalization before processing pathological image data to improve the accuracy and stability of pathological image classification. However, these processes require a lot of computing resources and time costs, especially for large-scale pathological image data, the processing time and cost will increase dramatically. The dilemma of the image preprocessing step is that the tens of thousands of image blocks generated by pathological images must be processed uniformly without distinction to avoid the impact of data with different distributions and their noise on the classification model. This will bring huge hardware costs and energy consumption problems, and require the use of high-performance computing equipment and large-scale storage systems. The problem of low efficiency seriously restricts the timely progress of medical diagnosis and disease research. Summary of the invention

[0007] In order to solve the above technical problems, the present invention proposes a method and device for classifying tissue in pathological images based on adaptive clustering, which can effectively solve the problems of reliance on manual labeling, low efficiency and poor generalization in the process of pathological image classification.

[0008] In order to achieve the above object, the technical solution of the present invention is as follows:

[0009] On the one hand, the present invention discloses a method for classifying tissue in pathological images based on adaptive clustering, comprising:

[0010] Step S1: obtaining a pathological image to be classified;

[0011] Step S2: Processing the pathological image according to a preset zoom ratio to obtain a thumbnail;

[0012] Step S3: segmenting the thumbnail to obtain a tissue mask image;

[0013] Step S4: dividing the tissue mask image into a number of image blocks, and recording the coordinate position of each image block;

[0014] Step S5: extracting image semantic features of each image block using a pre-trained feature extraction model;

[0015] Utilize statistical methods to extract statistical features of each image block;

[0016] Fusion of image semantic features and statistical features;

[0017] Step S6: clustering based on the fused features, dividing all image blocks into k clusters;

[0018] Step S7: predicting the tissue category of each cluster according to the clustering results;

[0019] Step S8: Optimizing and correcting the prediction results;

[0020] Step S9: according to the coordinate positions of the image blocks, the image blocks of different tissue categories are spliced ​​in different colors to form new thumbnails for visual display.

[0021] Based on the above technical solution, the following improvements can be made:

[0022] As a preferred solution, in step S5, the R value of each image block or the histogram of the grayscale image is counted as the statistical feature of the image block.

[0023] As a preferred solution, step S6 includes:

[0024] Step S6.1: fusing the image semantic features and statistical features of each image block to form a new image block feature;

[0025] Step S6.2: randomly select k image block features as initial clustering centers;

[0026] Step S6.3: Calculate the distance from each image block feature to each cluster center using equation (1), and assign each image block feature to the cluster where the nearest cluster center is located, and obtain k clusters {s1, s2, …, s k};

[0027]

[0028] Where: x i is the feature of the i-th image block, 1≤i≤n;

[0029] c j is the j-th cluster center, 1≤j≤k;

[0030] xv i is the image semantic feature of the i-th image block feature;

[0031] cv j is the image feature of the jth cluster center;

[0032] xh i is the histogram feature of the i-th image block feature;

[0033] ch j is the histogram feature of the jth cluster center;

[0034] α and β are modulation factors;

[0035] Step S6.4: For each cluster, recalculate formula (2) and select a new cluster center;

[0036]

[0037] Among them, c l is the lth cluster center, 1≤l≤k;

[0038] |s l | represents the number of image block features in the lth cluster;

[0039] x i represents the feature of the i-th image block in the l-th cluster, 1≤i≤|s l |;

[0040] Step S6.5: Repeat steps S6.3 to S6.4 until the stop condition is met.

[0041] As a preferred solution, step S7 includes:

[0042] Step S7.1: pre-build a pathology image block database, wherein the pathology image block database stores a number of image blocks of known tissue categories;

[0043] Step S7.2: matching the size of the image block to be matched with the image block in the pathological image block database;

[0044] If they match, continue to step S7.3;

[0045] If there is no match, the size of the image block in the pathological image block database is cropped to be consistent with the size of the image block to be matched, and then step S7.3 is continued;

[0046] Step S7.3: comparing the feature mean of any cluster image block to be matched with the feature mean of each known tissue category image block in the pathology image block database;

[0047] According to the comparison result, the tissue category with the smallest feature mean difference is assigned to the cluster as its label. As a preferred solution, step S8 includes:

[0048] Step S8.1: forming a category label map according to the coordinate positions of the image blocks and the organizational categories of the image blocks;

[0049] Step S8.2: traverse the tissue category of each image block in the category label map according to the traversal direction. If the tissue category of the image block with coordinates (x, y) is different from the tissue categories of other image blocks in its connected domain, record it using the category count matrix Count(x, y, label), where label is the tissue category label that appears most frequently in other image blocks in its connected domain.

[0050] Step S8.3: switch to other traversal directions, and repeat step S8.2 until all traversal directions have been traversed;

[0051] Step S8.4: According to the category count matrix Count(x, y, label), the tissue category of each image block is corrected. In addition, on the other hand, the present invention also discloses a pathological image tissue classification device based on adaptive clustering, comprising:

[0052] A pathological image acquisition module, used for acquiring pathological images to be classified;

[0053] A zoom module, used to process the pathological image according to a preset zoom ratio to obtain a thumbnail;

[0054] A segmentation module is used to segment the thumbnail image to obtain a tissue mask image;

[0055] A division module is used to divide the tissue mask image into a number of image blocks and record the coordinate position of each image block;

[0056] A feature extraction module, used to extract image semantic features of each image block using a pre-trained feature extraction model;

[0057] Utilize statistical methods to extract statistical features of each image block;

[0058] Fusion of image semantic features and statistical features;

[0059] Clustering module, used to cluster the fused features and divide all image blocks into k clusters;

[0060] The tissue category prediction module is used to predict the tissue category of each cluster based on the clustering results;

[0061] Optimization module, used to optimize and correct the prediction results;

[0062] The visualization display module is used to splice image blocks of different tissue categories in different colors according to the coordinate positions of the image blocks to form new thumbnails for visualization display.

[0063] As a preferred solution, in the feature extraction module, the R value of each image block or the histogram of the grayscale image is counted as the statistical feature of the image block.

[0064] As a preferred solution, the clustering module includes:

[0065] A sample point forming unit is used to fuse the image semantic features and statistical features of each image block to form a new image block feature;

[0066] Initialization unit, used to randomly select k image block features as initial clustering centers;

[0067] The allocation unit is used to calculate the distance from each image block feature to each cluster center through formula (1), and allocate each image block feature to the cluster where the nearest cluster center is located, so as to obtain k clusters {s1, s2, ..., s k};

[0068]

[0069] Where: x i is the feature of the i-th image block, 1≤i≤n;

[0070] c j is the j-th cluster center, 1≤j≤k;

[0071] xv i is the image semantic feature of the i-th image block feature;

[0072] cv j is the image feature of the jth cluster center;

[0073] xh i is the histogram feature of the i-th image block feature;

[0074] ch j is the histogram feature of the jth cluster center;

[0075] α and β are modulation factors;

[0076] The adjustment unit is used to recalculate the equation (2) for each cluster and select a new cluster center;

[0077]

[0078] Among them, c l is the lth cluster center, 1≤l≤k;

[0079] |s l | represents the number of image block features in the lth cluster;

[0080] x i represents the feature of the i-th image block in the l-th cluster, 1≤i≤|s l |;

[0081] The repeating execution unit is used to repeatedly execute the methods in the allocation unit and the adjustment unit until a stop condition is met.

[0082] As a preferred solution,

[0083] The tissue category prediction module includes:

[0084] A database construction unit, used to pre-construct a pathological image block database, wherein the pathological image block database stores a number of image blocks of known tissue categories;

[0085] A size matching unit, used for matching the image block to be matched with the image block in the pathological image block database;

[0086] If they match, continue to execute the method in the feature comparison unit;

[0087] If there is no match, the size of the image block in the pathological image block database is cut to be consistent with the size of the image block to be matched, and then the method in the feature comparison unit is continued;

[0088] A feature comparison unit, used for comparing the feature mean of any cluster image block to be matched with the feature mean of each known tissue category image block in the pathology image block database;

[0089] According to the comparison results, the tissue category with the smallest feature mean difference is assigned to the cluster as its label.

[0090] As a preferred solution, the optimization module includes:

[0091] A category label map forming unit, used to form a category label map according to the coordinate positions of the image blocks and the organizational categories of the image blocks;

[0092] A traversal unit is used to traverse the tissue category of each image block in the category label map according to the traversal direction. If the tissue category of the image block with coordinates (x, y) is different from the tissue categories of other image blocks in its connected domain, the category count matrix Count(x, y, label) is used to record it, and label is the tissue category label that appears most frequently in other image blocks in its connected domain.

[0093] A repeated execution unit is used to switch to other traversal directions and repeatedly execute the method in the traversal unit until all traversal directions have completed traversal;

[0094] The correction unit is used to correct the tissue category of each image block according to the category count matrix Count(x, y, label).

[0095] The present invention discloses a method and device for classifying tissue in pathological images based on adaptive clustering, which can adaptively process classification tasks of different pathological images, significantly reduce prediction time, improve classification efficiency, help to timely discover diseases and formulate treatment plans, and also have important practical significance for large-scale pathological image data processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0096] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0097] Figure 1 A flowchart of a method for classifying tissue in a pathological image provided by an embodiment of the present invention.

[0098] Figure 2 (a) is an image block with a category label of fat provided in an embodiment of the present invention;

[0099] Figure 2 (b) An image block with a category label of necrosis provided by an embodiment of the present invention;

[0100] Figure 2 (c) an image block with a category label of lymph provided by an embodiment of the present invention;

[0101] Figure 2 (d) An image block with a category label of mucus provided in an embodiment of the present invention;

[0102] Figure 2 (e) an image block with a category label of muscle provided in an embodiment of the present invention;

[0103] Figure 2 (f) an image block with a category label of intestinal mucosa provided in an embodiment of the present invention;

[0104] Figure 2 (g) an image block with a matrix as a category label provided by an embodiment of the present invention;

[0105] Figure 2 (h) An image block with a category label of tumor provided in an embodiment of the present invention.

[0106] Figure 3 (a) is a tissue mask image provided by an embodiment of the present invention;

[0107] Figure 3 (b) is the division of the tissue mask image shown in 3(a) into several image blocks of 64um×64um physical size;

[0108] Figure 3 (c) Figure 3 (b) The corresponding final visualization thumbnail;

[0109] Figure 3 (d) Figure 3 The tissue mask image shown in (a) is divided into several image blocks with a physical size of 128um×128um;

[0110] Figure 3 (e) Figure 3 (d) The corresponding final visualization thumbnail;

[0111] Figure 3 (f) Figure 3 The tissue mask image shown in (a) is divided into several image blocks with a physical size of 256um×256um;

[0112] Figure 3 (g) Figure 3 (f) The corresponding final visualization thumbnail.

[0113] Figure 4 (a) is one of the pathological images of colorectal cancer provided by an embodiment of the present invention;

[0114] Figure 4 (b) Figure 4 (a) The corresponding final visualization thumbnail;

[0115] Figure 4 (c) is the second pathological image of colorectal cancer provided by an embodiment of the present invention;

[0116] Figure 4 (d) Figure 4 (c) The corresponding final visualization thumbnail;

[0117] Figure 4 (e) A pathological image of cervical squamous cell carcinoma provided by an embodiment of the present invention;

[0118] Figure 4 (f) Figure 4 (e) The corresponding final visualization thumbnail;

[0119] Figure 4 (g) A pathological image of lung cancer provided by an embodiment of the present invention;

[0120] Figure 4 (h) Figure 4 (g) The corresponding final visualization thumbnail.

[0121] Figure 5 (a) is an image block of the intestinal mucosa provided in an embodiment of the present invention;

[0122] Figure 5 (b) is an image block of a tumor provided by an embodiment of the present invention.

[0123] Figure 6 (a) is a schematic diagram of the intestinal mucosa and tumor features before fusion provided by an embodiment of the present invention;

[0124] Figure 6 (b) is a schematic diagram of the fusion of the two types of features of intestinal mucosa and tumor provided in an embodiment of the present invention.

[0125] Figure 7 (a) is a heat map obtained by calculating the Euclidean distance after the two types of features of intestinal mucosa and tumor are fused according to an embodiment of the present invention;

[0126] Figure 7 (b) is a heat map obtained by calculating the fusion of the two types of features of intestinal mucosa and tumor using formula (1) provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0127] The preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0128] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0129] The expression of “comprising” an element is an “open” expression, which merely means that a corresponding component or step exists, and should not be interpreted as excluding additional components or steps.

[0130] In order to achieve the purpose of the present invention, in some embodiments of the method and device for classifying tissue in pathological images based on adaptive clustering,

[0131] like Figure 1 As shown, the tissue classification method in the pathological image includes:

[0132] Step S1: obtaining a pathological image to be classified;

[0133] Step S2: Processing the pathological image according to a preset zoom ratio to obtain a thumbnail;

[0134] Step S3: segmenting the thumbnail to obtain a tissue mask image;

[0135] Step S4: dividing the tissue mask image into a number of image blocks, and recording the coordinate position of each image block;

[0136] Step S5: extracting image semantic features of each image block using a pre-trained feature extraction model;

[0137] Use statistical methods to extract statistical features of each image block, such as histogram features;

[0138] Fusion of image semantic features and statistical features;

[0139] Step S6: clustering based on the fused features, dividing all image blocks into k clusters;

[0140] Step S7: predicting the tissue category of each cluster according to the clustering results;

[0141] Step S8: Optimizing and correcting the prediction results;

[0142] Step S9: according to the coordinate positions of the image blocks, the image blocks of different tissue categories are spliced ​​in different colors to form new thumbnails for visual display.

[0143] Each step is described in detail below.

[0144] Step S1, obtaining a pathological image WSI to be classified.

[0145] The pathological image WSI to be classified can be of any staining style, any acquisition site, and any image size.

[0146] Step S2, processing the pathological image WSI according to a preset zoom ratio to obtain a thumbnail thumb.

[0147] The size of the pathological image WSI layer 0 is H×B (height×width), the zoom ratio is set to B, and the size of the thumbnail is:

[0148] h×w=(H / B)×(W / B).

[0149] Step S3, using a threshold segmentation method to segment the thumbnail thumb, separating the tissue and background of the thumbnail thumb, and obtaining a tissue mask image.

[0150] The threshold segmentation formula is as follows:

[0151] mask t =R≤220∩G≤220∩B≤220

[0152] Among them: mask t represents the tissue mask;

[0153] R, G, and B represent the RGB three-channel values ​​of the thumbnail thumb respectively.

[0154] Because the background of the pathological image WSI is not the focus, removing the background can not only reduce redundant output, but also speed up the export of image tiles.

[0155] Of course, in some other embodiments, a deep learning model (such as MobileNet, ShuffleNet, etc.) may also be used to organize and segment the thumbnails.

[0156] Step S4: Divide the tissue mask image into a number of image tiles, and record the coordinate position of each image tile.

[0157] After obtaining the tissue mask image, define the physical size of the image tile length and width, such as: (64um×64um), (128um×128um), (256um×256um), etc. The conversion formula between physical size and pixel point is as follows:

[0158] Pixel = Actual size / mpp

[0159] Among them: mpp is the attribute recorded in the pathological image WSI itself, Pixel means pixel point, and Actual size refers to the actual physical size.

[0160] According to the organization mask t and Pixel, record the coordinate position of the corresponding image tile (row coordinate c, column coordinate r) as the image identifier. The size of the image tile determines the number and time of exporting the image tile.

[0161] The present invention exports corresponding image tiles according to a tissue mask image, reduces the number of redundant background image tiles to be exported, and improves export efficiency.

[0162] Step S5, extracting image semantic features of each image tile using a pre-trained feature extraction model;

[0163] The R value of each image block tile or the histogram of the grayscale image is counted as the histogram feature of the image block tile.

[0164] The present invention uses image semantic features and statistical features to fuse to obtain richer information. Statistical features include: image texture, color and histogram features. Although statistical features have problems such as small amount of information and fuzziness, they still have reference value. For example, histogram features are not affected by image rotation transformation. Recently, self-supervised learning has made image semantic features a new research direction.

[0165] The pre-trained feature extraction model is obtained through training on a large number of pathological images, which can effectively extract the tissue structure information in the image blocks and has better generalization performance. In addition, this method is used to compress the image and improve the efficiency of subsequent classification tasks.

[0166] The feature extraction model of the present invention can use a public self-supervised learning model, such as using the VisionTransformer model as the backbone network and training on millions of pathological images. Different from statistical features, the features output by the self-supervised model have been proven to have richer information and characteristics. This self-supervised learning can greatly simplify the use of images in downstream tasks and achieve better performance in downstream tasks.

[0167] In addition, the present invention extracts the histogram features of the image blocks for feature fusion, and uses h5 files to store and save the features for subsequent processing.

[0168] In existing solutions, machine learning or deep learning methods require training with a large number of images, which is highly dependent on equipment and consumes a lot of time.

[0169] By converting the image of the image block into a one-dimensional feature vector through a self-supervised learning model, the amount of computation and storage space based on the image block scale can be significantly reduced, the efficiency of downstream tasks can be improved, and better prediction results can be achieved.

[0170] The present invention uses a pre-trained deep learning model to extract image semantic features and a statistical method to extract histogram features. The step of converting images into features can significantly reduce the processing requirements of the entire image data set, thereby reducing computational complexity and time cost.

[0171] Step S6, clustering is performed based on the image semantic features and histogram features of each image tile, and all image tiles are divided into k clusters.

[0172] Specifically, step S6 includes:

[0173] Step S6.1: The image semantic features and histogram features of each image tile are fused to form a new image tile feature. The n image tile features {x1, x2, …, x n};

[0174] Step S6.2: Randomly select k image block features as initial cluster centers, k cluster centers {c1, c2, …, c k};

[0175] Step S6.3: Calculate the distance from each image block feature to each cluster center using equation (1), and assign each image block feature to the cluster where the nearest cluster center is located, and obtain k clusters {s1, s2, …, s k};

[0176]

[0177] Where: x i is the feature of the i-th image block, 1≤i≤n;

[0178] c j is the j-th cluster center, 1≤j≤k;

[0179] xv i is the image semantic feature of the i-th image block feature;

[0180] cv j is the image feature of the jth cluster center;

[0181] xh i is the histogram feature of the i-th image block feature;

[0182] ch j is the histogram feature of the jth cluster center;

[0183] α and β are modulation factors;

[0184] Step S6.4: For each cluster, recalculate formula (2) and select a new cluster center;

[0185]

[0186] Among them, c l is the lth cluster center, 1≤l≤k;

[0187] |sl | represents the number of image block features in the lth cluster;

[0188] x i represents the feature of the i-th image block in the l-th cluster, 1≤i≤|s l |;

[0189] Step S6.5: Repeat steps S6.3 to S6.4 until the stop condition is met.

[0190] The stopping condition can be that the cluster center no longer changes significantly or the preset maximum number of iterations is reached.

[0191] The clustering algorithm is used to classify the features of all image tiles of a single pathological image WSI, and all image tiles of a single pathological image WSI are automatically divided into different categories. The clustering algorithm does not require manual labeling and can process pathological images of different staining styles, different collection sites, and different image sizes.

[0192] When classifying pathological images of colorectal cancer, k is 8, which means that all image tiles are finally divided into 8 categories, which may include intestinal mucosa, lymph, necrosis, matrix and tumor. According to formula (1), the distance from each image tile to the cluster center is calculated, and the image tile with the smallest distance to the cluster center is assigned to its own cluster, and each image tile is assigned a category label of 1, 2, ..., k.

[0193] Step S7 is used to define a specific tissue category for each cluster, including:

[0194] Step S7.1: pre-constructing a pathological image block database, wherein the pathological image block database stores a number of image blocks of known tissue categories;

[0195] Specifically, the pathology image block database is established based on the previously collected pathology image WSI. The physical size of the image blocks in the pathology image block database is 512um×512um, and the image blocks include different staining styles and different tumors.

[0196] The pathological image block database is determined by pathology experts, and the image blocks are divided into 8 categories: intestinal mucosa, lymph, necrosis, matrix, tumor, muscle, mucus, and fat. The feature mean of each tissue category is calculated and stored;

[0197] Step S7.2: matching the size of the image block to be matched (e.g., 64um×64um) with the image block in the pathology image block database;

[0198] The image blocks to be matched are the above clustering results, that is, k clusters {s1, s2, …, s k};

[0199] If they match, continue to step S7.3;

[0200] If there is no match, the size of the image block in the pathology image block database is cropped to be consistent with the size of the image block to be matched (i.e., 64um×64um), and then step S7.3 is continued;

[0201] Step S7.3: comparing the feature mean of any cluster image block to be matched with the feature mean of each known tissue category image block in the pathology image block database;

[0202] According to the comparison results, the tissue category with the smallest feature mean difference is assigned to the cluster as its label.

[0203] Since there may be thousands or even tens of thousands of images in each category, matching based on image pixel values ​​would consume a huge amount of time and resources. Therefore, calculation based on cluster features can effectively speed up the retrieval process.

[0204] Step S8, optimizing and correcting the prediction result of each image block tile.

[0205] Step S8.1: forming a category label map label_map according to the coordinate position of the image block and the organizational category of the image block;

[0206] Step S8.2: traverse the tissue category of each image block in the category label map label_map according to the traversal direction (e.g., from left to right). If the tissue category of the image block with coordinates (x, y) is different from the tissue categories of other image blocks in its connected domain, use the category count matrix Count(x, y, label) to record it, where label is the tissue category label that appears most frequently in other image blocks in its connected domain.

[0207] Step S8.3: switch to other traversal directions, and repeat step S8.2 until all traversal directions are traversed. There are four traversal directions: from left to right, from right to left, from top to bottom, and from bottom to top;

[0208] Step S8.4: Correct the tissue category of each image block according to the category count matrix Count(x, y, label).

[0209] The tissue category corresponding to the maximum value of the category count matrix is ​​selected to correct the category label of the image block.

[0210] The present invention classifies based on cluster centers, but there may be scattered tissue blocks, which may be caused by outliers. The present invention uses a connected domain scanning algorithm to repair the category image. Scan the image and find and mark all connected domains in the image.

[0211] The present invention predicts and classifies based on the pathological image tissue block database, uses feature matching to save database resource occupancy and matching speed, and corrects the results according to the connected domain algorithm record.

[0212] Step S9, according to the coordinate positions of the image tiles, the image tiles of different tissue categories are spliced ​​in different colors to form new thumbnails for visual display.

[0213] The present invention uses different colors to visualize image blocks of each category label on thumbnails according to the row and column coordinates of each image tile and the category label. According to the mapping of the classification results from the pathological image WSI level to the thumbnail level, the pathologist can see the classification results at high resolution on the thumbnail. Finally, according to the tissue category, the images are exported to their respective folders, and the rapid classification results are fed back to users or medical experts in a timely manner, providing strong support for medical diagnosis and disease research.

[0214] Figure 2 (a) Figure 2 (b) Figure 2 (c) Figure 2 (d) Figure 2 (e) Figure 2 (f) Figure 2 (g) Figure 2 (h) Displays image tiles with category labels of fat, necrosis, lymph, mucus, muscle, intestinal mucosa, matrix, and tumor, respectively.

[0215] In order to better demonstrate the effectiveness of the present invention, Table 1 records the image blocks of different physical sizes (64um×64um, 128um×128um, 256um×256um) derived from a pathological image and the time consumption of each step, in seconds (s).

[0216] Table 1 Time consumption table of each step

[0217]

[0218] Figure 3 (a) is the tissue mask image.

[0219] Figure 3 (b) is to divide the tissue mask image shown in 3(a) into several image blocks with a physical size of 64um×64um. Figure 3 (c) Figure 3 (b) The corresponding final visualization thumbnail.

[0220] Figure 3 (d) Figure 3 The tissue mask image shown in (a) is divided into several image blocks with a physical size of 128um×128um. Figure 3 (e) Figure 3 (d) The corresponding final visualization thumbnail. Figure 3 (f) Figure 3 The tissue mask image shown in (a) is divided into several image blocks with a physical size of 256um×256um. Figure 3 (g) Figure 3 (f) The corresponding final visualization thumbnail.

[0221] The above results show that the present invention can process image tiles of different physical sizes. Since different physical sizes of image tiles will lead to different classification results, among which, the physical size of 64um×64um is more prominent in details.

[0222] Figure 4 (a) Figure 4 (c) Figure 4 (e) Figure 4 (g) shows four original pathological images WSI, which are from different collection sites, including colorectal cancer, squamous cell carcinoma, etc. Figure 4 (a) and Figure 4 (c) is a colorectal cancer pathology image. Figure 4 (e) is the pathological image of cervical squamous cell carcinoma. Figure 4 (g) is a lung cancer pathology image. They not only have differences in the acquisition sites, but also have different staining styles, artifacts and other issues. Existing methods cannot correctly classify these images.

[0223] After the invention of the present invention, the final visualization result is as follows Figure 4 (b) Figure 4 (d) Figure 4 (f) Figure 4 As shown in (h), the surface of the present invention is not affected by the dyeing style or different parts, and has good generalization and accuracy.

[0224] The present invention can adaptively classify image tiles of pathological images WSI of different scales and styles, improve efficiency, and accelerate the pathological image classification process.

[0225] Further, in order to verify that the present invention has excellent classification effect.

[0226] Figure 5 (a) Figure 5 (b) Image blocks of intestinal mucosa and tumor, respectively.

[0227] Figure 6 (a) Figure 6 (b) are the images before and after the fusion of the two types of features of intestinal mucosa and tumor. It can be seen that the feature fusion method shortens the intra-class distance, expands the inter-class distance, and has better separability.

[0228] Figure 7 (a) Figure 7 (b) The heat map comparison results using the Euclidean distance and the improved distance formula (formula (1) of the present invention) are shown in Figure 1. The new distance formula can better distinguish the categories of the two image tiles.

[0229] Specifically, it has the following beneficial effects:

[0230] First, the present invention is highly efficient. By extracting features from image tiles, the consumption of subsequent computing resources is significantly reduced, the time for classification prediction is reduced, and the classification efficiency is improved, which helps to timely detect diseases and formulate treatment plans, meeting the needs of fast and accurate classification.

[0231] Second, the present invention is adaptive. An improved clustering method is used to classify image tiles, which solves the problem of image rotation transformation. The method is applicable to pathological images of different staining styles, different scales, and different parts, and is not limited to the size of pathological image tiles.

[0232] Third, the present invention has accuracy. The feature extraction model pre-trained by a large number of pathological images is used to extract features from image blocks, and the image information is converted into feature vectors. The features retain the tissue detail information and spatial information of the pathological image, ensuring the accuracy and completeness of the classification.

[0233] Fourth, the present invention is practical. The present invention is applicable to large-scale pathological image data processing, can provide faster and more reliable support for related research, has important practical significance, and helps improve the efficiency of medical diagnosis and disease research.

[0234] Fifth, the present invention is intuitive. By visualizing the results of rapid classification on a thumbnail scale, doctors or researchers can more intuitively observe the classification boundaries and regions, reduce analysis time and effort, improve work efficiency, and facilitate them to quickly understand the classification results.

[0235] Sixth, the present invention reduces the cost of manual annotation and improves the robustness of the model.

[0236] In addition, an embodiment of the present invention further discloses a pathological image tissue classification device based on adaptive clustering, comprising:

[0237] A pathological image acquisition module, used for acquiring pathological images to be classified;

[0238] A zoom module, used to process the pathological image according to a preset zoom ratio to obtain a thumbnail;

[0239] A segmentation module is used to segment the thumbnail image to obtain a tissue mask image;

[0240] A division module is used to divide the tissue mask image into a number of image blocks and record the coordinate position of each image block;

[0241] A feature extraction module, used to extract image semantic features of each image block using a pre-trained feature extraction model;

[0242] Utilize statistical methods to extract statistical features of each image block;

[0243] Fusion of image semantic features and statistical features;

[0244] The clustering module is used to perform clustering based on the fused features and divide all image blocks into k clusters;

[0245] The tissue category prediction module is used to predict the tissue category of each cluster based on the clustering results;

[0246] Optimization module, used to optimize and correct the prediction results;

[0247] The visualization display module is used to splice image blocks of different tissue categories in different colors according to the coordinate positions of the image blocks to form new thumbnails for visualization display.

[0248] Furthermore, in the feature extraction module, the R value of each image block or the histogram of the grayscale image is counted as the statistical feature of the image block.

[0249] Furthermore, the clustering module includes:

[0250] A sample point forming unit is used to fuse the image semantic features and statistical features of each image block to form a new image block feature;

[0251] Initialization unit, used to randomly select k image block features as initial clustering centers;

[0252] The allocation unit is used to calculate the distance from each image block feature to each cluster center through formula (1), and allocate each image block feature to the cluster where the nearest cluster center is located, so as to obtain k clusters {s1, s2, ..., s k};

[0253]

[0254] Where: x i is the feature of the i-th image block, 1≤i≤n;

[0255] c j is the j-th cluster center, 1≤j≤k;

[0256] xv i is the image semantic feature of the i-th image block feature;

[0257] cv j is the image feature of the jth cluster center;

[0258] xh i is the histogram feature of the i-th image block feature;

[0259] ch j is the histogram feature of the jth cluster center;

[0260] α and β are modulation factors;

[0261] The adjustment unit is used to recalculate the equation (2) for each cluster and select a new cluster center;

[0262]

[0263] Among them, c l is the lth cluster center, 1≤l≤k;

[0264] |s l | represents the number of image block features in the lth cluster;

[0265] x i represents the feature of the i-th image block in the l-th cluster, 1≤i≤|s l |;

[0266] The repeating execution unit is used to repeatedly execute the methods in the allocation unit and the adjustment unit until a stop condition is met.

[0267] Furthermore, the tissue category prediction module includes:

[0268] A database construction unit, used to pre-construct a pathological image block database, wherein the pathological image block database stores a number of image blocks of known tissue categories;

[0269] A size matching unit, used for matching the image block to be matched with the image block in the pathological image block database;

[0270] If they match, continue to execute the method in the feature comparison unit;

[0271] If there is no match, the size of the image block in the pathological image block database is cut to be consistent with the size of the image block to be matched, and then the method in the feature comparison unit is continued;

[0272] A feature comparison unit, used for comparing the feature mean of any cluster image block to be matched with the feature mean of each known tissue category image block in the pathology image block database;

[0273] According to the comparison results, the tissue category with the smallest feature mean difference is assigned to the cluster as its label.

[0274] Further, the optimization module includes:

[0275] A category label map forming unit, used to form a category label map according to the coordinate positions of the image blocks and the organizational categories of the image blocks;

[0276] A traversal unit is used to traverse the tissue category of each image block in the category label map according to the traversal direction. If the tissue category of the image block with coordinates (x, y) is different from the tissue categories of other image blocks in its connected domain, the category count matrix Count(x, y, label) is used to record it, and label is the tissue category label that appears most frequently in other image blocks in its connected domain.

[0277] A repeated execution unit is used to switch to other traversal directions and repeatedly execute the method in the traversal unit until all traversal directions have completed traversal;

[0278] The correction unit is used to correct the tissue category of each image block according to the category count matrix Count(x, y, label).

[0279] In this embodiment, the specific content of the pathological image tissue classification device based on adaptive clustering is similar to the content of the pathological image tissue classification method based on adaptive clustering disclosed in the above embodiment, and will not be repeated here.

[0280] The present invention discloses a method and device for classifying tissue in pathological images based on adaptive clustering, which can adaptively process classification tasks of different pathological images, significantly reduce prediction time, improve classification efficiency, help to timely discover diseases and formulate treatment plans, and also have important practical significance for large-scale pathological image data processing.

[0281] It should be understood that the various techniques described herein can be implemented in combination with hardware or software, or a combination thereof. Thus, the method and apparatus of the present invention, or some aspects or portions of the method and apparatus of the present invention, can take the form of program codes (i.e., instructions) embedded in a tangible medium, such as a floppy disk, a CD-ROM, a hard disk drive, or any other machine-readable storage medium, wherein when the program is loaded into a machine such as a computer and executed by the machine, the machine becomes a device for practicing the present invention.

[0282] The above shows and describes the basic principles and main features of the present invention and the advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for illustrating the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which shall fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A method for classifying tissue in pathological images based on adaptive clustering, characterized in that: include: Step S1: obtaining a pathological image to be classified; Step S2: Processing the pathological image according to a preset zoom ratio to obtain a thumbnail; Step S3: segmenting the thumbnail to obtain a tissue mask image; Step S4: dividing the tissue mask image into a number of image blocks, and recording the coordinate position of each image block; Step S5: extracting image semantic features of each image block using a pre-trained feature extraction model; Utilize statistical methods to extract statistical features of each image block; Fusion of image semantic features and statistical features; Step S6: clustering based on the fused features, dividing all image blocks into k clusters; Step S7: predicting the tissue category of each cluster according to the clustering results; Step S8: Optimizing and correcting the prediction results; Step S9: according to the coordinate positions of the image blocks, the image blocks of different tissue categories are spliced ​​in different colors to form new thumbnails for visual display.

2. The method for classifying tissue in pathological images according to claim 1, characterized in that: In step S5, the R value of each image block or the histogram of the grayscale image is counted as the statistical feature of the image block.

3. The method for classifying tissue in pathological images according to claim 1, characterized in that: The step S6 comprises: Step S6.1: fusing the image semantic features and statistical features of each image block to form a new image block feature; Step S6.2: randomly select k image block features as initial clustering centers; Step S6.3: Calculate the distance from each image block feature to each cluster center using equation (1), and assign each image block feature to the cluster where the nearest cluster center is located, and obtain k clusters {s1, s2, …, s k }; Where: x i is the feature of the i-th image block, 1≤i≤n; c j is the jth cluster center, 1≤j≤k; xv i is the image semantic feature of the i-th image block feature; cv j is the image feature of the jth cluster center; xh i is the histogram feature of the i-th image block feature; ch j is the histogram feature of the jth cluster center; α and β are modulation factors; Step S6.4: For each cluster, recalculate formula (2) and select a new cluster center; Among them, c l is the lth cluster center, 1≤l≤k; |s l | represents the number of image block features in the lth cluster; x i represents the feature of the i-th image block in the l-th cluster, 1≤i≤|s l |; Step S6.5: Repeat steps S6.3 to S6.4 until the stop condition is met.

4. The method for classifying tissue in pathological images according to claim 3, characterized in that: The step S7 comprises: Step S7.1: pre-constructing a pathological image block database, wherein the pathological image block database stores a number of image blocks of known tissue categories; Step S7.2: matching the size of the image block to be matched with the image block in the pathological image block database; If they match, continue to step S7.3; If there is no match, the size of the image block in the pathological image block database is cropped to be consistent with the size of the image block to be matched, and then step S7.3 is continued; Step S7.3: comparing the feature mean of any cluster image block to be matched with the feature mean of each known tissue category image block in the pathology image block database; According to the comparison results, the tissue category with the smallest feature mean difference is assigned to the cluster as its label.

5. The method for classifying tissue in pathological images according to claim 1, characterized in that: The step S8 comprises: Step S8.1: forming a category label map according to the coordinate positions of the image blocks and the organizational categories of the image blocks; Step S8.2: traverse the tissue category of each image block in the category label map according to the traversal direction. If the tissue category of the image block with coordinates (x, y) is different from the tissue categories of other image blocks in its connected domain, record it using the category count matrix Count(x, y, label), where label is the tissue category label that appears most frequently in other image blocks in its connected domain. Step S8.3: switch to other traversal directions, and repeat step S8.2 until all traversal directions have been traversed; Step S8.4: Correct the tissue category of each image block according to the category count matrix Count(x, y, label).

6. A device for classifying tissue in pathological images based on adaptive clustering, characterized in that: include: A pathological image acquisition module, used for acquiring pathological images to be classified; A zoom module, used to process the pathological image according to a preset zoom ratio to obtain a thumbnail; A segmentation module is used to segment the thumbnail image to obtain a tissue mask image; A division module is used to divide the tissue mask image into a number of image blocks and record the coordinate position of each image block; A feature extraction module, used to extract image semantic features of each image block using a pre-trained feature extraction model; Utilize statistical methods to extract statistical features of each image block; Fusion of image semantic features and statistical features; The clustering module is used to perform clustering based on the fused features and divide all image blocks into k clusters; The tissue category prediction module is used to predict the tissue category of each cluster based on the clustering results; Optimization module, used to optimize and correct the prediction results; The visualization display module is used to splice image blocks of different tissue categories in different colors according to the coordinate positions of the image blocks to form new thumbnails for visualization display.

7. The device for classifying tissue in pathological images according to claim 6, characterized in that: In the feature extraction module, the R value of each image block or the histogram of the grayscale image is counted as the statistical feature of the image block.

8. The device for classifying tissue in pathological images according to claim 6, characterized in that: The clustering module comprises: A sample point forming unit is used to fuse the image semantic features and statistical features of each image block to form a new image block feature; Initialization unit, used to randomly select k image block features as initial clustering centers; The allocation unit is used to calculate the distance from each image block feature to each cluster center through formula (1), and allocate each image block feature to the cluster where the nearest cluster center is located, so as to obtain k clusters {s1, s2, ..., s k }; Where: x i is the feature of the i-th image block, 1≤i≤n; c j is the jth cluster center, 1≤j≤k; xv i is the image semantic feature of the i-th image block feature; cv j is the image feature of the jth cluster center; xh i is the histogram feature of the i-th image block feature; ch j is the histogram feature of the jth cluster center; α and β are modulation factors; The adjustment unit is used to recalculate the equation (2) for each cluster and select a new cluster center; Among them, c l is the lth cluster center, 1≤l≤k; |s l | represents the number of image block features in the lth cluster; x i represents the feature of the i-th image block in the l-th cluster, 1≤i≤|s l |; The repeating execution unit is used to repeatedly execute the methods in the allocation unit and the adjustment unit until a stop condition is met.

9. The device for classifying tissue in pathological images according to claim 8, characterized in that: The tissue category prediction module includes: A database construction unit, used to pre-construct a pathological image block database, wherein the pathological image block database stores a number of image blocks of known tissue categories; A size matching unit, used for matching the image block to be matched with the image block in the pathological image block database; If they match, continue to execute the method in the feature comparison unit; If there is no match, the size of the image block in the pathological image block database is cut to be consistent with the size of the image block to be matched, and then the method in the feature comparison unit is continued; A feature comparison unit, used for comparing the feature mean of any cluster image block to be matched with the feature mean of each known tissue category image block in the pathology image block database; According to the comparison results, the tissue category with the smallest feature mean difference is assigned to the cluster as its label.

10. The device for classifying tissue in pathological images according to claim 6, characterized in that: The optimization module includes: A category label map forming unit, used to form a category label map according to the coordinate positions of the image blocks and the organizational categories of the image blocks; A traversal unit is used to traverse the tissue category of each image block in the category label map according to the traversal direction. If the tissue category of the image block with coordinates (x, y) is different from the tissue categories of other image blocks in its connected domain, the category count matrix Count(x, y, label) is used to record it, and label is the tissue category label that appears most frequently in other image blocks in its connected domain. A repeated execution unit is used to switch to other traversal directions and repeatedly execute the method in the traversal unit until all traversal directions have completed traversal; The correction unit is used to correct the tissue category of each image block according to the category count matrix Count(x, y, label).

Images