A method for classifying cell nuclei in full-field digital slice histopathology images

Abstract

The present invention relates to a method for classifying cell nuclei in full-view digital slice histopathology images, including: preprocessing the histopathology WSI of the patient, stratifying virtual categories according to the empirical density from the nucleus centroid to the pixel in the processed WSI, and obtaining a training set and a validation set; taking the spatial region as the target, improving the classification F1 level of the image with class imbalance, and extracting the nuclear features; after the nuclear features are extracted, using three independent densely connected decoders in Hovernet to respectively perform nuclear detection, horizontal and vertical distance estimation from the pixel to the nuclear centroid, and classification of nuclear prediction; using a cost function with a combination of multiple objective functions to minimize the objective function, and obtaining the final cell nucleus classification result. Compared with the prior art, the present invention has the advantages of significantly improving MSE, helping the model to better learn the class, and improving the accuracy of the classification results.

Description

Technical Field

[0001] The invention relates to the technical field of computer-aided medical image processing, and in particular to a method for classifying cell nuclei in full-view digital slice tissue pathology pictures. Background Art

[0002] With the rapid development of digital pathology, methods for classifying cell nuclei by evaluating the microtissue environment have been greatly improved. Due to the heterogeneity within tumors, qualitative techniques are often difficult and lead to false negative diagnoses, while deep learning methods have provided competitive and accurate classification results for many types of cell nuclei in different tissue regions. Depending on the tissue region, cell nuclei often form clusters; in cell class imbalanced datasets, this poses a challenge to deep learning models, so models are highly needed to incorporate these spatial features.

[0003] Nucleus segmentation is a key step in cell classification models because the performance of segmentation directly affects the classification performance. Images with dense cell regions often result in clustering and overlapping nuclei, which makes segmentation difficult when the clustered nuclei have similar morphological features. The watershed model has been a common approach for segmentation models, but when data augmentation is used, local minima in the image lead to over-segmentation. Models that focus on morphology and spatial awareness can better segment overlapping nuclei. Clustered nuclei show intensity contrast, so when the cost function includes the gradient of the nuclei, the segmentation results will be improved. In recent years, spatially aware biomedical image methods have made progress in the discrimination of malignant and normal nuclei with differences in spatial structural properties. However, many models focus on dense nuclei areas and are not effective for pathological images with very unbalanced nuclei distribution. Summary of the invention

[0004] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a method for classifying cell nuclei in full-field digital slice tissue pathology images.

[0005] The purpose of the present invention can be achieved by the following technical solutions:

[0006] A method for classifying cell nuclei in a full-field digital slice tissue pathology image, the method comprising:

[0007] Preprocess the patient's histopathological WSI, stratify virtual categories according to the empirical density of the nuclear centroid of the cell nucleus to the pixel in the processed WSI, and obtain the training set and the validation set;

[0008] Taking the spatial region as the target, the classification F1 level of the image with class imbalance is improved and the kernel features are extracted;

[0009] After kernel feature extraction, three independent densely connected decoders in Hovernet are used to perform kernel detection, estimation of horizontal and vertical distances from pixels to kernel centroid, and classification of kernel predictions, respectively;

[0010] The cost function with multiple objective function combinations is used to minimize the objective function to obtain the final cell nucleus classification result.

[0011] Furthermore, the preprocessing includes sequentially performing a cropping process, a color standardization process, and a data enhancement process on the patient's histopathological WSI.

[0012] The cropping process is to crop the patient's tissue pathology WSI into overlapping blocks of size 270×270, and then extract information from the cropped overlapping blocks and perform down-sampling.

[0013] The color normalization process is to use Macenko's color normalization algorithm to normalize the cropped color image block.

[0014] The specific content of the data enhancement processing is:

[0015] The normalized image patches are augmented by rotating, mirroring, scaling, translating, Gaussian blurring, median blurring, blending, cutting, and adjusting brightness, contrast, and saturation.

[0016] Furthermore, in the process of stratifying virtual categories, k-fold cross validation was used to divide 80 / 20% of the samples into training and validation sets.

[0017] The expression for stratifying the virtual categories is as follows:

[0018] v0: r≤0.01,

[0019] v1: 0.01<r≤0.2,

[0020] v2: 0.2<r≤0.5,

[0021] v3: r>0.5.

[0022] Where r is the ratio of the number of cell nucleus categories to the number of pixels, and v0, v1, v2, and v3 are different layers of virtual categories.

[0023] Furthermore, a ResNet-50-SE encoder with squeeze and excitation blocks is used to extract nuclear features, and spatial pyramid pooling is adopted to extract more features from the low-balanced classes outside the dense clusters of cell nuclei. At the same time, atrous convolution is used to implement spatial pyramid pooling, and the bias term is initialized only in the last layer.

[0024] Furthermore, the objective function is to minimize multiple cost functions, which are expressed as:

[0025]

[0026] In the formula, is the cost function, j=1…4, representing the layer; i=1…N, representing each pixel, and N is the number of pixels.

[0027] Furthermore, among the multiple cost functions minimized, the first cost function It is an extension of the Huber function, and its expression is:

[0028]

[0029] In the formula, y i is the distance from the pixel to the center, is its fitting value, α is the parameter for controlling the criterion of outliers, δ is the uniformity parameter for controlling penalties, and η j is a continuous function;

[0030] The expression of the second part of the objective function is:

[0031]

[0032] In the formula, is the longitudinal distance, is the lateral distance;

[0033] The third and fourth parts of the objective function are Tversky and Focal functions for class imbalance, respectively:

[0034]

[0035] In the formula, y ik for The actual type of nucleus in the cell, for The predicted category of the cell nucleus, β is a weight, and γ is a parameter that controls the shape of the curve;

[0036] as well as:

[0037]

[0038] In the formula, y i (I) The actual type of nucleus in the cell, for and the predicted category of the cell nucleus, ∈ is a stability parameter, and κ is a parameter for controlling false positives.

[0039] The method for classifying cell nuclei in full-field digital slice tissue pathology images provided by the present invention has at least the following beneficial effects compared with the prior art:

[0040] 1) The present invention detects outliers and improves the classification of class-imbalanced datasets by using a composite convex cost function, controls the smoothness by introducing a new parameter, and improves the classification score based on stratified k-fold cross-validation of the image space empirical density. Compared with the prior art, the cost function significantly improves the MSE of the F1 score.

[0041] 2) The present invention stratifies virtual variables based on the empirical density from the cell nucleus centroid to the pixel to cope with class imbalance. By dividing 80 / 20% of the samples into training set and validation set within the layer, the present invention can ensure that high-density and low-density cell nucleus regions will be similarly distributed in the two data sets so that low-frequency cell nuclei can be located, which helps the model to better learn the class and thus improve the accuracy of the classification results. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 Schematic diagram of the process of the method for classifying cell nuclei in full-field digital slice tissue pathology images in the embodiment.

[0043] Figure 2 A schematic diagram of the principle of layering virtual categories in an embodiment. DETAILED DESCRIPTION

[0044] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work should belong to the scope of protection of the present invention.

[0045] Example

[0046] The present invention relates to a method for classifying cell nuclei in full-field digital slice tissue pathology images, and SpNet will be used to represent the method of the present invention hereinafter. The main improvement principles of the method are: (1) First, by improving the structure of the cost function, using the improved Huber cost function, and using additional parameters to achieve smoothing. (2) According to the empirical density of the image block, the data set is layered, and the training set and the validation set are divided into layers, so that there are enough class imbalance data points in the validation set.

[0047] Based on the above improved principle, the specific implementation steps of the method of the present invention include:

[0048] 1. Data Preparation

[0049] Since the size of WSI (Whole-slide image, pathology image) is gigapixel, the GPU will not be able to process these images if they are directly inserted into the model. Therefore, WSI is cut into 270x270 blocks for training. In order to eliminate any inconsistencies and variations in color staining, the staining is normalized by the Macenko algorithm (Macenko staining normalization method).

[0050] 2. Image Preprocessing

[0051] Data augmentation: Augment images with different colors, textures, and patterns. Based on an image block of size 270×270, the present invention introduces a set of auxiliary variables Z1,...,Z r to represent various transformations. We used rotation (90°, 180°, 270°), mirroring, scaling, translation, Gaussian blur, median blur, blending images, cutting images, adjusting brightness, contrast, and saturation. We also cut a fixed rectangle of one image and replaced it in another, and generated impulse noise caused by sharp and sudden disturbances in the image signal.

[0052] Stratified training validation set: To separate the training and validation image patches, we first stratify them according to the cell nucleus density of the image patches, and implement stratified k-fold cross-validation to maintain a balanced dataset. In this example, k=5 is selected to create an 80 / 20% split. Since the image ratio of the validation set is also low, this method also allows us to select image patches with high density cells, so that no patches without cells are used, and the validation can be optimized. The following virtual categories are stratified according to the empirical density of each image patch:

[0053] v0: r≤0.01,

[0054] v1: 0.01<r≤0.2,

[0055] v2: 0.2<r≤0.5,

[0056] v3: r>0.5.

[0057] Where r is the ratio of the number of cell nucleus categories to the number of pixels. v0, v1, v2, and v3 represent different layers of virtual categories. Figure 2 shown.

[0058] 3. Model Structure

[0059] The present invention uses a ResNet-50-SE encoder to extract nuclear features, and then uses three independent densely connected decoders to extract more features from the low-balanced class outside the dense cluster of cell nuclei through nearest neighbor upsampling using spatial pyramid pooling. Spatial pyramid pooling is implemented using dilated convolution. In order to further solve the class imbalance problem, the bias term is initialized only in the last layer as:

[0060]

[0061] The constant k is set to 5e^(-4). Assume that the training and test images follow the same distribution Then use the Sigmoid function to make category predictions.

[0062] The process of the method of the present invention is summarized as follows Figure 1 shown.

[0063] For the model architecture, consider that the objective function is to minimize a multinomial cost function, which is expressed as:

[0064]

[0065] In the formula, is the cost function, j = 1…4, representing the layer; i = 1…N, representing each pixel, N is the number of pixels; the first cost function is an extension of the Huber function:

[0066]

[0067] Among them, y i is the distance from the pixel to the center, is its fitting value, α is the parameter for controlling the criterion of outliers, δ is the uniformity parameter for controlling penalties, and η j is a continuous function.

[0068] The present invention achieves more accurate classification of data points in class imbalanced data sets by improving the Huber function. Existing models do not pay attention to class imbalance, so the classification accuracy is low. J1 consists of two independent parts to punish outliers at different rates accordingly. Since nuclei tend to cluster together, the penalty of MSE (mean square error) is used within the nucleus radius α. Pixels outside this radius are often outliers, and the penalty of the approximate MAE (mean absolute error) function is used. The higher the α value (α>1.5), the fewer outliers, and the smaller the value (α<1.5), the more outliers. The penalty parameter (the smoothing parameter that controls the penalty) δ determines the degree of penalty for outliers, where the higher the 6 value, the more penalty for outliers. Approximate this function to allow faster convergence and control smoothness, rather than having a constant gradient. The smoothing parameter δ that controls the penalty is used to further control the steepness of the curve, where δ=1 is used for experiments. The function is plotted in two dimensions for different parameter values ​​to show the convergence to |x|.

[0069] The second part of the objective function is introduced from the HoVerNet related paper as:

[0070]

[0071] in, Indicates the vertical distance, Indicates the horizontal distance.

[0072] The above-mentioned paper is specifically "Graham S, Vu QD, Raza SEA, Azam A, Tsang YW, Kwak JT, Rajpoot N. Hover-Net: Simultaneous segmentation and classification of nuclei in multi-tissue histology images. Med Image Anal. 2019 Dec; 58: 101563. doi: 10.1016 / j.media.2019.101563. Epub 2019 Sep 18. PMID: 31561183".

[0073] Segmentation is performed by using the phenomenon that connected cell nuclei have contrasting gradients. The third and fourth parts are Tversky and Focal functions for class imbalance, respectively:

[0074]

[0075] Among them, y ik is the actual class of cell nuclei, is the predicted category of the cell nucleus, β is a weight, and γ is a parameter that controls the shape of the curve. The larger γ is, the greater the attention paid to rare category cells.

[0076] as well as:

[0077]

[0078] where y i (I)Yes The actual type of nucleus in the cell, yes The predicted category of the cell nucleus in , ∈ is a stable parameter, and κ is a parameter to control false positives.

[0079] 4. Example Verification

[0080] 1) Dataset

[0081] The Colorectal Nuclear SEgmentation and Phenotypes (CoNSeP) dataset is used, which consists of 41 colorectal WSIs with a size of 1000x1000 pixels and a magnification of 40x. Each image belongs to a single patient and is extracted from 16 colorectal adenocarcinoma WSIs. The number of different categories of cells in the CoNSeP dataset is shown in Table 1. Although this dataset has 24,319 exhaustively annotated cell nuclei from pathologists, there is a high degree of class imbalance, which greatly affects the model performance. We found that miscellaneous nuclei accounted for 2.38% of the training dataset and 6.39% of the test dataset. Therefore, due to the imbalance of histopathological images, randomly assigning training and validation image sets will not represent the entire WSI and give biased results. This embodiment divides the training set and validation set based on the data stratified by nuclear density.

[0082] Table 1 Number of cells of different categories in the CoNSeP dataset

[0083]

[0084] 2) Data preparation

[0085] To eliminate any inconsistencies and variations in color tinting, the tinting is first normalized. Since the WSIs are gigapixel in size, the GPU will not be able to process these images if they are directly plugged into the model. In addition, each WSI is of a different size, so this example uses 270x270 image blocks.

[0086] Data augmentation enables training on images that cannot be collected, such as images of different colors, textures, and patterns, which are generated by internal variations, so WSI may show different patterns in different medical settings. Using a patch size of 270×270, this example introduces a set of auxiliary variables Z1,...,Z r Represents various transformations. Typical examples include rotation (90°, 180°, 270°), mirroring, scaling, translation, Gaussian blur, median blur, blending images, cutting images, and adjusting brightness, contrast, and saturation. Recent enhancement techniques have also shown promising results, including CutMix, which cuts a fixed rectangle of one image and replaces it in another. This example implements examples of other enhancement methods, including CoarseDropOut, which creates random black rectangles in the image and may help learn outliers. SaltPepperNoise generates impulse noise caused by sharp and sudden interference in the image signal, which helps generate more artifacts that may sometimes be present.

[0087] To separate the training and validation image patches, the present embodiment implements stratified k-fold cross validation to maintain a balanced dataset, preferably choosing k=5 to create an 80 / 20% split. The present embodiment stratifies based on the kernel density of each image. Since the frequency of miscellaneous kernels is so low, the present embodiment decided not to stratify based on the class because the present embodiment wants to retain the high frequency of this class in the training set. The miscellaneous kernel class is generally not present in the clusters in both datasets, so this method will allow the present embodiment to have this class in the validation set, which random sampling cannot guarantee. Since the proportion of images in the validation set is also low, this method also allows the present embodiment to select image patches with a high density of cells, thereby not using any patches without cells and optimizing validation. The following virtual categories are stratified based on the empirical density of each image patch.

[0088] v0: r≤0.01,

[0089] v1: 0.01<r≤0.2,

[0090] v2: 0.2<r≤0.5,

[0091] v3: r>0.5.

[0092] Among them, r is the ratio of the number of cell nucleus categories to the number of pixels. v0, v1, v2, and v3 represent different layers of virtual categories.

[0093] 3) Results

[0094] This embodiment analyzes the CoNSeP dataset with and without stratification (random sampling). The results show that the stratified method of this embodiment outperforms random selection in classification by improving the classification scores of all kernel types. This embodiment also compares the results of using the present invention with the latest state-of-the-art simultaneous segmentation and classification model HoVer-Net model, and the results are shown in Table 2. It can be seen in the table that the overall F1-score of the stratified method of the present invention is as high as 0.7575 compared with other methods, and in each category (class total, it also exceeds the random method and the method proposed by Hover-Net.

[0095] Table 2 Comparison of the results of using stratified partitioning and random partitioning for training and testing sets on the CoNSep dataset

[0096]

[0097] This example performs an ablation analysis on the new loss function and tests its different types, including MSE, MAE, Huber, and Pseudo Huber, which enables us to test the effectiveness. The results show that the cost function of the method of the present invention directly improves the classification F1 score of the class imbalance kernel (miscellaneous type) and other types of kernels. The results are shown in Table 3. The table shows that the PQ measurement of each method is similar, but SpNet's overall recognition F1-score exceeds other methods, and the recognition F1-score of the mixed class, inflammation class, and spindle class also exceeds other methods, indicating that the accuracy of the method of the present invention is higher.

[0098] Table 3 Accuracy comparison of different cost functions

[0099]

[0100] This embodiment also analyzes different encoders, as shown in Table 4, and finds that the spatial encoder (ResNet50-SE) F1-score is higher than other non-spatial encoders, which means that spatial attention is crucial to the task of the method of the present invention. In addition, from Table 4 we can see that ResNet101 has the best segmentation effect, but is poor in classification. It can be clearly seen that the omission of the full set of pre-trained weights of DenseNet has a great impact on the results and cannot effectively produce high-quality results. Although ResNet50-SE is not the fastest model to train, it is faster than the Inception and ResNet-101 models. It also produces excellent results for each core and improves the classification of class-imbalanced cores by 11%. This embodiment has demonstrated improvements to the original HoVer Net paper, and the DenseNet and ResNet-34 models are the only models that perform faster.

[0101] Table 4 Encoder comparison

[0102]

[0103] In summary, the present invention detects outliers and improves classification of class-imbalanced datasets by using composite and convex cost functions. The present invention focuses on spatial regions using squeeze and excitation blocks and spatial pyramid pooling to improve overall detection by more than 3% and improve classification of class-imbalanced kernels by 21.8%. The results show that although ResNet-101 sometimes has higher segmentation metrics, ResNet50-SE has better classification metrics than any other encoder on both datasets. Since ResNet50-SE is the best model, this illustrates the importance of spatially aware models in histology datasets, because many datasets have class imbalance problems, and standardized models will severely penalize outliers during training, which then affects downstream analysis of real-world test datasets that have a large number of artifacts and class-imbalanced kernels. Overall, this shows that by combining stain normalization, stratified sampling, Nadam optimization, SPP, and data augmentation, this helps to significantly improve the present model, even if the present invention has the same encoder as the original HoVer Net model. By stratifying the dataset according to the empirical density of each image patch to create class-balanced training and validation datasets, the accuracy of miscellaneous nuclei in the CoNSeP dataset was improved by 21.8%. Therefore, the proposed model can accurately segment and classify various types of cell nuclear morphology and structure, and can be successfully adapted to the clinical setting to help cancer diagnosis.

[0104] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person familiar with the technical field can easily think of various equivalent modifications or substitutions within the technical scope disclosed by the present invention, and these modifications or substitutions should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A method for classifying cell nuclei in full-field digital slice histopathological images, characterized in that: include: Preprocess the patient's histopathological WSI, stratify virtual categories according to the empirical density of the nuclear centroid of the cell nucleus to the pixel in the processed WSI, and obtain the training set and the validation set; Taking the spatial region as the target, the classification F1 level of the image with class imbalance is improved and the kernel features are extracted; After kernel feature extraction, three independent densely connected decoders in Hovernet are used to perform kernel detection, estimation of horizontal and vertical distances from pixels to kernel centroid, and classification of kernel predictions, respectively; The cost function with multiple objective function combinations is used to minimize the objective function to obtain the final cell nucleus classification result.

2. The method for classifying cell nuclei in full-view digital slice histopathological images according to claim 1, characterized in that: The preprocessing includes sequentially performing a cropping process, a color standardization process, and a data enhancement process on the patient's histopathological WSI.

3. The method for classifying cell nuclei in full-view digital slice histopathological images according to claim 2, characterized in that: The cropping process is to crop the patient's tissue pathology WSI into overlapping blocks of size 270×270, and then extract information from the cropped overlapping blocks and perform down-sampling.

4. The method for classifying cell nuclei in full-view digital slice histopathological images according to claim 3, characterized in that: The color normalization process is to use Macenko's color normalization algorithm to normalize the cropped color image block.

5. The method for classifying cell nuclei in full-view digital slice histopathological images according to claim 4, characterized in that: The specific content of the data enhancement processing is: The normalized image patches are augmented by rotating, mirroring, scaling, translating, Gaussian blurring, median blurring, blending, cutting, and adjusting brightness, contrast, and saturation.

6. The method for classifying cell nuclei in full-view digital slice histopathological images according to claim 1, characterized in that: In the process of stratifying virtual categories, k-fold cross validation was used to split 80 / 20% of the samples into training and validation sets.

7. The method for classifying cell nuclei in full-view digital slice histopathological images according to claim 6, characterized in that: The expression for stratifying the virtual categories is as follows: v0：r≤0.01， v1：0.01＜r≤0.2， v2：0.2＜r≤0.5， v3：r＞0.

5. Where r is the ratio of the number of cell nucleus categories to the number of pixels, and v0, v1, v2, and v3 are different layers of virtual categories.

8. The method for classifying cell nuclei in full-view digital slice histopathological images according to claim 1, characterized in that: A ResNet-50-SE encoder with squeeze and excitation blocks is used to extract nuclear features, and spatial pyramid pooling is used to extract more features from the low-balanced classes outside the dense clusters of cell nuclei. At the same time, atrous convolution is used to implement spatial pyramid pooling, and the bias term is initialized only in the last layer.

9. The method for classifying cell nuclei in full-view digital slice histopathological images according to claim 8, characterized in that: The objective function is to minimize multiple cost functions, which are expressed as: In the formula, is the cost function, j=1…4, representing the layer; i=1…N, representing each pixel, and N is the number of pixels.

10. The method for classifying cell nuclei in full-view digital slice histopathological images according to claim 9, characterized in that: Minimize the first cost function among multiple cost functions It is an extension of the Huber function, and its expression is: In the formula, y i is the distance from the pixel to the center, is its fitting value, α is the parameter for controlling the criterion of outliers, δ is the uniformity parameter for controlling penalties, and η j is a continuous function; The expression of the second part of the objective function is: In the formula, is the longitudinal distance, is the lateral distance; The third and fourth parts of the objective function are Tversky and Focal functions for class imbalance, respectively: In the formula, y ik for The actual type of nucleus in the cell, for The predicted category of the cell nucleus, β is a weight, and γ is a parameter that controls the shape of the curve; as well as: In the formula, y i (I) The actual type of nucleus in the cell, for The predicted category of the cell nucleus in , ∈ is a stability parameter, and κ is a parameter for controlling false positives.

Images