A method for detecting cell nuclei in histopathological images based on point supervision

Abstract

The present invention provides a method for detecting cell nuclei in histopathological images based on point supervision. By adopting the point supervision learning method, first, on the basis of pixel-level representation, the relationship between point labels and their surrounding regions is inferred to determine which regions cover the entire instance with high quality; then, a deep learning framework is used to construct a dense cell nucleus detection network and the network is trained, and the point annotations and regional pseudo-annotations are used as supervision constraints during training; finally, the trained network is used to complete cell nucleus detection. The present invention can learn a large number of relationships between cell nuclei and their adjacent background regions in histopathological images, enabling the network to obtain good detection performance in dense cell nucleus scenarios and improving the cell nucleus detection effect in scenarios with intra-class heterogeneity and overlapping cell nucleus distributions.

Description

Technical Field

[0001] The present invention belongs to the technical field of image detection, and particularly relates to a method for detecting cell nuclei in histopathological images based on point supervision. Background Art

[0002] Analysis of histopathological images is one of the effective auxiliary diagnostic methods for cancer in the current medical field, which can help pathologists determine the type, stage, treatment decision and prognosis of cancer. Among them, cell nucleus detection is one of the most basic and challenging problems in histopathological image analysis. It can locate cell nucleus instances and provide effective auxiliary diagnosis for cancer diagnosis. In recent years, with the development of deep convolutional neural network (CNN), deep learning-based cell nucleus detection in digital histopathological images has attracted the attention of some researchers, and the detection performance has been greatly improved. Although deep learning models trained with fully supervised learning using finely labeled data have achieved very good performance in histopathological image detection, training such models requires a large amount of time and effort to provide accurate annotations for images. In the real world, medical experts need to spend a lot of time accurately annotating a large number of densely distributed cell nuclei on high-resolution digital images. Under the current massive data, this mode cannot meet the requirements of object detection algorithms. Therefore, to balance the data annotation cost and model performance, weakly supervised object detection (WSOD) has gradually become an important research direction in this field, which can greatly reduce the burden on annotators in practical applications.

[0003] Weak labels for supervised detection frameworks include various types, such as noisy annotations, image-level annotations, point annotations, and scribble annotations. Among them, point labels refer to annotating a single pixel in each instance as labeled data, which can save the annotation cost and provide supervision information for each cell nucleus, facilitating the establishment of a high-performance cell nucleus detection model on histopathological images. In the field of histopathological image analysis, researchers have established some point supervised learning frameworks, which are widely used in various tasks of medical images, such as classification, localization, detection, and 3D segmentation. However, these methods usually share semantic and instance representations in pixel-level cell nucleus segmentation tasks. In scenarios with intra-class heterogeneity and overlapping cell nucleus distributions, it is still a challenge to detect dense cell nuclei using point supervised learning frameworks. Summary of the Invention

[0004] To overcome the deficiencies of the prior art, the present invention provides a method for detecting cell nuclei in histopathological images based on point supervision. By adopting the point supervision learning method, first, on the basis of pixel-level representation, the relationship between the point labels and their surrounding areas is inferred to determine which areas cover the entire instance with high quality; then, a dense cell nucleus detection network is constructed using a deep learning framework and the network is trained, with the point annotations and regional pseudo-annotations as supervision constraints during training; finally, the trained network is used to complete the cell nucleus detection. The present invention can learn a large number of relationships between cell nuclei and their adjacent background areas in histopathological images, enabling the network to obtain good detection performance in dense cell nucleus scenarios and improving the cell nucleus detection effect in scenarios with intra-class heterogeneity and overlapping cell nucleus distributions.

[0005] A method for detecting cell nuclei in histopathological images based on point supervision, characterized by the following steps:

[0006] S1. Given a histopathological training image set, assuming that the training images contain I cell nucleus instances and their manually annotated point labels, each original image in the training image set is cut into slices of size 250×250 using the overlapping cutting method, and then the resolution of each slice is adjusted to 1000×1000 using the nearest neighbor interpolation method. The obtained slices are used as network inputs.

[0007] S2. Using the manually annotated point labels contained in each slice as vertices to form a Delaunay triangulation network.

[0008] S3. Using the Delaunay triangulation network to construct a Voronoi diagram, dividing the slice into several closed spaces.

[0009] S4. Using all the point labels on the slices obtained in step S1 as initial clustering centers, and performing clustering on the pixel points in the slice images using the K-mean method, dividing all pixels into three categories, denoted as foreground class, background class, and uncertain class respectively.

[0010] S5. For the closed spaces obtained in step S3, find the maximum circumscribed rectangle of all the pixel points belonging to the foreground class in the space, and use this rectangle as the rough pseudo-label of the cell nucleus instance corresponding to the space. All the rough pseudo-labels form a rough pseudo-label set of the cell nucleus instances.

[0011] S6. Compare the rough pseudo-labels obtained in step S5 with the manually annotated point labels one by one, remove the redundant pseudo-label rectangles that do not contain manually annotated points, and supplement the manually annotated point rectangles that have not been detected to obtain a refined pseudo-label set.

[0012] S7. Preset M square predefined regions with different aspect ratios and scales for the slices obtained in step S1, calculate the Euclidean distance between each manually labeled point tag and the center point of the predefined region, and select the k predefined regions with the closest distances as the candidate positive regions for the point tag. Use to identify whether the m-th predefined region is a candidate positive region for the i-th cell nucleus. If so, then is 1, otherwise it is 0. Where M≥500, 50≤k≤65, m = 1, 2, …, M, i = 1, 2, …, I;

[0013] S8. Calculate the overlapping area between the pseudo-label of each cell nucleus obtained in step S6 and all the candidate positive regions of this cell nucleus obtained in step S7, and calculate the mean v and standard deviation s of all the overlapping areas. If the overlapping area with the candidate positive region r m is greater than the threshold th, then retain this candidate positive region and keep as 1. Otherwise, remove this region and set to 0. Where the threshold th = v + s, m = 1, 2, …, M, i = 1, 2, …, I;

[0014] S9. Calculate the quality q of each candidate positive region obtained in step S8 according to m , where a is a hyperparameter and a = 1 / 9 is set. If is 1, is the Euclidean distance between the point tag of the cell nucleus i and the center point of the candidate positive region r m , otherwise is 0; |g m | is the diagonal length of the candidate positive region r m ; m = 1, 2, …, M, i = 1, 2, …, I;

[0015] S10. The constructed dense cell nucleus detection network includes a deep feature map generation module and a dense cell nucleus detection module. Among them, the deep feature map generation module uses Resnet 50 and Transformer as two Backbones and FPN as the Neck to generate deep feature maps. The dense cell nucleus detection module contains three parallel branches, namely a classification branch, a localization branch, and a regional quality regression branch. The classification branch uses 4 convolutional layers, with the input being the deep feature map and the output being the classification score, and the number of channels is A*M. A is the number of categories. If the predefined region contains a cell nucleus, then A = 1, otherwise A = 0. The localization branch uses 4 convolutional layers, with the input being the deep feature map and the output being the localization score, and the number of channels is 4*M. The regional quality regression branch uses 4 convolutional layers, with the input being the deep feature map and the output being the regional quality score;

[0016] S11. Input all the slices obtained in step S1 into the dense nucleus detection network for network training. When training, set the total loss of the network as:

[0017]

[0018] where, L pseudo represents the loss supervised by the refined pseudo-label obtained in step S6, and L point represents the loss supervised by the manually labeled point labels;

[0019] The loss L pseudo under the pseudo-label supervision is calculated as follows:

[0020]

[0021] where, L class represents the loss of the classification branch, is the classification score of the region r m predicted by the detection network; y m is the class label of the region r m . When there is a nucleus in the region r m , y m is set to 1, otherwise y m = 0; L locate represents the loss of the localization branch, is the localization score of the region r m predicted by the detection network, and t m is the localization label of the region r m , that is, the regression parameter of the target region box;

[0022] The loss L point under the point label supervision is calculated as follows:

[0023]

[0024] where, L quality represents the loss of the region quality regression branch, is the region quality score predicted by the detection network for the region r m , and q m is the quality of the region r m calculated in step S9;

[0025] S12. Input the histopathological image to be detected into the dense nucleus detection network trained in step S11. The common outputs of the classification and localization branches are the final detection results, which are the class and location of the nucleus respectively.

[0026] Specifically, the specific processing procedure of step S2 is as follows: First, a convex hull is constructed using some point labels, and Delaunay triangulation is performed on the convex hull to obtain an initial triangular mesh; then, the remaining point labels are respectively placed into the triangular mesh one by one, and the triangular mesh is continuously improved by finding local optimal solutions to obtain a Delaunay triangular mesh containing the complete set of point labels.

[0027] The beneficial effects of the present invention are as follows: Since point annotation and pseudo-annotation are introduced as supervision constraints in the framework of point supervision learning, the relationship between point labels and their surrounding areas is better utilized, taking into account both the nucleus position and structural information, and the performance of nucleus detection can be improved in scenarios with intra-class heterogeneity and overlapping nucleus distributions; since a regional quality regression branch is introduced, not only can the quality of the training region be estimated, but it can also be used as the target of regression, so that the nuclei in the overlapping scenario can be effectively detected, and the performance of the dense nucleus detector under point supervision can be improved. Description of the Drawings

[0028] Figure 1 is a flowchart of a method for detecting nuclei in histopathological images based on point supervision of the present invention;

[0029] Figure 2 is a schematic diagram of the generation of candidate regions and quality. Specific Embodiments

[0030] The present invention will be further described below in conjunction with the drawings and embodiments, and the present invention includes but is not limited to the following embodiments.

[0031] The present invention provides a method for detecting nuclei in histopathological images based on point supervision, mainly solving the problem of how to improve the detection performance in scenarios with intra-class heterogeneity and overlapping nucleus distributions in the task of detecting nuclei in histopathological images under point supervision. As Figure 1 shown, its specific implementation process is as follows:

[0032] S1. Given a histopathological training image set, assuming that the training images contain I nucleus instances and their manually annotated point labels, each original image in the training image set is cut into slices of size 250×250 by means of overlapping cutting, and then the resolution of each slice is adjusted to 1000×1000 by the method of nearest neighbor interpolation. The obtained several slices are used as the network input.

[0033] S2. Use the manually labeled point tags included in each slice as vertices to form a Delaunay triangulation network. The specific processing procedure is as follows: First, construct a convex hull using some point tags and perform Delaunay triangulation on the convex hull to obtain an initial triangulation network. Then, sequentially place the remaining point tags into the triangulation network one by one, and continuously improve the triangulation network by finding local optimal solutions to obtain a Delaunay triangulation network containing the complete set of point tags. This process is documented in the literature "Two Algorithms for Constructing a Delaunay Triangulation".

[0034] S3. Use the Delaunay triangulation network to construct a Voronoi diagram, that is, a Thiessen polygon or a Dirichlet diagram. The Voronoi diagram is a set of multiple continuous polygons composed of the perpendicular bisectors of the lines connecting two adjacent point tags. The plane can be divided into I closed spaces by I point tags on the plane. This process is documented in the literature "Two Algorithms for Constructing a Delaunay Triangulation".

[0035] S4. Use all the point tags on the slice obtained in step S1 as the initial clustering centers, and use the K-mean method to perform clustering processing on the pixel points in the slice image. All pixels are divided into three categories, denoted as the foreground category, the background category, and the uncertain category respectively.

[0036] S5. For the closed space obtained in step S3, find the maximum circumscribed rectangle of all the pixel points belonging to the foreground category in this space, and use this rectangle as the rough pseudo-label of the corresponding cell nucleus instance in this space. All the rough pseudo-labels constitute the rough pseudo-label set of the cell nucleus instances.

[0037] S6. Compare the rough pseudo-labels obtained in step S5 with the manually labeled point tags one by one, remove the redundant pseudo-label rectangles that do not contain manually labeled points, and supplement the rectangles where manually labeled points are not detected to obtain a refined pseudo-label set.

[0038] S7. Preset M square predefined regions with different aspect ratios and different scales for the slice obtained in step S1. Calculate the Euclidean distance between each manually labeled point tag and the center point of the predefined region, and select the k predefined regions with the closest distances as the candidate positive regions for this point tag. Use to indicate whether the m-th predefined region is the candidate positive region for the i-th cell nucleus. If so, then is 1, otherwise it is 0; where M≥500, 50≤k≤65, m = 1, 2, …, M, i = 1, 2, …, I.

[0039] S8. Calculate the area of overlap between the pseudo-label of each cell nucleus obtained in step S6 and all candidate positive regions of this cell nucleus obtained in step S7, and calculate the mean v and standard deviation s of all the overlapping areas. If the overlapping area with the candidate positive region r m is greater than the threshold th, then retain this candidate positive region and keep as 1, otherwise, remove this region and set to 0; where the threshold th = v + s, m = 1, 2, …, M, i = 1, 2, …, I.

[0040] S9. Calculate the quality q of each candidate positive region obtained in step S8 according to m , where a is a hyperparameter and a = 1 / 9 is set; if is 1, is the Euclidean distance between the point label of cell nucleus i and the center point of the candidate positive region r m , otherwise is 0; |g m | is the diagonal length of the candidate positive region r m ; m = 1, 2, …, M, i = 1, 2, …, I. Figure 2 Figure shows the schematic diagram of the candidate region and quality generation process.

[0041] S10. The constructed dense cell nucleus detection network includes a deep feature map generation module and a dense cell nucleus detection module. The deep feature map generation module adopts two Backbones, Resnet 50 and Transformer, and FPN as the Neck to generate the deep feature map. Among them, Resnet 50 contains 5 stages, mainly performing calculations of convolution, regularization, activation function, and max pooling on the input. Except for the first stage, all contain residual blocks. The Transformer contains 12 encoding modules, and each encoding module is composed of layer normalization, multi-head self-attention module, multi-layer perceptron, Patch Merge, and residual connection, etc.

[0042] The dense cell nucleus detection module contains three parallel branches, namely the classification branch, the localization branch, and the region quality regression branch. Among them, the classification branch adopts 4 convolutional layers, the input is the deep feature map, the output is the classification score, and the number of channels is A * M. A is the number of categories. If there is a cell nucleus in the predefined region, then A = 1, otherwise A = 0; the localization branch adopts 4 convolutional layers, the input is the deep feature map, the output is the localization score, and the number of channels is 4 * M; the region quality regression branch adopts 4 convolutional layers, the input is the deep feature map, the output is the region quality score, and the number of channels is 1 * M.

[0043] S11. Input all the slices obtained from step S1 into the dense nucleus detection network for network training, and use point labels for supervision. When training, set the total loss of the network as:

[0044]

[0045] Among them, L pseudo represents the loss under the supervision of the refined pseudo-labels obtained from step S6, and L point represents the loss under the supervision of the manually annotated point labels.

[0046] L pseudo is calculated according to the following formula:

[0047]

[0048] Among them, L class represents the loss of the classification branch, is the classification score of the region r m predicted by the detection network; y m is the class label of the region r m . When there is a nucleus in the region r m , y m is set to 1, otherwise y m = 0; L locate represents the loss of the localization branch, is the localization score of the region r m predicted by the detection network, and t m is the localization label of the region r m , which is the regression parameter of the target region box.

[0049] L point is calculated according to the following formula:

[0050]

[0051] Among them, L quality represents the loss of the region quality regression branch, is the region quality score predicted by the detection network for the region r m , and q m is the quality of the region r m calculated in step S9.

[0052] S12. Input the histopathological image to be detected into the dense nucleus detection network trained in step S11. The common output of the classification and localization branches is the final detection result, which includes the class and location of the nucleus.

[0053] To verify the effectiveness of the method of the present invention, experiments were carried out on the MO and Lizard datasets for verification. Among them, MO is a histopathological image dataset stained with hematoxylin and eosin (H&E), which includes 30 patients, benign / lesion tissues of 7 organs, with more than 21,000 manually annotated epithelial and stromal nuclear boundaries. Its training set contains more than 13,000 pathological sections of four organs, namely breast, kidney, liver and prostate, from 16 patients. The test set not only contains the four organs that appear in the training set, but also contains three new organs that do not appear - bladder, colon and stomach. Lizard is a dataset of colon tissue pathological images stained with H&E, including images of various normal, inflammatory, hyperplastic and cancerous conditions, with nearly 500,000 manually annotated nuclear boundaries of various types. In the training set and test set, there are 6 different types of cell nuclei - epithelial cells, connective tissue cells, lymphocytes, plasma cells, neutrophils and eosinophils.

[0054] On the MO and Lizard datasets, the method of the present invention was compared with 10 state-of-the-art multivariate time series classification methods, including: Rloss, Point, Mixed, Partial, MaskGA, WNS, RetinaNet, Swin, Dino, ViT-Adapter. The mean average accuracy (mAP) at different thresholds obtained by different methods was calculated respectively. Among them, avg-mAP[0.5:0.05:0.95] represents the average mAP at different IoU thresholds (from 0.5 to 0.95, step 0.05). mAP@tIoU(%) lists the average mAP at 0.75 and 0.50 IoU thresholds. Table 1 shows the experimental result data of the mAP values calculated by different methods for nucleus detection. It can be seen that the method of the present invention has a relatively high average accuracy value and can obtain better nucleus detection performance.

[0055] Table 1

[0056]

[0057]

[0058] The present invention can overcome the problem that the existing methods have unsatisfactory nucleus detection performance in scenarios with intra-class heterogeneity and overlapping nuclear distributions. The method of Point Supervised Learning is adopted for the detection task, and the relationship between region levels is further inferred based on the pixel-level representation. By inferring the relationship between the point label and its surrounding regions, it is determined which regions cover the entire instance with high quality. In particular, the present invention can learn the relationship between a large number of cell nuclei and their adjacent background regions in histopathological images, enabling the network to obtain good detection performance in dense cell nucleus scenarios.

[0059] Based on the characteristics of cell nucleus data in histopathological images, the present invention introduces point annotation and pseudo-annotation as supervision constraints in the framework of point supervised learning, and explores the relationship between the point label and its surrounding regions from two aspects: on the one hand, an adaptive sampling strategy is adopted to adaptively sample the corresponding positive regions for each nuclear appearance and position, so as to improve the detection performance of the detector in the case of intra-class heterogeneity; on the other hand, a region quality regression branch is introduced, which can not only estimate the quality of the training region, but also use it as the target of regression, and thus can effectively detect cell nuclei in overlapping scenarios and improve the performance of the dense nuclear detector under point supervision.

Claims

1. A method for detecting cell nuclei in histopathological images based on point supervision, characterized in that The steps are as follows: S1. Given a histopathology training image set, assuming that the training images contain I nuclear instance and their manually annotated point labels, each original image in the training image set is cut into slices of size 250×250 by means of overlapping cutting, and then the resolution of each slice is adjusted to 1000×1000 by the method of nearest neighbor interpolation. A number of processed slices are used as network inputs; S2. A Delaunay triangulation is formed with the manually annotated point labels contained in each slice as vertices; S3. A Voronoi diagram is constructed using the Delaunay triangulation, and the slice is divided into several closed spaces; S4. Using all the point labels on the slices obtained in step S1 as the initial clustering centers, the K-mean method is used to cluster the pixel points in the slice images. All pixels are divided into three categories, denoted as foreground class, background class, and uncertain class respectively; S5. For the closed spaces obtained in step S3, the maximum circumscribed rectangle of all pixel points belonging to the foreground class in the space is found, and this rectangle is used as the rough pseudo-label of the nuclear instance corresponding to the space. All the rough pseudo-labels constitute the rough pseudo-label set of the nuclear instance; S6. The rough pseudo-labels obtained in step S5 are compared with the manually annotated point labels one by one. The redundant pseudo-label rectangles that do not contain manually annotated points are removed, and the manually annotated point rectangles that are not detected are supplemented to obtain a refined pseudo-label set; S7. Preset M square predefined regions with different aspect ratios and different scales for the slices obtained in step S1, calculate the Euclidean distance between each manually annotated point label and the center point of the predefined region, and select the k predefined regions with the closest distances as the candidate positive regions for the point label. Use to identify whether the m-th predefined region is a candidate positive region for the i-th cell nucleus. If so, is 1, otherwise it is 0; where M ≥ 500, 50 ≤ k ≤ 65, m = 1, 2, …, M, and i = 1, 2, …, I; S8. Calculate the area of overlap between the pseudo-label of each cell nucleus obtained in step S6 and all candidate positive regions of this cell nucleus obtained in step S7, and calculate the mean value v and standard deviation s of all overlapping areas. If the overlapping area with candidate positive region r m is greater than the threshold th, then retain this candidate positive region and keep as 1. Otherwise, remove this region and set to 0; where the threshold th = v + s, m = 1, 2, …, M, i = 1, 2, …, I; S9. According to the quality q of each candidate positive region obtained in calculation step S8 m , where a is a hyperparameter and a = 1 / 9 is set; if is 1, is the Euclidean distance between the point label of nucleus i and the center point of candidate positive region r m otherwise is 0; |g m | is the diagonal length of candidate positive region r m ; m = 1, 2, …, M, i = 1, 2, …, I; S10. The constructed dense nuclear detection network includes a deep feature map generation module and a dense nuclear detection module. Among them, the deep feature map generation module uses two Backbones of Resnet 50 and Transformer and FPN as the Neck to generate deep feature maps; the dense nuclear detection module contains three parallel branches, namely a classification branch, a localization branch, and a regional quality regression branch. The classification branch uses 4 convolutional layers, the input is the deep feature map, the output is the classification score, and the number of channels is A*M. A is the number of categories. If there is a nucleus in the predefined region, then A = 1, otherwise A = 0; the localization branch uses 4 convolutional layers, the input is the deep feature map, the output is the localization score, and the number of channels is 4*M; the regional quality regression branch uses 4 convolutional layers, the input is the deep feature map, and the output is the regional quality score; S11. All the slices obtained by processing in step S1 are input into the dense nuclear detection network for network training. When training, the total loss of the network is set as: Among them, L pseudo represents the loss under the supervision of the refined pseudo-labels obtained in step S6, and L point represents the loss under the supervision of the manually labeled point labels; The loss L under the supervision of the pseudo-label pseudo is calculated as follows: Among them, L class represents the loss of the classification branch, where r m is the classification score predicted by the detection network; y m is the class label of the region r m , and when there is a cell nucleus in the region r m , y m is set to 1, otherwise y m = 0; L locate represents the loss of the localization branch, where r m is the localization score predicted by the detection network, and t m is the localization label of the region r m , that is, the regression parameter of the target region box; The loss L under the supervision of the point label point is calculated as follows: Among them, L quality represents the loss of the region quality regression branch, is the region quality score predicted by the detection network for region r m , q m is the quality of region r calculated in step S9 m ; S12. The histopathology image to be detected is input into the dense nuclear detection network trained in step S11. The common outputs of the classification and localization branches are the final detection results, which are the category and location of the nucleus respectively.

2. The method for detecting cell nuclei in histopathological images based on point supervision according to claim 1, wherein: The specific processing process of step S2 is as follows: First, a convex hull is constructed using some point labels, and the convex hull is subjected to Delaunay triangulation to obtain an initial triangulation; then, the remaining point labels are respectively put into the triangulation one by one, and the triangulation is continuously improved by finding local optimal solutions to obtain a Delaunay triangulation containing the complete point label set.

Images