Weak supervision cell nucleus detection and segmentation method based on partial point labels

Abstract

The invention discloses a weak supervision cell nucleus detection and segmentation method based on partial point labels. The method comprises the following steps: acquiring a Hamp; e, obtaining a data set from the dyeing image, dividing the data set, generating a part of point tags and then generating a pseudo tag after Gaussian mask expansion, and respectively carrying out first-stage rough cell nucleus detection and second-stage cell nucleus detection self-supervised training to obtain a final cell nucleus detection result; generating an H single-component image as an input image of the segmentation network based on a dyeing separation technology; generating a pseudo tag in a segmentation stage based on the point tag, and carrying out segmentation training through the pseudo tag in the segmentation stage; generating a cell nucleus image edge information false label based on an edge detection strategy; and carrying out cell nucleus segmentation training based on the edge information pseudo tag to obtain a prediction result of the trained segmentation network. According to the method, the Hamp can be realized by only using part of point tags of a weak supervision level; and E, detecting and segmenting the cell nucleus of the dyed image.

Description

Technical Field

[0001] The present invention belongs to the technical field of digital pathology and deep learning, and particularly relates to a weakly supervised nucleus detection and segmentation method based on partial point labels. Background Art

[0002] Histopathological sections contain rich microenvironment information and can play an important role in cancer diagnosis, prognosis, and treatment. In recent years, the development of computational pathology has made the automated analysis of pathological sections a research hotspot. Nucleus segmentation is a key step in pathological image analysis, and its shape, size, density, and quantity are closely related to cancer diagnosis and treatment. For example, the results of nucleus segmentation can be used to evaluate the structural characteristics of tumor cells and serve as an important indicator for the prognosis of cancer patients. To facilitate microscopic observation, different staining methods are usually used to enhance the contrast of each structure in pathological images. The hematoxylin-eosin (H&E) staining method is the most commonly used staining technique for nucleus analysis. Hematoxylin stains the nucleus blue-violet (H component) and the cytoplasm and other components pink (E component). Therefore, automated nucleus segmentation methods have been widely studied and play an important role in the analysis of histopathological sections. Due to the high complexity of cell structures, it is very cumbersome for pathologists to accurately delineate them at the pixel level. Therefore, in clinical practice, automated nucleus segmentation can greatly reduce the amount of manual annotation and improve work efficiency.

[0003] In recent years, significant progress has been made in applying deep learning methods in the field of medical image segmentation. However, the most advanced deep learning methods in the current nucleus segmentation field usually require a large number of annotated nucleus images for training. In addition, as the annotation process continues, the accuracy and speed of manual annotation by pathologists will gradually decrease. In contrast, using weak annotations (such as bounding boxes or point annotations) is more efficient and time-saving in annotation, saving 42% and 88% of the annotation amount respectively. In histopathological images, nuclei are small and densely distributed. Point annotation is more suitable for weakly supervised nucleus segmentation. Therefore, one of the core challenges of this task is how to efficiently utilize point annotation information to improve segmentation performance. Existing studies usually address this problem by integrating the knowledge of the nucleus neighborhood into the segmentation model. For example, using the shape and prior knowledge of the nucleus to generate supervision signals, including Gaussian heatmaps, Voronoi diagrams, and distance maps. Although these methods reduce the annotation burden to some extent, they are still limited by incomplete and inaccurate supervision signals, especially in the processing of nucleus boundaries.

[0004] Some studies have shown that combining incomplete annotations with weakly supervised learning can further reduce the annotation workload. Incomplete annotations refer to providing point labels only for some nuclear units, rather than comprehensively annotating all units. However, achieving weakly supervised segmentation under incomplete annotations still faces great challenges because even the reduced annotation information is still insufficient to accurately represent foreground nuclei. In addition, for highly heterogeneous and complex histopathological images, it is difficult to clearly depict the nuclear boundaries even under full supervision. Summary of the Invention

[0005] The object of the present invention is to provide a weakly supervised nuclear detection and segmentation method based on partial point labels. Using digital pathology and deep learning segmentation algorithms, it can achieve cell segmentation of H&E stained images using partial point labels, generate pixel-level segmentation results, and intuitively display the spatial distribution of the internal tissue structure of tumors.

[0006] To achieve the above object, the present invention provides a weakly supervised nuclear detection and segmentation method based on partial point labels, including:

[0007] Obtain an H&E stained image to get a dataset, divide the dataset, and generate partial point labels based on the divided dataset;

[0008] Generate pseudo-labels after Gaussian mask expansion based on the partial point labels, and perform rough nuclear detection in the first stage based on the pseudo-labels after Gaussian expansion;

[0009] Based on the background image and the original point labels, perform self-supervised training for nuclear detection in the second stage to obtain the final nuclear detection result;

[0010] In the segmentation stage, generate an H single-component image as the input image of the segmentation network based on the staining separation technique;

[0011] Generate pseudo-labels for the segmentation stage based on the point labels, and perform segmentation training through the pseudo-labels of the segmentation stage;

[0012] Generate pseudo-labels for the nuclear image edge information based on the edge detection strategy;

[0013] Perform nuclear segmentation training based on the edge information pseudo-labels, obtain the prediction result of the trained segmentation network, and use the prediction result as the final segmentation result.

[0014] Preferably, the process of obtaining an H&E stained image to get a dataset and dividing the dataset includes:

[0015] Collect the pathological section images of cells, perform staining treatment on the pathological section images to obtain H&E stained pathological sections; then perform digital processing on the H&E stained pathological sections to obtain the H&E staining images.

[0016] Preferably, the process of generating partial point labels based on the divided dataset includes:

[0017] Perform pixel-level annotation on the H&E staining images, and generate point labels based on the pixel-level annotation;

[0018] According to the set magnification, randomly sample the point label images to obtain partial point label images.

[0019] Preferably, the process of generating pseudo-labels after Gaussian mask expansion based on the partial point labels includes:

[0020] Generate pseudo-labels after Gaussian mask expansion by replacing the point annotation labels with Gaussian heatmaps;

[0021] Among them, the formula expression for defining the Gaussian heatmap is:

[0022]

[0023] where H i is the i-th Gaussian expansion mask map, σ is the standard deviation, r 1 and r 2 respectively represent predefined hyperparameters of the centroid point to the foreground radius and background radius of the cell nucleus; D i is the Euclidean distance between the i-th pixel and its nearest point annotation. For pixels with an Euclidean distance greater than r 2 are regarded as ignored regions, denoted as -1, and not calculated during training. Among them, the Euclidean distance D i is expressed as:

[0024]

[0025] where N represents the number of point annotations in the image.

[0026] Preferably, the process of performing rough cell nucleus detection in the first stage based on the pseudo-labels after Gaussian expansion includes:

[0027] Through the per-pixel regression method, train the cell nucleus detection network with the mask after Gaussian expansion to obtain unlabeled points;

[0028] Among them, the per-pixel regression method uses the mean square error as the loss function, and the formula expression is:

[0029]

[0030] Among them, Y det is the predicted heat map, and N is the total number of pixels in the image that are not ignored.

[0031] Preferably, the process of obtaining the final nucleus detection result through self-supervised training for nucleus detection in the second stage based on the background map and the original point labels includes:

[0032] Using the unlabeled area, optimize and iterate the background map in the self-training process through background propagation. In the first round of self-training, use the initially trained probability map. In each round of self-training, first generate a probability map from the model of the previous round;

[0033] Subsequently, combine the probability map with the original labeled points to generate a new training mask; when generating the background map, determine the pixels whose prediction probability meets specific conditions as background pixels; among them, the specific conditions include: if the prediction probability of a pixel is close to 0, it is regarded as background; for pixels with a prediction probability close to 1, further screen and only retain the connected components with an area larger than the average nucleus area;

[0034] Finally, integrate the new background information into the extended mask to generate an updated training mask.

[0035] Preferably, the process of generating the H single-component image as the input image of the segmentation network based on the staining separation technology includes:

[0036] Separate the nuclei from the background pixels according to the principle of H&E staining; among them, the principle of H&E staining includes staining the nuclei blue-violet with hematoxylin and staining the cytoplasm and matrix pink with eosin;

[0037] Then adopt the color deconvolution method to convert the red, green, and blue values of hematoxylin and eosin into the corresponding H&E staining values through a pseudo-inverse matrix, and separate the hematoxylin and eosin components in the stained image;

[0038] Among them, the RGB values of hematoxylin and eosin are [0.644, 0.717, 0.267] and [0.093, 0.954, 0.283] respectively, and the formula expression of the H&E staining value is:

[0039]

[0040] Among them, HE i is the hematoxylin and eosin channel values of pixel i, and RGB i is the red, green, and blue channel values of pixel i, and + represents the pseudo-inverse of the matrix;

[0041] Obtain the hematoxylin component X by keeping the hematoxylin channel h :

[0042] xh = exp(-HE i [0, :])

[0043] where X h is a set of hematoxylin component images extracted from H&E stained images.

[0044] Preferably, generating pseudo-labels for the segmentation stage based on point labels, the process of segmentation training through the pseudo-labels of the segmentation stage includes:

[0045] Adopting the Voronoi diagram and the mean clustering method to generate Voronoi labels and clustering labels respectively;

[0046] For the Voronoi labels, dividing the image into convex polygons through point annotation; assuming that each point annotation is located at the center of the cell nucleus and the cell nucleus shape is convex, delimiting an independent region for each cell nucleus to obtain preliminary nuclear boundary information;

[0047] Using the double mean method to obtain the supervision information of the nuclear boundary and shape of the clustering labels. First, starting from the point annotation, generating a distance map by calculating the distance transform between any two points, and then combining the distance map with the original H&E stained image for mean clustering to divide the pixels into three categories: cell nucleus, background, and ignored region;

[0048] Marking the clustering with the largest overlapping area with the point annotation as the cell nucleus, the clustering with the smallest overlapping area as the background, and the remaining undetermined categories as the ignored region to obtain the initial clustering labels;

[0049] Applying morphological operations including connected component labeling, discrete region removal, morphological operations, and binary hole filling to the initial clustering labels to optimize the clustering results and obtain the final clustering labels.

[0050] Preferably, the process of generating pseudo-labels for the cell nucleus image edge information based on the edge detection strategy includes:

[0051] When optimizing the edge contour through the edge optimization strategy, generating edge optimization labels to assist the segmentation training in the second stage;

[0052] The process of generating edge optimization labels to assist the segmentation training in the second stage includes:

[0053] First, using the Sobel operator to perform edge detection on the H component image to generate a Sobel edge map;

[0054] Next, performing dilation and erosion operations on the segmentation mask obtained from the weakly supervised rough segmentation in the first stage, calculating the difference between the dilated mask and the eroded mask, and generating a mask edge map;

[0055] Subsequently, a pixel-by-pixel logical AND operation is performed on the Sobel edge map and the mask edge map to obtain an edge optimization label.

[0056] Preferably, the edge optimization label is used as a new weakly supervised label for the second-stage training, and the training loss of edge segmentation is calculated through a cross-entropy loss function; the formula expression is:

[0057]

[0058] where, S i represents the edge optimization label value at the i-th pixel, and y i is the mapping probability of pixel i;

[0059] Then the total training loss in the second stage is:

[0060] L total = L vor + L cul + L sob + α·L cor

[0061] where, α is a hyperparameter that controls the loss of the staining self-supervised task, N is the current training round, and N max is the total number of training rounds.

[0062] Compared with the prior art, the present invention has the following advantages and technical effects:

[0063] The present invention utilizes digital pathology and a deep learning segmentation algorithm to achieve the detection and segmentation of cell nuclei in H&E staining images using only partially point labels at the weakly supervised level, generating a pixel-level segmentation result and intuitively showing the spatial distribution of the internal structure of tumors.

[0064] The present invention proposes a weakly supervised cell nucleus detection and segmentation method based on partially point labels, using a deep learning method to construct a model to assist in segmenting cells in pathological images and presenting them intuitively. In principle, it can segment cell types with any number of categories, helping doctors analyze the spatial heterogeneity of the internal cell structure of tumors, thereby contributing to the prognosis analysis of lung cancer / breast cancer patients and formulating more appropriate treatment plans, which has great clinical significance. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] The drawings constituting a part of this application are used to provide a further understanding of this application. The schematic embodiments of this application and their descriptions are used to explain this application and do not constitute an improper limitation of this application. In the drawings:

[0066] Figure 1 is a schematic flowchart of the method of the embodiment of the present invention;

[0067] Figure 2 Schematic diagram of the overall algorithm for the embodiment of the present invention;

[0068] Figure 3 Schematic diagram of the nucleus detection algorithm based on partial point labels for the embodiment of the present invention. Detailed implementation manners

[0069] It should be noted that, without conflict, the embodiments in the present application and the features in the embodiments may be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.

[0070] It should be noted that the steps shown in the flowchart of the accompanying drawings may be executed in a computer system such as a set of computer-executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than here.

[0071] As Figures 1-3 shown, in this embodiment, a weakly supervised nucleus detection and segmentation method based on partial point labels is provided, including:

[0072] Obtain an H&E stained image to obtain a data set, divide the data set, and generate initial partial point labels based on the divided data set;

[0073] Generate pseudo-labels after Gaussian mask expansion based on the partial point labels;

[0074] Perform rough nucleus detection in the first stage based on the pseudo-labels after Gaussian expansion;

[0075] Perform self-supervised training for nucleus detection in the second stage by combining the background image and the original point labels to obtain the final nucleus detection result;

[0076] In the segmentation stage, generate an H single-component image as the input image of the segmentation network based on the staining separation technique;

[0077] Generate pseudo-labels (Voronoi labels and Cluster labels) for the segmentation stage based on the point labels for rough segmentation training;

[0078] Generate pseudo-labels of nucleus image edge information based on the edge detection strategy;

[0079] Perform fine nucleus segmentation training based on the generated edge information pseudo-labels, obtain the prediction result of the trained segmentation network, and use the prediction result as the final segmentation result.

[0080] Furthermore, the process of obtaining an H&E stained image to obtain a data set and dividing the data set includes:

[0081] Collect cytopathological section images, stain the pathological section images to obtain H&E stained pathological sections, and then digitally process the H&E stained pathological sections to obtain the H&E stained images; the staining process used in this embodiment is to stain the chromatin in the cell nucleus and the nucleic acid in the cytoplasm purple-blue with hematoxylin staining solution, and stain the components in the cytoplasm and extracellular matrix red with eosin staining solution. Then, a pathologist performs pixel-level annotation on each H&E stained image, carefully outlines each cell nucleus in the image, and then binarizes the outlined label to obtain a segmentation mask, and generates the required point labels based on the pixel-level annotation; in order to obtain the required point labels, this embodiment calculates the longest horizontal and vertical lengths of each cell nucleus according to the segmentation mask, and connects the lines at the maximum distance, and the intersection point is the centroid point of the required cell nucleus.

[0082] Further, the process of obtaining the H&E stained images to form a dataset and dividing the dataset includes:

[0083] Perform pixel-level annotation on each H&E stained image, generate the required point labels based on the pixel-level annotation, and randomly sample the point label images according to the set magnification to obtain partial point label images.

[0084] Further, the process of generating pseudo-labels after Gaussian mask expansion based on the partial point labels includes:

[0085] As Figure 3 shown, due to the serious class imbalance problem in the original point annotations, this embodiment converts the point annotations into Gaussian heatmaps, which is a commonly used method in detection tasks. The Gaussian heatmap is defined as:

[0086]

[0087] where H i is the i-th Gaussian expansion mask image, σ is the standard deviation, r 1 and r 2 respectively represent predefined hyperparameters of the centroid point to the foreground radius and background radius of the cell nucleus. D i is the Euclidean distance between the i-th pixel and its nearest point annotation. For pixels with an Euclidean distance greater than r 2 , this embodiment regards them as ignored regions, denoted as -1, and does not calculate them during training, where the Euclidean distance D i is expressed as:

[0088]

[0089] where N represents the number of point annotations in the image.

[0090] Further, the process of performing rough nucleus detection in the first stage based on the pseudo-labels of Gaussian expansion includes:

[0091] In this embodiment, the mask after Gaussian expansion is used to train a nucleus detection network to find those unlabeled points. This process is a per-pixel regression task. The loss function used is the mean squared error:

[0092]

[0093] where Y det is the predicted heatmap, and N is the total number of non-ignored pixels in the image.

[0094] Further, the process of obtaining the final nucleus detection result through self-supervised training for nucleus detection in the second stage by combining the background map and the original point labels includes:

[0095] In the initial training stage, due to the small number of labeled nuclei and the large background area being ignored, a large number of false positives appeared in the detection results of the initial model. These false positives may interfere with the learning process of the model in subsequent iterative training, thereby reducing the detection performance. To solve this problem, considering using the unlabeled area, the model performance is improved through a self-training learning method. For this purpose, this embodiment uses an iterative learning strategy to further optimize the background map in the self-training process through background propagation. This method aims to more precisely divide the nucleus region and the background region, thereby reducing the false positive labels generated by the model and improving the overall performance.

[0096] Specifically, in each round of self-training, first generate a probability map from the model of the previous round (in the first round of self-training, use the probability map of the initial training). Subsequently, combine the probability map with the original labeled points to generate a new training mask. When generating the background map, in this embodiment, pixels whose prediction probability meets specific conditions are determined as background pixels. Among them, a simple criterion is: if the prediction probability of a pixel is close to 0, it is regarded as the background because a probability of 0 indicates a very high likelihood that it does not belong to the nucleus. However, in the first stage of training, since most background pixels are ignored, the predicted values of many background pixels may be close to 1, which is an expected phenomenon. If the initial model wrongly predicts these pixels as nuclei, this prediction tendency will continue.

[0097] For those pixels whose prediction probability is close to 1, this embodiment further filters them, only retaining the larger connected components with an area greater than the average nucleus area. Finally, integrate the new background information into the extended mask to generate an updated training mask. Through this process, the background recognition ability of the model is enhanced, and the false positives are significantly reduced, thus achieving better detection performance.

[0098]

[0099] This embodiment proposes a detection network called FB-Net, whose design is based on an improved ResUnet-34 backbone network. Although ResUnet-34 is a simple and efficient medical image analysis model, it performs inadequately when dealing with complex histopathological images, especially in the segmentation task of the nucleus edge region. To address this issue, this embodiment improves ResUnet-34 in FB-Net and proposes two key modules, aiming to enhance the edge depiction ability based on foreground and background textures and effectively distinguish adjacent nuclei.

[0100] In histopathological images, the boundaries of nuclei are usually relatively blurred, which easily leads to noise generation during edge segmentation. For this reason, this embodiment introduces a feature denoising mechanism into the structure of ResUnet-34. Although the skip connections between the encoder and the decoder can retain spatial information to improve the segmentation effect, such connections often carry a large amount of unnecessary background noise. To alleviate this problem, this embodiment inserts a denoising function into the skip connections to reduce noise interference.

[0101] Specifically, this embodiment designs a Feature Denoising Module (FDM). Its core is based on the non-local mean method, and feature denoising is achieved by calculating the weighted mean of all positions in the image. The input feature hm of the module comes from the encoder and is processed by the non-local mean module. In addition, this embodiment introduces 1×1 convolution and residual connections to further fuse features, and this design refers to the literature. The finally generated denoised feature is passed to the decoder for subsequent processing.

[0102] After the feature denoising process of the edge, this embodiment still finds that the boundaries between adjacent nuclei in some difficult-to-separate samples are not clear. This phenomenon mainly stems from the similar texture features of adjacent nuclei, making it difficult for the network to correctly distinguish them. This embodiment observes that compared with high-level features, low-level features contain more detailed information, which helps to identify the foreground or boundaries. However, when the foreground features are similar, most existing models tend to focus on the background region, which has limited effect on segmenting closely contacting nuclei.

[0103] To solve this problem, this embodiment designs and inserts a Background Weakening Module (BWM) into the decoder. The role of BWM is to reduce the model's attention to the background region by fusing low-level features and high-level features. The combination of these two modules enables FB-Net to effectively enhance the edge segmentation accuracy and reduce noise interference when dealing with complex histopathological images, thus significantly improving the model performance.

[0104] Furthermore, the process of generating the H single-component image as the input image of the segmentation network based on the staining separation technique includes:

[0105] As Figure 2 shown, according to the principle of H&E staining, hematoxylin (H component) stains the cell nucleus blue-violet, while eosin (E component) stains the cytoplasm and matrix pink. Therefore, in the stained histopathological image, the cell nucleus and cytoplasm can usually be easily distinguished. To further clearly separate the cell nucleus from the background pixels, in this embodiment, a color decomposition technique is adopted to separate the hematoxylin and eosin components in the stained image. This technique relies on the color deconvolution method, and converts the red, green, and blue (RGB) values of hematoxylin and eosin into the corresponding H&E staining values through a pseudo-inverse matrix. The RGB values of hematoxylin and eosin are [0.644, 0.717, 0.267] and [0.093, 0.954, 0.283] respectively. Based on this, the calculation formula for the H&E staining value is as follows:

[0106]

[0107] where HE i is the hematoxylin and eosin channel values of pixel i, RGB i is the red, green, and blue channel values of pixel i, and + represents the pseudo-inverse of the matrix. Then, the hematoxylin component X h is obtained by keeping the hematoxylin channel:

[0108] x h = exp(-HE i [0,:])

[0109] where X h is a set of hematoxylin component images extracted from the H&E stained picture.

[0110] Furthermore, generating pseudo-labels for the segmentation stage based on point labels, the process of performing segmentation training through the pseudo-labels of the segmentation stage includes:

[0111] Based on the prediction map generated by the previous-stage detection network, this embodiment can construct a label map containing most point annotations. However, in practical applications, directly using point annotations for nucleus segmentation may lead to data imbalance problems due to insufficient supervision information. To solve this problem, this embodiment adopts the Voronoi diagram and mean clustering method to generate Voronoi labels and clustering labels respectively to enhance the supervision signal. Specifically, for Voronoi labels, the image is divided into convex polygons through point annotations. It is assumed that each point annotation is located at the center of the nucleus, and the nucleus shape is convex (although this assumption does not always hold). Under this assumption, the Voronoi label delimits an independent region for each nucleus, providing preliminary nuclear boundary information. For the generation of clustering labels, this embodiment uses the double-mean method to obtain more supervision information on nuclear boundaries and shapes. First, starting from point annotations, a distance map is generated by calculating the distance transformation between any two points. Then, the distance map is combined with the original H&E stained image for mean clustering, and the pixels are divided into n = 3 categories: nucleus, background, and ignored region. The cluster with the largest overlapping area with the point annotation is labeled as the nucleus, the cluster with the smallest overlapping area is labeled as the background, and the remaining undetermined categories are defined as the ignored region. The ignored region does not participate in the training, thus ensuring that the clustering can assign correct pixel labels as much as possible. Finally, to further optimize the clustering result, a series of morphological operations are applied to the initial clustering labels, including connected component labeling, discrete region removal, morphological operations, and binary hole filling.

[0112] Using the generated pixel-level rough labels (i.e., Voronoi and clustering pseudo-labels), preliminary nucleus segmentation can be achieved. For the segmentation network, this embodiment adopts ResUNet as the segmentation network integrating residual blocks. However, considering the differences in information at different scales in this embodiment, a segmentation network with a double-branch fusion framework is introduced and trained with respect to the cross-entropy loss of Voronoi labels and clustering labels:

[0113]

[0114] where, v i and v r represent the Voronoi label and clustering label categories of the nucleus at the i-th pixel, y i is the mapping probability of pixel i, that is, the probability that pixel i is predicted as the nucleus, v i and c i represent the Voronoi label or adaptive clustering label of pixel i respectively, the nucleus is labeled as 1, the background is labeled as 0, and N represents the total number of pixels in the image.

[0115] To make full use of the nuclear boundary information, in this embodiment, a self-supervised visual representation learning method for staining is designed, and the segmentation of cell nuclei in pathological images is achieved through image coloring based on the H component. The pipeline of this method consists of two networks arranged in sequence: the first network generates the probability map of cell nuclei from the H-component image, and the second network reconstructs the original H&E-stained image based on the segmentation probability map. It should be noted that the segmentation process is trained through Voronoi labels and clustering labels. The coloring process converts the H-component image into an H&E-stained image in an unsupervised manner. The segmentation network and the coloring network are connected through the cell nucleus probability map to achieve end-to-end joint training. Through this design, the coloring task can implicitly promote the representation learning of the cell nucleus features by the segmentation network.

[0116] Finally, the model is optimized by calculating the coloring loss between the predicted image and the original image, and its specific formula is as follows:

[0117]

[0118] where \(x\) i represents the \(i\)-th pixel of the original H&E-stained image, \(\Omega\) represents the set of all pixels, \(C(\cdot)\) represents the prediction of the coloring network, and \(S(X\) h ) represents the probability map generated by the segmentation network from the H-component image. By solving the proposed coloring task, the segmentation network can capture more low-level features from the H-component image and implicitly model the relationship between the cell nucleus and the cytoplasm.

[0119] Furthermore, the process of generating the pseudo-label of the cell nucleus image edge information based on the edge detection strategy includes:

[0120] After the weakly supervised coarse segmentation training in the first stage is completed, the error of the model on the edge contour is still relatively large. Therefore, optimizing the edge contour becomes the key to improving the segmentation accuracy. For this purpose, this embodiment proposes a new edge optimization strategy to improve the edge segmentation performance of the model.

[0121] Specifically, this strategy assists the fine segmentation training in the second stage by generating edge optimization labels. Since the color difference between the cell nuclei and the background regions in the pathological slice image after extracting the H component is significant, in this embodiment, the Sobel operator is first used to perform edge detection on the H-component image to generate the Sobel edge map. Then, dilation and erosion operations are performed on the segmentation mask obtained from the weakly supervised coarse segmentation in the first stage, and the difference between the dilated mask and the eroded mask is calculated to generate the edge map of the mask. Subsequently, a pixel-by-pixel logical AND operation is performed on the Sobel edge map and the mask edge map to obtain the refined edge optimization label.

[0122] These edge optimization labels are used as new weakly supervised labels for the second-stage training, and the training loss of edge segmentation is calculated through the cross-entropy loss function:

[0123]

[0124] where S i represents the edge optimization label value at the i-th pixel, and y i is the mapping probability of pixel i. Therefore, the total training loss in the second stage is:

[0125] L total = L vor + L cul + L sob + α·L cor

[0126] where α is a hyperparameter that controls the loss of the staining self-supervised task. Considering that as the training process progresses, the training focus should gradually change from representation learning (coloring) to the target task (segmentation), the hyperparameter α should gradually decrease. Therefore, in this embodiment, it is set as a dynamic hyperparameter, where N is the current training round, and N max is the total number of training rounds.

[0127] The present invention can utilize digital pathology and deep learning classification algorithms to automatically identify cells within a tumor from H&E staining images, generate a final segmentation result, and intuitively display the spatial distribution within the tumor, helping doctors grade patients and perform prognostic analysis.

[0128] The above is only a preferred specific embodiment of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by those skilled in the art within the technical scope disclosed in the present application should be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A weakly supervised cell nucleus detection and segmentation method based on partial point labels, characterized in that: include: Acquire an H&E staining image to obtain a data set, divide the data set, and generate partial point labels based on the divided data set; Generate a pseudo label after Gaussian mask expansion based on the partial point labels, and perform a rough cell nucleus detection in the first stage based on the Gaussian expanded pseudo label; The final cell nucleus detection result is obtained by performing self-supervised training of the second stage of cell nucleus detection based on the combination of background image and original point label; In the segmentation stage, H single-component images are generated based on the color separation technique as the input image of the segmentation network; Generate pseudo labels for the segmentation stage based on the point labels, and perform segmentation training using the pseudo labels for the segmentation stage; Generate pseudo labels of edge information of cell nucleus images based on edge detection strategy; Based on the edge information pseudo-label, cell nucleus segmentation training is performed to obtain a prediction result of the trained segmentation network, and the prediction result is used as the final segmentation result.

2. The method according to claim 1, characterized in that The H&E staining image is obtained to obtain a data set, and the process of dividing the data set includes: Pathological section images of cells are collected, and the pathological section images are stained to obtain H&E stained pathological sections; and the H&E stained pathological sections are digitized to obtain the H&E staining images.

3. The method according to claim 1, characterized in that The process of generating partial point labels based on the divided data set includes: Performing pixel-level annotation on the H&E staining image, and generating point labels based on the pixel-level annotation; According to the set magnification, the point label image is randomly sampled to obtain a partial point label image.

4. The method according to claim 1, characterized in that: The process of generating a pseudo label after Gaussian mask expansion based on the partial point label includes: Generate pseudo labels after Gaussian mask expansion by converting point labels into Gaussian heatmaps; The formula for defining the Gaussian heat map is: Among them, H i is the i-th Gaussian expansion mask, σ is the standard deviation, r1 and r2 represent the predefined hyperparameters from the centroid point to the foreground radius and background radius of the cell nucleus, respectively; D i is the Euclidean distance between the i-th pixel and its nearest annotation. Pixels with a Euclidean distance greater than r2 are considered ignored and are represented as -1 and are not calculated during training. i It is expressed as: Where N represents the number of point annotations in the image.

5. The method according to claim 1, characterized in that The process of performing the first stage rough cell nucleus detection based on the Gaussian extended pseudo-labels includes: Using the pixel-by-pixel regression method, the Gaussian expanded mask is used to train a cell nucleus detection network to obtain unlabeled points; The pixel-by-pixel regression method uses mean square error as the loss function, and the formula is: Among them, Y det is the predicted heatmap and N is the total number of pixels in the image that are not ignored.

6. The method according to claim 1, characterized in that The process of performing the second stage of self-supervised training of cell nucleus detection based on the background image and the original point label to obtain the final cell nucleus detection result includes: Using unlabeled regions, the background map in the iterative self-training process is optimized by background propagation. The first round of self-training uses the probability map of the initial training. In each round of self-training, the probability map is first generated from the model of the previous round. Subsequently, the probability map is combined with the original annotated points to generate a new training mask; when generating the background map, pixels whose predicted probabilities meet specific conditions are determined as background pixels; wherein the specific conditions include: if the predicted probability of a pixel is close to 0, it is considered as background; for pixels whose predicted probability is close to 1, only connected components whose areas are larger than the average cell nucleus area are retained after further screening; Finally, the new background information is incorporated into the expanded mask to generate an updated training mask.

7. The method according to claim 1, characterized in that The process of generating H single-component images as the input image of the segmentation network based on the color separation technology includes: According to the principle of H&E staining, the cell nucleus is separated from the background pixels; wherein the principle of H&E staining includes staining the cell nucleus into blue-purple by hematoxylin and staining the cytoplasm and matrix into pink by eosin; Then, the color deconvolution method is used to convert the red, green, and blue values ​​of hematoxylin and eosin into corresponding H&E staining values ​​through a pseudo-inverse matrix to separate the hematoxylin and eosin components in the staining image; Among them, the RGB values ​​of hematoxylin and eosin are [0.644, 0.717, 0.267] and [0.093, 0.954, 0.283], respectively, and the formula expression of the H&E staining value is: Among them, HE i is the hematoxylin and eosin channel value of pixel i, RGB i are the red, green, and blue channel values ​​of pixel i, and + represents the pseudo-inverse of the matrix; By keeping the hematoxylin channel, the hematoxylin fraction X is obtained. h : x h =exp(-HE i [0,:]) Among them, X h A collection of hematoxylin fraction images extracted from H&E stained images.

8. The method according to claim 1, characterized in that The process of generating pseudo labels in the segmentation stage based on the point labels and performing segmentation training using the pseudo labels in the segmentation stage includes: The Voronoi diagram and mean clustering methods are used to generate Voronoi labels and cluster labels respectively; For the Voronoi label, the image is divided into convex polygons by point annotation; assuming that each point annotation is located at the center of the cell nucleus and the shape of the cell nucleus is convex, an independent area is delineated for each cell nucleus to obtain preliminary nuclear boundary information; The bimean method is used to obtain the supervision information of the nuclear boundary and shape of the cluster label. First, starting from the point annotation, a distance map is generated by calculating the distance transformation between any two points. Then, the distance map is combined with the original H&E staining image to perform mean clustering and divide the pixels into three categories: cell nucleus, background, and ignored area. The cluster with the largest overlapping area with the point annotation is marked as the cell nucleus, the cluster with the smallest overlapping area is marked as the background, and the remaining undetermined categories are defined as the ignored area to obtain the initial cluster labels; Morphological operations including connected domain labeling, discrete region removal, morphological operations and binary hole filling are applied to the initial cluster labels to optimize the clustering results and obtain final cluster labels.

9. The method according to claim 1, characterized in that: The process of generating pseudo labels of edge information of cell nucleus images based on edge detection strategy includes: When optimizing edge contours through edge optimization strategies, the second stage of segmentation training is assisted by generating edge optimization labels; The process of generating edge-optimized labels to assist the second stage of segmentation training includes: First, the Sobel operator is used to perform edge detection on the H component image to generate a Sobel edge map; Next, the segmentation mask obtained by the weakly supervised coarse segmentation in the first stage is expanded and eroded, and the difference between the expanded mask and the eroded mask is calculated to generate a mask edge map; Subsequently, a pixel-by-pixel logical AND operation is performed on the Sobel edge map and the mask edge map to obtain an edge optimization label.

10. The method according to claim 9, characterized in that The edge optimization label is used as a new weak supervision label for the second stage training, and the training loss of edge segmentation is calculated by the cross entropy loss function; the formula expression is: Among them, S i represents the edge optimization label value at the i-th pixel, y i is the mapping probability of pixel i; The total training loss in the second stage is: L total =L vor +L cul +L sob +α·L cor Among them, α is a hyperparameter that controls the loss of the self-supervised coloring task, N is the current training round, and N max is the total number of training rounds.

Images