A pathological image cell segmentation method based on a U-Net model
Patent Information
- Application Number
- CN202411146505.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-20
- Publication Date
- 2026-08-18
- Estimated Expiration
- 2044-08-20
AI Technical Summary
在标准临床流程中，病理学家需要在高放大倍率下检查组织切片并手动计数500-1000个细胞，这种方法既耗时又容易出错，并且需要病理学家具备高水平的专业知识
[0017] 1. This invention incorporates a multi-scale attention module in the decoder stage of the network. This module effectively combines multi-scale feature fusion and soft attention mechanisms, enabling it to comprehensively capture global and local features in the image and significantly enhance the representation of key cell features, thereby improving the accuracy of cell segmentation;
Smart Images
Figure CN119006497B_ABST
Abstract
Description
Technical Field
[0001] This invention provides a cell segmentation method for pathological images based on the U-Net model, belonging to the field of deep learning for medical image segmentation. Background Technology
[0002] Invasive breast cancer is considered a heterogeneous disease with diverse morphologies, treatment responses, and molecular characteristics. In clinical practice, at least four heterogeneous molecular subtypes are identified, and the differentiation of these subtypes relies on the genetic expression of four biomarkers: estrogen receptor (ER) status, progesterone receptor (PgR) status, human epidermal growth factor receptor-2 (HER2) status, and the Ki-67 proliferation index. Ki-67 is an excellent marker of cell proliferation and a recognized marker of tumorigenesis. In addition to Ki-67, tumor-infiltrating lymphocytes (TILs) are also a key biomarker. TILs are lymphocytes present in tumor tissue and are crucial for the prognosis and treatment response of breast cancer patients. In standard clinical procedures, pathologists examine tissue sections at high magnification and manually count 500-1000 cells, a time-consuming, error-prone method requiring a high level of expertise.
[0003] To more efficiently identify and process these two types of markers, an increasing number of artificial intelligence methods have been proposed. In recent years, digital pathology has received increasing attention as whole-slice scanning capabilities and computer hardware capable of processing high-resolution whole-slice images have become widely available. A major benefit of using AI-based systems in pathology is the provision of reliable, objective, and reproducible results, thereby reducing discrepancies in outcome analysis among pathologists and enabling automation. Summary of the Invention
[0004] The technical problem solved by this invention is: This invention proposes a cell segmentation method for pathological images based on the U-Net model, which can effectively segment Ki-67 positive cells, Ki-67 negative cells and tumor-infiltrating lymphocytes in pathological images stained with IHC.
[0005] The technical solution of this invention is: a cell segmentation method for pathological images based on the U-Net model, comprising the following:
[0006] Step 1: First, preprocess the original point labels by converting the original point labels at the cell centers into density distribution maps to optimize the learning effect of the network.
[0007] Step 2: Building upon the U-Net model, a Multi-Scale Attention Module (MA Module) is used in the encoder. This module combines multi-scale feature fusion and soft attention mechanisms to comprehensively capture global and local features in the image. By processing inputs at different scales and integrating information from the image, this module significantly improves the network's ability to capture cell features of various sizes. This enables the model to more accurately identify and analyze cells of different sizes and shapes in microscope images, thereby improving segmentation and localization accuracy. A Pixel Channel Fusion Module (PCF Module) is used between the decoders. This module fuses pixel-level and channel-level attention mechanisms between decoder layers, optimizing the detail representation of feature maps.
[0008] Step 3: Finally, in post-processing, a specific threshold is first set. Based on the threshold, the network output density distribution map is converted into a discrete binary mask, and then the cell center position is determined by using distance transformation and watershed algorithm.
[0009] As a further aspect of the present invention, Step 1 defines a point annotation preprocessing method, which converts each original cell point annotation into a probability density distribution map. The specific steps of Step 1 are as follows:
[0010] Step 1.1. For each pixel in the image, calculate its Euclidean distance to the nearest and second-nearest cell centers. These two distances are denoted as the nearest distance and the second-nearest distance, respectively.
[0011] Step 1.2. For each pixel, calculate a new intermediate variable by combining its distances to the nearest and second-nearest cell centers. This intermediate variable reflects the distance weight of the pixel after considering the influence of neighboring cells;
[0012] Step 1.3. Assign a rejection coding value to each pixel based on the intermediate variable. This value is adjusted according to the magnitude of the intermediate variable, giving higher weight to pixels closer to the cell center, while suppressing responses between adjacent cells. This method effectively solves the problem of overlapping when cells that are too close together are generated in the density map.
[0013] As a further aspect of the present invention, Step 2 includes an improved encoder and decoder section of the U-Net. The specific steps of Step 2 are as follows:
[0014] Step 2.1. We replace the traditional encoder with a Multi-Scale Attention Module (MA Module). This module contains three parallel paths in the encoding stage. The first path consists of two consecutive 3×3 convolutions to extract basic convolutional features. As network depth increases, the need for larger receptive fields increases, but using excessively large receptive fields in the early stages of the network has limited effect. Therefore, this module is designed to be more flexible in the encoding stage. The second path of the first two encoding layers uses two 3×3 dilated convolutions with a dilation rate of 2, and the last two encoding layers use 3×3 dilated convolutions with a dilation rate of 4. Since large receptive field connections rely on the assistance of small receptive field connections, the output channels of the first 3×3 convolution branch are twice that of the second branch to enhance the efficiency of feature extraction and information integration. The third path extracts attention through compression and upsampling to generate a global soft attention mask. This attention mask enhances the model's focus on important regions by highlighting key features and suppressing unimportant information by weighting each position in the feature map element-wise. Finally, the module implements residual connections through 1×1 convolutions. This design enhances the network's ability to capture multi-scale features and focus attention on key features.
[0015] Step 2.2. We inserted a Pixel Channel Fusion Module (PCFModule) between the encoders. This module first processes the input feature map using Local Average Pooling (LAP) and Global Average Pooling (GAP). LAP captures features from specific regions, while GAP captures information from the entire feature map. These two pooling operations are processed in parallel, each interacting across channels through one-dimensional convolution. Subsequently, the global pooling features are upsampled to the same size as the local pooling features, and both are multiplied by learnable coefficients and then added together. The merged features are upsampled to the original feature map size and multiplied with the original features. Finally, a pixel attention layer is added. The channel features of each pixel are weighted, allowing the model to more accurately focus on important features in the image. This pixel-level attention processing not only improves the model's ability to capture local details but also optimizes the information representation of the overall feature map, thereby improving segmentation accuracy.
[0016] The beneficial effects of this invention are:
[0017] 1. This invention incorporates a multi-scale attention module in the decoder stage of the network. This module effectively combines multi-scale feature fusion and soft attention mechanisms, enabling it to comprehensively capture global and local features in the image and significantly enhance the representation of key cell features, thereby improving the accuracy of cell segmentation;
[0018] 2. This invention adds a pixel channel fusion module between the layers of the decoder. This module further enhances the model's ability to recognize details by integrating local information and pixel-level attention into the channel attention mechanism.
[0019] 3. This invention can more accurately focus on important features in an image; pixel-level attention processing not only improves the model's ability to capture local details, but also optimizes the information representation of the overall feature map, thereby improving segmentation accuracy;
[0020] 4. This invention can accurately locate Ki-67 positive, Ki-67 negative and tumor-infiltrating lymphocytes in pathological images and calculate the corresponding biomedical indicators (Ki-67 proliferation index), thereby assisting doctors in disease diagnosis. Attached Figure Description
[0021] Figure 1 The left image shows the original point label, and the right image shows the image after rejection coding preprocessing.
[0022] Figure 2 This is a network structure diagram of the present invention;
[0023] Figure 3 This is the Multi-Scale Attention Module of the present invention;
[0024] Figure 4 The left image shows the Pixel Channel Fusion Module, and the right image shows the Pixel Attention Module.
[0025] Figure 5 This is a comparison chart of the results of this invention. Red dots represent Ki-67 positive tumor cells, green dots represent Ki-67 negative tumor cells, and blue dots represent tumor-infiltrating lymphocytes.
[0026] Figure 6 This is a flowchart of the method of the present invention. Detailed Implementation
[0027] Example 1: As Figures 1-6 As shown, a cell segmentation method for pathological images based on the U-Net model includes the following:
[0028] Step 1: First, preprocess the original point labels by converting the original point labels at the cell centers into density distribution maps to optimize the learning effect of the network.
[0029] As Step 1 in this invention, its specific steps are as follows:
[0030] Step 1.1. For each pixel in the image, calculate its Euclidean distance to the nearest and second-nearest cell centers. These two distances are denoted as the nearest distance and the second-nearest distance, respectively.
[0031] Step 1.2. For each pixel, calculate a new intermediate variable by combining its distances to the nearest and second-nearest cell centers. This intermediate variable reflects the distance weight of the pixel after considering the influence of neighboring cells;
[0032] Step 1.3. Assign a rejection coding value to each pixel based on the intermediate variable. This value is adjusted according to the magnitude of the intermediate variable, giving higher weights to pixels closer to the cell center while suppressing responses between adjacent cells. This method effectively solves the problem of overlapping when cells that are too close together are generated in the density map. The effect after preprocessing is as follows: Figure 1 As shown in the three pictures on the right.
[0033] Step 2: Building upon the U-Net model, a Multi-Scale Attention Module (MA Module) is used in the encoder. This module combines multi-scale feature fusion and soft attention mechanisms to comprehensively capture global and local features in the image. By processing inputs at different scales and integrating information from the image, this module significantly improves the network's ability to capture cell features of various sizes. This enables the model to more accurately identify and analyze cells of different sizes and shapes in microscope images, thereby improving segmentation and localization accuracy. A Pixel Channel Fusion Module (PCF Module) is used between the decoders. This module fuses pixel-level and channel-level attention mechanisms between decoder layers, optimizing the detail representation of feature maps. The overall network structure is as follows: Figure 2 As shown.
[0034] Step 2 in this invention comprises the following specific steps:
[0035] Step 2.1. We replace the traditional encoder with a Multi-Scale Attention Module (MA Module). This module contains three parallel paths in the encoding stage. The first path consists of two consecutive 3×3 convolutions to extract basic convolutional features. As network depth increases, the need for larger receptive fields increases, but using excessively large receptive fields in the early stages of the network has limited effect. Therefore, this module is designed to be more flexible in the encoding stage. The second path of the first two encoding layers uses two 3×3 dilated convolutions with a dilation rate of 2, and the last two encoding layers use 3×3 dilated convolutions with a dilation rate of 4. Since large receptive field connections rely on the assistance of small receptive field connections, the output channels of the first 3×3 convolution branch are twice that of the second branch to enhance the efficiency of feature extraction and information integration. The third path extracts attention through compression and upsampling to generate a global soft attention mask. This attention mask enhances the model's focus on important regions by highlighting key features and suppressing unimportant information by weighting each position in the feature map element-wise. Finally, the module implements residual connections through 1×1 convolutions. This design enhances the network's ability to capture multi-scale features and its focus on key features. The module structure is as follows: Figure 3 As shown.
[0036] Step 2.2. We inserted a Pixel Channel Fusion Module (PCFModule) between the encoders. This module first processes the input feature map using Local Average Pooling (LAP) and Global Average Pooling (GAP). LAP captures features from specific regions, while GAP captures information from the entire feature map. These two pooling operations are processed in parallel, each interacting across channels through one-dimensional convolution. Subsequently, the global pooling features are upsampled to the same size as the local pooling features, and both are multiplied by learnable coefficients and then added together. The merged features are upsampled to the original feature map size and multiplied with the original features. Finally, a pixel attention layer is added. The channel features of each pixel are weighted, allowing the model to more accurately focus on important features in the image. This pixel-level attention processing not only improves the model's ability to capture local details but also optimizes the information representation of the overall feature map, thereby improving segmentation accuracy. The module structure is as follows: Figure 4 As shown.
[0037] Step 3: In post-processing, a specific threshold is first set. Based on the threshold, the network output density distribution map is converted into a discrete binary mask, and then the cell center position is determined by using distance transformation and watershed algorithm.
[0038] According to the above implementation process, it can be combined with Figure 2 The working principle of this invention can be summarized as follows:
[0039] 1. The original point annotations of the pathological images are preprocessed as in Step 1 and converted into density distribution maps to optimize the learning effect of the network.
[0040] 2, in Figure 2 The improved model is used for training and prediction of image data. The improvements include the use of a Multi-Scale Attention Module in the encoder and a Pixel Channel Fusion Module between the decoders, covering the content of Step 2.
[0041] Therefore, a model for cell segmentation based on IHC-stained pathological images can be obtained to assist doctors in counting the number of Ki-67 positive cells, Ki-67 negative cells, and tumor-infiltrating lymphocytes, such as... Figure 5 The aforementioned effects allow for a more accurate diagnosis of the condition.
[0042] To further illustrate the experimental results of this invention, the experimental results of this invention are now presented:
[0043] First, the experimental environment for this invention was as follows: a 12v CPU Intel(R) Xeon(R) Platinum 8255C CPU @ 2.50GHz, 43.0GB of RAM, and an NVIDIA GeForce RTX3090 graphics card with 24GB of VRAM. The optimizer used was Adam, employing a cosine annealing learning rate decay strategy with hot restart, restarting the learning rate every 30 epochs, and reducing it to a minimum of 1e-5.
[0044] Following the experimental procedure, tests were performed on 700 IHC-stained images with a resolution of 256×256. The average precision for the three cell types was 0.7790, the recall was 0.8251, and the F1 score was 0.8014. The evaluation metrics are shown in Table 1.
[0045] Table 1 Experimental results of the present invention
[0046] Method of the present invention IHC staining images 0.7790 0.8251 0.8014
[0047] As can be seen, the method of this invention performs well in the overall detection of the three types of cells on IHC staining images, indicating that the method of this invention can effectively assist doctors in the quantitative analysis of the above-mentioned biomarkers. Therefore, this invention has certain clinical significance.
[0048] The specific embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.
Claims
1. A method for cell segmentation in pathological images based on the U-Net model, characterized in that, Including the following: Step 1: First, preprocess the original point annotations to convert the original point annotations at the cell centers into a density distribution map. Step 2: Based on the U-Net model, a multi-scale attention module is used in the encoder part. The multi-scale attention module combines multi-scale feature fusion and soft attention mechanism to comprehensively capture global and local features in the image. A pixel-channel fusion module is used between decoders. The pixel-channel fusion module fuses pixel-level and channel-level attention mechanisms between different layers of the decoder, which optimizes the detail representation of the feature map. Step 3: Finally, in terms of post-processing, a threshold is first set, and the network output density distribution map is converted into a discrete binary mask based on the threshold. Then, the cell center position is determined by using distance transformation and watershed algorithm. In Step 1, exclusion coding is used to convert the original point markers into a density distribution map. The specific steps in Step 1 are as follows: Step 1.
1. For each pixel in the image, calculate its Euclidean distance to the nearest and second nearest cell centers, denoted as the nearest distance and the second nearest distance, respectively. Step 1.
2. For each pixel, combine its distance to the nearest and second nearest cell center to calculate a new intermediate variable. This intermediate variable reflects the distance weight of the pixel after considering the influence of neighboring cells. Step 1.
3. Assign a rejection coding value to each pixel based on the intermediate variable. This value is adjusted according to the size of the intermediate variable so that pixels closer to the cell center have higher weights, while the response between adjacent cells is suppressed. In Step 2, the encoder and decoder parts of U-Net were modified. The specific steps in Step 2 are as follows: Step 2.
1. Replace the traditional encoder with a multi-scale attention module. The multi-scale attention module contains three parallel paths in the encoding stage. The first path consists of two consecutive 3×3 convolutions to extract basic convolutional features. The second path of the first two encoding layers uses two 3×3 dilated convolutions with a dilation rate of 2, and the last two encoding layers use 3×3 dilated convolutions with a dilation rate of 4. The output channels of the first 3×3 convolution branch are twice that of the second branch. The third path extracts attention through compression and upsampling to generate a global soft attention mask. This attention mask highlights key features and suppresses unimportant information by weighting each position in the feature map element by element. Finally, residual connections are achieved through 1×1 convolutions. Step 2.
2. A pixel channel fusion module is inserted between the encoders. This module first processes the input feature map through Local Average Pooling (LAP) and Global Average Pooling (GAP). LAP captures features of specific regions, while GAP acquires information from the entire feature map. These two pooling operations are processed in parallel, and each interacts across channels through one-dimensional convolution. Subsequently, the global pooling features are upsampled to the same size as the local pooling features. Both are multiplied by learnable coefficients and then added together. The merged features are upsampled to the original feature map size and multiplied with the original features. Finally, a pixel attention layer is added to weight the channel features of each pixel.