Pathology image cell segmentation method based on U-Net model
Through the pathological image cell segmentation method based on U-Net model, combined with multi-scale attention and pixel-level attention mechanism, the time-consuming and error-prone cell segmentation in the prior art is solved, and Ki-67 cells and The precise positioning and segmentation of lymphocytes improves the accuracy of disease judgment.
Patent Information
- Application Number
- CN202411146505.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-20
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2044-08-20
AI Technical Summary
In clinical practice, pathologists need to manually count large numbers of cells, which makes it time-consuming and error-prone, and the prior art is difficult to efficiently identify and segment Ki-67-positive cells, Ki-67-negative cells and tumor-infiltrating lymphocytes, affecting the disease Prognosis and treatment response.
Using the pathological image cell segmentation method based on the U-Net model, the original point labeled as density distribution map is used, combined with the Multi-Scale Attention Module and Pixel Channel Fusion Module, the network capture ability and segmentation of multi-scale features is significantly improved. Accuracy, and the cell center position is determined by distance transformation and watershed algorithms.
The precise positioning and segmentation of Ki-67-positive, Ki-67-negative and tumor-infiltrating lymphocytes was achieved, which improved the accuracy and accuracy of cell segmentation, assisted doctors in disease judgment, and reduced artificial errors.
Smart Images

Figure CN119006497A_ABST
Abstract
Description
Technical Field
[0001] The present invention provides a pathological image cell segmentation method based on a U-Net model, belonging to the field of deep learning for medical image segmentation. Background Art
[0002] Invasive breast cancer is considered a heterogeneous disease with different morphology, treatment response, and molecular characteristics. In clinical practice, at least four heterogeneous molecular subtypes are distinguished, and the distinction between these subtypes depends on the genetic expression of four biomarkers: estrogen receptor (ER) status, progesterone receptor (PgR) status, human epidermal growth factor receptor-2 (HER2) status, and Ki-67 proliferation index. Ki-67 is an excellent cell proliferation marker and a recognized hallmark of tumorigenesis. In addition to Ki-67, tumor infiltrating lymphocytes (TILs) are also key biomarkers. TILs are lymphocytes present in tumor tissue and are crucial for the prognosis and treatment response of breast cancer patients. In standard clinical procedures, pathologists need to examine tissue sections under high magnification and manually count 500-1000 cells, which is time-consuming, error-prone, and requires a high level of expertise on the part of the pathologist.
[0003] In order to more efficiently identify and process the above two markers, more and more artificial intelligence methods have been proposed. In recent years, as whole-slide scanning capabilities and computer hardware capable of processing high-resolution whole-slide images have become widely available, digital pathology has received increasing attention. A major benefit of using AI-based systems in pathology is to provide reliable, objective, and reproducible results, thereby reducing differences in result analysis between pathologists and achieving automation. Summary of the invention
[0004] The technical problem solved by the present invention is: the present invention proposes a pathological image cell segmentation method based on a U-Net model, which can effectively segment Ki-67 positive cells, Ki-67 negative cells and tumor infiltrating lymphocytes in pathological images stained by IHC.
[0005] The technical solution of the present invention is: a pathological image cell segmentation method based on a U-Net model, comprising the following:
[0006] Step 1: First, preprocess the original point annotations and convert the original point annotations of the cell center into a density distribution map to optimize the learning effect of the network;
[0007] Step 2. Secondly, based on the U-Net model, the Multi-Scale Attention Module (MA Module) is used in the encoder part. The Multi-Scale Attention Module combines multi-scale feature fusion and soft attention mechanism to fully capture the global and local features in the image. This module processes inputs of different scales and integrates information in the image in the encoder, which significantly improves the network's ability to capture features of cells of various sizes. It enables the model to more accurately identify and analyze cells of different sizes and shapes in microscope images, thereby improving the accuracy of segmentation and positioning. The Pixel Channel Fusion Module (PCF Module) is used between decoders. This module fuses pixel-level and channel-level attention mechanisms between the decoder layers to optimize the detail performance of the feature map.
[0008] Step 3. Finally, in terms of post-processing, a specific threshold is set first. According to the threshold, the network output density distribution map is converted into a discrete binary mask, and then the cell center position is determined by using distance transformation and watershed algorithm.
[0009] As a further solution of the present invention, in Step 1, a point annotation preprocessing method is defined, and for each cell original point annotation, it is converted into a probability density distribution map. The specific steps of Step 1 are:
[0010] Step 1.1. For each pixel in the image, calculate its Euclidean distance to the nearest cell center and the second nearest cell center. These two distances are recorded as the nearest distance and the second nearest distance respectively;
[0011] Step 1.2. For each pixel, calculate a new intermediate variable by combining its distance to the nearest cell center and the second nearest cell center. This intermediate variable reflects the distance weight of the pixel after considering the influence of neighboring cells;
[0012] Step 1.3. According to the intermediate variable, an exclusion coding value is assigned to each pixel. This value is adjusted according to the size of the intermediate variable, so that the pixels closer to the cell center have higher weights, and the responses between adjacent cells are suppressed. This method effectively solves the problem of overlapping cells that are too close when generating density maps.
[0013] As a further solution of the present invention, Step 2 includes an improved encoder part and a decoder part of U-Net. The specific steps of Step 2 are:
[0014] Step2.1. We use the Multi-Scale Attention Module (MA Module) module to replace the traditional encoder. This module contains three parallel paths in the encoding stage. The first path consists of two consecutive 3×3 convolutions to extract basic convolution features. As the network depth increases, the demand for larger receptive fields increases, but using too large a receptive field in the early stage of the network has limited effect. Therefore, the design of this module in the encoding stage is more flexible. The second path of the first two encoding layers uses two 3×3 dilated convolutions with a dilation rate of 2, and the last two encoding layers use 3×3 dilated convolutions with a dilation rate of 4. Since large receptive field connections rely on the assistance of small receptive field connections, the output channels of the first 3×3 convolution branch are twice that of the second branch to enhance the efficiency of feature extraction and information integration. The third path extracts through compression and attention, and generates a global soft attention mask through upsampling. This attention mask highlights key features and suppresses unimportant information by weighting each position in the feature map element by element, thereby enhancing the model's attention to important areas. Finally, the module implements residual connections through 1×1 convolutions. This design improves the network's ability to capture multi-scale features and focus on key features.
[0015] Step 2.2. We inserted a Pixel Channel Fusion Module (PCFModule) between the encoders. The module first processes the input feature map through local average pooling (LAP) and global average pooling (GAP). LAP captures the features of a specific area, while GAP obtains the information of the entire feature map. These two pooling operations are processed in parallel and each interacts across channels through a one-dimensional convolution. Subsequently, the global pooling features are upsampled to the same size as the local pooling features, and the two are multiplied by a learnable coefficient and added together. The merged features are upsampled to the original feature map size and multiplied with the original features. Finally, a pixel attention layer is added. The channel features of each pixel are weighted so that the model can focus on important features in the image more accurately. This pixel-level attention processing not only improves the model's ability to capture local details, but also optimizes the information expression of the overall feature map, thereby improving segmentation accuracy.
[0016] The beneficial effects of the present invention are:
[0017] 1. The present invention adds a multi-scale attention module in the decoder stage of the network. This module effectively combines multi-scale feature fusion and soft attention mechanism, can fully capture the global and local features in the image, and significantly enhance the ability to represent key cell features, thereby improving the accuracy of cell segmentation;
[0018] 2. The present invention adds a pixel channel fusion module between the decoder layers. This module further enhances the model's ability to recognize details by integrating local information and pixel-level attention into the channel attention mechanism;
[0019] 3. The present invention can focus on important features in the image more accurately; pixel-level attention processing not only improves the model's ability to capture local details, but also optimizes the information expression of the overall feature map, thereby improving segmentation accuracy;
[0020] 4. The present invention can accurately locate Ki-67 positive, Ki-67 negative and tumor infiltrating lymphocytes in pathological images and calculate the corresponding biomedical index (Ki-67 proliferation index), thereby assisting doctors in disease diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] Figure 1 The left image is the image with original point annotations, and the right image is the image preprocessed with exclusion coding;
[0022] Figure 2 This is a network structure diagram of the present invention;
[0023] Figure 3 The Multi-Scale Attention Module module of the present invention;
[0024] Figure 4 The left picture shows the Pixel Channel Fusion Module, and the right picture shows the Pixel Attention Module;
[0025] Figure 5 This is a comparison chart of the results of the present invention, red dots represent Ki-67 positive tumor cells, green dots represent Ki-67 negative tumor cells, and blue dots represent tumor infiltrating lymphocytes;
[0026] Figure 6 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION
[0027] Example 1: Figure 1-Figure 6 As shown, a pathological image cell segmentation method based on the U-Net model includes the following:
[0028] Step 1: First, preprocess the original point annotations and convert the original point annotations of the cell center into a density distribution map to optimize the learning effect of the network;
[0029] As Step 1 in the present invention, the specific steps are:
[0030] Step 1.1. For each pixel in the image, calculate its Euclidean distance to the nearest cell center and the second nearest cell center. These two distances are recorded as the nearest distance and the second nearest distance respectively;
[0031] Step 1.2. For each pixel, calculate a new intermediate variable by combining its distance to the nearest cell center and the second nearest cell center. This intermediate variable reflects the distance weight of the pixel after considering the influence of neighboring cells;
[0032] Step 1.3. According to the intermediate variable, assign an exclusion coding value to each pixel. This value is adjusted according to the size of the intermediate variable, so that the pixels closer to the cell center have higher weights, and the responses between adjacent cells are suppressed. This method effectively solves the problem of overlapping of cells that are too close when generating density maps. The effect after preprocessing is as follows Figure 1 Shown in the three pictures on the right.
[0033] Step 2. Secondly, based on the U-Net model, the Multi-Scale Attention Module (MA Module) is used in the encoder part. The Multi-Scale Attention Module combines multi-scale feature fusion and soft attention mechanism to comprehensively capture the global and local features in the image. This module processes inputs of different scales and integrates information in the image in the encoder, which significantly improves the network's ability to capture features of cells of various sizes. It enables the model to more accurately identify and analyze cells of different sizes and shapes in microscope images, thereby improving the accuracy of segmentation and positioning. The Pixel Channel Fusion Module (PCF Module) is used between decoders. This module fuses pixel-level and channel-level attention mechanisms between the decoder layers to optimize the detail performance of the feature map. The overall network structure is as follows: Figure 2 shown.
[0034] As Step 2 in the present invention, the specific steps are:
[0035] Step2.1. We use the Multi-Scale Attention Module (MA Module) module to replace the traditional encoder. This module contains three parallel paths in the encoding stage. The first path consists of two consecutive 3×3 convolutions to extract basic convolution features. As the network depth increases, the demand for larger receptive fields increases, but using too large a receptive field in the early stage of the network has limited effect. Therefore, the design of this module in the encoding stage is more flexible. The second path of the first two encoding layers uses two 3×3 dilated convolutions with a dilation rate of 2, and the last two encoding layers use 3×3 dilated convolutions with a dilation rate of 4. Since large receptive field connections rely on the assistance of small receptive field connections, the output channels of the first 3×3 convolution branch are twice that of the second branch to enhance the efficiency of feature extraction and information integration. The third path extracts through compression and attention, and generates a global soft attention mask through upsampling. This attention mask highlights key features and suppresses unimportant information by weighting each position in the feature map element by element, thereby enhancing the model's attention to important areas. Finally, the module implements residual connections through 1×1 convolutions. This design improves the network's ability to capture multi-scale features and focus on key features. The module structure is as follows: Figure 3 shown.
[0036] Step 2.2. We inserted the Pixel Channel Fusion Module (PCFModule) between the encoders. This module first processes the input feature map through local average pooling (LAP) and global average pooling (GAP). LAP captures the features of a specific area, while GAP obtains the information of the entire feature map. These two pooling operations are processed in parallel, and each interacts across channels through one-dimensional convolution. Subsequently, the global pooling features are upsampled to the same size as the local pooling features. The two are multiplied by a learnable coefficient and added together. The merged features are upsampled to the original feature map size and multiplied with the original features. Finally, by adding a pixel attention layer. The channel features of each pixel are weighted so that the model can focus more accurately on important features in the image. This pixel-level attention processing not only improves the model's ability to capture local details, but also optimizes the information expression of the overall feature map, thereby improving segmentation accuracy. The module structure is as follows Figure 4 shown.
[0037] Step 3: In post-processing, a specific threshold is set first. According to the threshold, the network output density distribution map is converted into a discrete binary mask, and then the cell center position is determined by using distance transformation and watershed algorithm.
[0038] According to the above implementation process, you can combine Figure 2 , the working principle of the present invention is summarized as follows:
[0039] 1. Perform the preprocessing in Step 1 on the original point annotations of the pathological image and convert them into a density distribution map to optimize the learning effect of the network;
[0040] 2. In Figure 2 The image data is trained and predicted under the improved model. The improvements include using Multi-Scale Attention Module in the encoder part and Pixel Channel Fusion Module between decoders, covering the content of Step 2.
[0041] Thus, a model for cell segmentation of pathological images stained by IHC can be obtained to assist doctors in counting the number of Ki-67 positive cells, Ki-67 negative cells and tumor infiltrating lymphocytes, such as Figure 5 The above effect can help to more accurately judge the condition.
[0042] In order to further illustrate the experimental results of the present invention, the experimental results of the present invention are now shown:
[0043] First, the experimental environment of the present invention is: processor 12vCPU Intel(R)Xeon(R)Platinum 8255CCPU@2.50GHz, 43.0GB memory, graphics card NVDIA GeForce RTX3090, video memory 24G. Adam is used as optimizer, and the learning rate decay strategy of cosine annealing with hot restart is adopted. The learning rate is restarted every 30 epochs, and the minimum is reduced to 1e-5.
[0044] According to the experimental steps, the test was conducted on 700 IHC staining images with a resolution of 256×256. The average precision of the three types of cells was 0.7790, the recall rate was 0.8251, and the F1 value was 0.8014. The test evaluation indicators are shown in Table 1.
[0045] Table 1 Experimental results of the present invention
[0046] Network Model Classification Type Accuracy Recall F1 value Method of the present invention IHC staining images 0.7790 0.8251 0.8014
[0047] It can be seen that the method of the present invention performs well in the overall detection of the three types of cells on the IHC staining image, indicating that the method of the present invention can well assist doctors in quantitative analysis of the above biomarkers. Therefore, it can be concluded that the present invention has certain clinical significance.
[0048] The specific implementation modes of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above implementation modes, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.
Claims
1. A pathological image cell segmentation method based on U-Net model, characterized in that: These include: Step 1: First, pre-process the original point annotations and convert the original point annotations of the cell center into a density distribution map; Step 2. Based on the U-Net model, a multi-scale attention module is used in the encoder part. The multi-scale attention module combines multi-scale feature fusion and soft attention mechanism to fully capture the global and local features in the image. A pixel-channel fusion module is used between decoders. The pixel-channel fusion module fuses pixel-level and channel-level attention mechanisms between decoder layers to optimize the detail representation of feature maps. Step 3. Finally, in terms of post-processing, a threshold is set first, and the network output density distribution map is converted into a discrete binary mask according to the threshold, and then the cell center position is determined by using distance transform and watershed algorithm.
2. The pathological image cell segmentation method based on the U-Net model according to claim 1, characterized in that: In the Step 1, exclusion coding is used to convert the original point mark into a density distribution map. The specific steps in Step 1 are: Step 1.
1. For each pixel in the image, calculate its Euclidean distance to the nearest cell center and the second nearest cell center. These two distances are recorded as the nearest distance and the second nearest distance respectively. Step 1.
2. For each pixel, a new intermediate variable is calculated by combining its distance to the nearest cell center and the second nearest cell center. This intermediate variable reflects the distance weight of the pixel after considering the influence of neighboring cells. Step 1.
3. According to the intermediate variable, an exclusion coding value is assigned to each pixel. This value is adjusted according to the size of the intermediate variable, so that the pixels closer to the cell center have higher weights, and the responses between adjacent cells are suppressed.
3. The pathological image cell segmentation method based on the U-Net model according to claim 1, characterized in that: In the Step 2, the encoder and decoder parts of U-Net are modified. The specific steps in Step 2 are: Step 2.
1. Use a multi-scale attention module to replace the traditional encoder; the multi-scale attention module contains three parallel paths in the encoding stage; the first path consists of two consecutive 3×3 convolutions to extract basic convolutional features; the second path of the first two encoding layers uses two 3×3 dilated convolutions with a dilation rate of 2, and the last two encoding layers use 3×3 dilated convolutions with a dilation rate of 4; the output channel of the first 3×3 convolution branch is twice that of the second branch; the third path extracts through compression and attention, and generates a global soft attention mask through upsampling. This attention mask highlights key features and suppresses unimportant information by weighting each position in the feature map element by element; finally, residual connection is achieved through 1×1 convolution. Step 2.
2. A pixel channel fusion module is inserted between the encoders. The module first processes the input feature map through local average pooling LAP and global average pooling GAP. LAP captures the features of a specific area, while GAP obtains the information of the entire feature map. The two pooling operations are processed in parallel, and each interacts across channels through one-dimensional convolution. Subsequently, the global pooling features are upsampled to the same size as the local pooling features. The two are multiplied by a learnable coefficient and added together. The merged features are upsampled to the original feature map size and multiplied with the original features. Finally, by adding a pixel attention layer, the channel features of each pixel are weighted.
Citation Information
Patent Citations
Multi-feature cyclic convolution saliency target detection method based on attention mechanism
CN110648334A
MRI (Magnetic Resonance Imaging) brain tumor segmentation method based on multi-scale feature fusion of improved U-Net
CN117876399A
Fusion convolutional adaptive network skin lesion segmentation method
CN118072024A
Medical image segmentation method based on u-net
US20220309674A1
Cited By
Medical image hysteromyoma segmentation method
CN119672334A
Lithium battery cap welding mark defect segmentation method and device, electronic equipment and medium
CN120833330A
Lithium battery cap welding mark defect segmentation method and device, electronic equipment and medium
CN120833330B
Pathological section data feature description method based on self-supervised learning
CN121075572A
