Cell nucleus immunohistochemical image segmentation method based on SAM2 and HIPT models

Abstract

The invention relates to a nuclear immunohistochemical image segmentation method based on SAM2 and HIPT models. For the data of breast cancer, bladder cancer and lung cancer slices, a preliminary segmentation mask is generated by using an SAM2 model, then the preliminary segmentation mask is input into an HIPT model, and a segmentation result is further optimized through multi-modal feature learning. In order to effectively fuse the tasks of the two stages, a joint loss function is designed, and global optimization is carried out by combining segmentation loss and multi-modal feature loss, so that the model not only pays attention to segmentation precision, but also can capture multi-modal information. According to the method, efficient cell nucleus segmentation is realized, the performance in the aspects of segmentation precision, robustness and the like is good, and the purpose of effectively carrying out cell nucleus segmentation on breast cancer, bladder cancer and lung cancer images is achieved.

Description

Technical Field

[0001] The present invention provides a cell nucleus immunohistochemistry image segmentation method based on SAM2 and HIPT models, belonging to the field of deep learning for medical image segmentation. Background Art

[0002] Cancer is a growing health problem worldwide, with millions of new cases each year, and cancer is the second leading cause of death after cardiovascular disease. Despite significant advances in non-invasive imaging technologies (such as CT and MRI) in recent years, biopsy of tissue samples and microscopic analysis remain the "gold standard" method for diagnosing cancer and evaluating treatment efficacy. Pathologists can draw conclusions about potential treatment options by observing cells and their distribution in tissue sections. For example, tumor-infiltrating lymphocytes or inflammatory cells in the tumor microenvironment can provide key information for diagnosis. However, traditional cell analysis relies on manual annotation, which is not only time-consuming and labor-intensive, but also susceptible to intra- and inter-observer variability. With the development of digital pathology, especially the advancement of high-throughput scanner technology, whole slide images (WSI) of pathology slides can be digitized and used for computer vision (CV) algorithm analysis.

[0003] With the rapid development of deep learning technology, it has shown great potential and advantages in the field of biomedical image processing. Especially in the task of cell nucleus segmentation, the large model has significantly improved the accuracy and efficiency of segmentation through its powerful feature extraction and pattern recognition capabilities. Traditional cell nucleus segmentation methods rely on manually designed features and rules, which are difficult to cope with complex and changeable cell morphology. The large model based on the convolutional neural network (CNN) and transformer architecture can automatically learn and extract multi-level features in the image, thereby achieving accurate segmentation of the cell nucleus.

[0004] Even so, the complex background, uneven staining and high-noise areas in immunohistochemistry (IHC) images still pose a huge challenge to existing segmentation methods. Traditional segmentation methods are usually unable to effectively handle these complex situations when facing IHC images, resulting in inaccurate segmentation results or being easily interfered by noise. In order to automatically and reliably segment cells and extract cell features in WSI, it is urgent to develop new methods. The present invention proposes a new cell segmentation method, SAM2HIPT, which combines the preliminary segmentation capability of the Segment Anything Model 2 (SAM2) model and the multimodal feature learning capability of the Hierarchical Image Pyramid Transformer (HIPT) model, thereby achieving accurate segmentation of cells in IHC images. The SAM2 model provides a basis for subsequent processing by quickly generating preliminary segmentation masks; while the HIPT model further optimizes the segmentation results through multimodal feature learning, thereby improving the accuracy and robustness of segmentation. The present invention focuses on the cell nucleus segmentation of immunohistochemistry images of breast cancer, bladder cancer and lung cancer in the field of deep learning. Summary of the invention

[0005] In view of the above defects or deficiencies in the prior art, and in view of the key challenges in the current research on cell nucleus immunohistochemical image segmentation, the present invention proposes a cell nucleus immunohistochemical image segmentation method based on SAM2 and HIPT models. This method combines the preliminary segmentation capability of the SAM2 model and the multimodal feature learning of the HIPT model, designs an innovative joint loss function, realizes the accurate segmentation of cell nuclei in immunohistochemical (IHC) images, and effectively solves the problem that traditional methods are difficult to handle under complex morphologies. Through this method, a higher accuracy of cell nucleus segmentation is achieved in complex medical image tasks.

[0006] The technical solution of the present invention is: a method for segmenting cell nucleus immunohistochemical images based on SAM2 and HIPT models, comprising the following steps:

[0007] Step 1: Use IHC image data and the corresponding cell nucleus segmentation mask image as training data, perform preprocessing operations, and generate an image data set suitable for the input model;

[0008] Step 2: Perform preliminary cell nucleus segmentation using the SAM2 model to generate preliminary segmentation results;

[0009] Step 3: Use the HIPT model to perform multimodal feature learning based on the preliminary segmentation to optimize the final segmentation result of the cell nucleus; the HIPT model can further refine the segmentation result and improve the accuracy and consistency of the final segmentation by effectively capturing multi-level features;

[0010] Step 4. Combined with the adaptively designed joint loss function, the SAM2 and HIPT models are jointly optimized based on the segmentation loss and multimodal feature loss to improve the accuracy and robustness of the segmentation results.

[0011] As a further solution of the present invention, in the Step 1, when preprocessing the training data, the IHC image and the corresponding cell nucleus segmentation mask image are cropped to form image data of a size of 512×512, and a variety of data enhancement methods such as random rotation, scaling, and color perturbation are used in training to improve the generalization ability of the model.

[0012] In the preprocessing stage of Step 1, a specific image preprocessing process is formulated for different morphological features in immunohistochemistry images. Through cropping and splicing operations, the IHC images and corresponding cell nucleus mask images are properly processed to meet the input requirements of the model, and the diversity of training data and the generalization ability of the model are improved through data enhancement operations.

[0013] As a further solution of the present invention, in Step 2, the SAM2 model is responsible for performing preliminary cell nucleus segmentation on the IHC image data and generating a preliminary cell nucleus segmentation mask. The SAM2 model is based on a self-attention mechanism to effectively capture global and local features in the image, thereby generating an accurate preliminary segmentation mask.

[0014] As a further solution of the present invention, in Step 3, the HIPT model is used to perform further multimodal feature learning based on the preliminary segmentation mask generated by SAM2, and utilizes its sensitivity to features at different levels to improve the detail expression of the segmentation results and the ability to understand complex structures. The HIPT model performs refined segmentation by fusing high-level and low-level information of the image.

[0015] As a further solution of the present invention, in Step 4, the joint loss function includes segmentation loss and multimodal feature loss. The segmentation loss is used to evaluate the difference between the generated segmentation result and the true segmentation mask. The segmentation loss ensures that the model can accurately segment the cell nucleus region. The multimodal feature loss is used to optimize the semantic consistency of the segmentation result, ensuring the consistency of the segmentation result under different modalities, with higher accuracy and reliability. The joint loss function takes into account the segmentation loss and the multimodal feature loss at the same time, so that the model can balance the two losses during the optimization process, thereby improving the reliability of the segmentation result and the ability to interpret biomarkers.

[0016] As a further solution of the present invention, in the Step 4, a phased training strategy is used in the joint optimization training process of the SAM2 and HIPT models. First, the SAM2 model is trained independently, and after the initial segmentation, the result is input into the HIPT model for multimodal feature optimization to ensure the collaborative optimization effect of the model.

[0017] The beneficial effects of the present invention are:

[0018] 1. Innovative application of multimodal feature learning: By combining the SAM2 model and the HIPT model, the present invention achieves accurate segmentation of complex morphologies of cell nuclei, especially in IHC images such as breast cancer, bladder cancer and lung cancer, showing excellent segmentation accuracy and robustness;

[0019] 2. Innovative joint loss function design: By combining segmentation loss and multimodal feature loss, the model can maintain segmentation accuracy and semantic consistency during the optimization process, significantly improving the accuracy and interpretability of segmentation results, and is suitable for complex medical image segmentation tasks;

[0020] 3. Wide application potential: The method of the present invention can effectively process IHC image segmentation of various pathological types such as breast cancer, bladder cancer and lung cancer, providing more accurate cell nucleus segmentation results for pathological diagnosis and biomarker analysis, thereby enhancing the support for pathological diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 The left image is an IHC image, the middle image is an H&E image, and the right image is a segmentation mask image;

[0022] Figure 2 Schematic diagram of images of different diseases, from top to bottom are bladder cancer, lung cancer and breast cancer, from left to right are IHC images, true segmentation mask images, mask images in the preliminary segmentation stage and mask images of the final segmentation results;

[0023] Figure 3 It is a joint loss function optimization framework diagram of the present invention;

[0024] Figure 4 It is the novel SAM2 and HIPT method framework of the present invention;

[0025] Figure 5 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0026] Example 1: Figure 1-Figure 5 As shown, the cell nucleus immunohistochemistry image segmentation method based on SAM2 and HIPT models includes the following:

[0027] Step 1: Use IHC image data and the corresponding cell nucleus segmentation mask image as training data for preprocessing;

[0028] First, the IHC (immunohistochemistry) images and the corresponding cell nucleus segmentation mask images are used as training data for preprocessing. Specifically, the IHC images and the corresponding cell nucleus segmentation mask images need to be cropped, normalized, and other operations are performed to form image data of size 512×512 to ensure that the input image meets the model requirements. For data enhancement, rotation, translation, scaling, flipping, noise, and other methods are used to expand the training data set to improve the generalization ability of the model. This step helps to improve the quality of the image, ensure the diversity of the input data, and increase the model's adaptability to complex cell nucleus morphology.

[0029] Step 2: Perform preliminary cell nucleus segmentation using the SAM2 model to generate preliminary segmentation results;

[0030] Next, the Segment Anything Model (SAM2) is used to perform preliminary nucleus segmentation on the preprocessed IHC images to generate preliminary segmentation results. The SAM2 model is based on the self-attention mechanism to effectively capture the global and local features in the image, thereby generating an accurate preliminary segmentation mask; the SAM2 model can quickly and effectively generate preliminary nucleus segmentation results, and use its powerful global and local feature extraction capabilities to effectively process complex nucleus morphology. This step outputs a preliminary segmentation mask as input for subsequent multimodal feature learning.

[0031] Step 3: Use the HIPT model to perform multimodal feature learning based on the preliminary segmentation to optimize the final segmentation result of the cell nucleus;

[0032] Based on the preliminary segmentation of SAM2, the HIPT (Hierarchical Image Patch Transformer) model is used to further process the segmentation results; the HIPT model is used to perform further multimodal feature learning based on the preliminary segmentation mask generated by SAM2. The HIPT model refines the segmentation by fusing the high-level and low-level information of the image. The HIPT model can capture more detailed multimodal information in the image through a hierarchical self-attention mechanism, thereby improving the accuracy of the segmentation results. By taking the IHC image, the preliminary segmentation results and their local features as input, the HIPT model can finely segment the details of the cell nucleus and provide more biologically meaningful segmentation results.

[0033] Step 4: Combine the adaptively designed joint loss function to jointly optimize the SAM2 and HIPT models based on the segmentation loss and multimodal feature loss to improve the accuracy and robustness of the segmentation results.

[0034] In order to improve the overall performance of the model, the present invention designs a joint loss function that combines segmentation loss with multimodal feature loss; segmentation loss is used to evaluate the difference between the generated segmentation result and the true segmentation mask, and multimodal feature loss is used to optimize the semantic consistency of the segmentation result to ensure the consistency of the segmentation result under different modalities. Specifically, the segmentation loss optimizes the segmentation result by calculating the Dice Loss between the generated segmentation mask and the true segmentation mask; the multimodal feature loss optimizes the model's understanding of multimodal information by measuring the similarity between the feature representation generated by the IHC image and the HIPT model. This joint loss function design ensures that the model can not only accurately segment the cell nucleus, but also effectively utilize multimodal information to improve segmentation accuracy. In the Step 4, in the joint optimization training process of the SAM2 and HIPT models, a phased training strategy is used. First, the SAM2 model is trained independently, and after the preliminary segmentation is completed, the result is input into the HIPT model for multimodal feature optimization to ensure the collaborative optimization effect of the model.

[0035] Step 5: Final result generation

[0036] Finally, the SAM2 and HIPT models were trained iteratively for multiple times to generate the final nucleus segmentation results. Combining the global feature extraction capability of SAM2 and the detail feature processing capability of HIPT, this method can perform high-precision segmentation of positive and negative nuclei in immunohistochemistry images, thus improving the application value of the model in clinical pathology analysis.

[0037] This embodiment effectively improves the segmentation effect of complex cell nuclear morphology in immunohistochemical images and verifies the advantages of the combined application of SAM2 and HIPT models.

[0038] According to the above implementation process, you can combine Figure 4 , the working principle of the present invention is summarized as follows:

[0039] 1. Preprocessing is done by cropping, normalizing, and data augmentation of IHC images and corresponding nucleus segmentation mask images to improve the quality and diversity of input data and ensure that the model can handle complex nucleus morphology.

[0040] 2. Use Segment Anything Model (SAM2) for preliminary segmentation. The SAM2 model uses its powerful global feature extraction capabilities to quickly generate preliminary segmentation results for the cell nucleus. This result captures important morphological features in the image and lays the foundation for subsequent processing.

[0041] 3. Input the preliminary segmentation results generated by the SAM2 model into the Hierarchical Image Patch Transformer (HIPT) model. The HIPT model uses a multi-level self-attention mechanism to further refine the segmentation results by analyzing local and global features layer by layer. This method can effectively handle complex textures and morphologies in images and improve segmentation accuracy.

[0042] 4. In order to ensure the overall performance of the model, the present invention designs a joint loss function, combining segmentation loss and multimodal feature loss to optimize the model. The accuracy of the segmentation results is evaluated by Dice Loss, and the feature loss is used to improve the ability to capture IHC image features, ensuring the precision and biological significance of the segmentation results.

[0043] 5. Through multiple iterations of training, the combination of SAM2 and HIPT models can generate highly accurate nucleus segmentation results. This method can not only accurately segment positive and negative nuclei, but also show excellent adaptability in complex immunohistochemical images, providing reliable data support for pathological analysis and clinical applications.

[0044] Thus, an effective segmentation model for cell nucleus segmentation of IHC images can be obtained, which can assist doctors to accurately identify and segment cell nuclei, thereby helping to determine the type, grade and prognosis of the tumor.

[0045] In order to further illustrate the experimental results of the present invention, the experimental results of the present invention are now shown:

[0046] First, in the present invention, the experiment was run on a high-performance Linux server equipped with an Intel Core i9-13900KF processor, 62GB of memory, and two NVIDIA GeForce RTX 3090 GPUs (24GB of video memory each, a total of 48GB). The implementation of all models is based on the Python environment, and the PyTorch framework is used to develop and train the model. During the model training process, we used a phased training strategy. In the first stage, the SAM2 model was trained independently as a preliminary segmenter, mainly for the preliminary segmentation of cell nuclei. In order to improve the training efficiency and accuracy of the model, we used a transfer learning method. Specifically, we introduced weights pre-trained on a large-scale natural image dataset to speed up the convergence of the model and improve the segmentation performance. In the second stage, the output of the trained SAM2 model was fixed, and the preliminary segmentation results generated by it were used as the input of the HIPT model for multimodal feature learning and optimization. In this process, the HIPT model further captures more subtle contextual and structural features in the image through the Vision Transformer architecture, especially learning the interaction and fusion of different modal features through the global attention mechanism.

[0047] According to the experimental steps, tests were conducted on breast cancer, bladder cancer and lung cancer, with 402 images for breast cancer and 598 images for bladder cancer and lung cancer. They were carefully compared with four current advanced methods. The experimental results showed that the overall segmentation performance was excellent, with Recall scores of 88% and 89% respectively, and PixAcc scores of 85% and 83% respectively. The specific experimental evaluation indicators are shown in Table 1 and Table 2:

[0048] Table 1 Experimental results of the present invention on breast cancer data

[0049] Network Model Dice coefficient Accuracy Recall F1 value Intersection and Union Pixel Accuracy The present invention 0.86 0.83 0.88 0.84 0.84 0.85 DeepLIIF 0.48 0.71 0.39 0.48 0.33 0.94 HoVerNet 0.68 0.62 0.42 0.44 0.50 0.84 StarDist 0.59 0.49 0.38 0.42 0.50 0.77 Cellpose 0.72 0.63 0.49 0.55 0.50 0.88

[0050] Table 2 Experimental results of the present invention on bladder cancer and lung cancer data

[0051] Network Model Dice coefficient Accuracy Recall F1 value Intersection and Union Pixel Accuracy The present invention 0.85 0.86 0.89 0.87 0.84 0.83 DeepLIIF 0.45 0.67 0.36 0.45 0.30 0.90 HoVerNet 0.57 0.64 0.23 0.33 0.50 0.70 StarDist 0.64 0.54 0.45 0.49 0.50 0.83 Cellpose 0.57 0.58 0.33 0.42 0.50 0.87

[0052] It can be seen that, based on the comparison of the experimental results in Table 1 and Table 2, the method of the present invention is closest to the actual situation among many advanced methods and has achieved relatively ideal results. The present invention has certain clinical significance.

[0053] The specific implementation modes of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above implementation modes, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. A method for segmenting cell nucleus immunohistochemical images based on SAM2 and HIPT models, characterized in that: These include: Step 1: Use IHC image data and the corresponding cell nucleus segmentation mask image as training data for preprocessing; Step 2: Perform preliminary cell nucleus segmentation using the SAM2 model to generate preliminary segmentation results; Step 3: Use the HIPT model to perform multimodal feature learning based on the preliminary segmentation to optimize the final segmentation result of the cell nucleus; Step 4. Combined with the adaptively designed joint loss function, the SAM2 and HIPT models are jointly optimized based on the segmentation loss and multimodal feature loss to improve the accuracy and robustness of the segmentation results.

2. The method for nuclear immunohistochemical image segmentation based on SAM2 and HIPT models according to claim 1, characterized in that: In the Step 1, the training data is preprocessed by cropping the IHC image and the corresponding cell nucleus segmentation mask image to form image data of a size of 512×512, and multiple data enhancement methods such as random rotation, scaling, and color perturbation are used in training.

3. The method for nuclear immunohistochemistry image segmentation based on SAM2 and HIPT models according to claim 1, characterized in that: In the Step 2, the SAM2 model is responsible for performing preliminary cell nucleus segmentation on the IHC image data. The SAM2 model is based on a self-attention mechanism to effectively capture global and local features in the image, thereby generating an accurate preliminary segmentation mask.

4. The method for nuclear immunohistochemical image segmentation based on SAM2 and HIPT models according to claim 1, characterized in that: In the Step 3, the HIPT model is used to perform further multimodal feature learning based on the preliminary segmentation mask generated by SAM2. The HIPT model performs refined segmentation by fusing high-level and low-level information of the image.

5. The method for nuclear immunohistochemical image segmentation based on SAM2 and HIPT models according to claim 1, characterized in that: In the Step 4, the joint loss function includes segmentation loss and multimodal feature loss. The segmentation loss is used to evaluate the difference between the generated segmentation result and the true segmentation mask, and the multimodal feature loss is used to optimize the semantic consistency of the segmentation result to ensure the consistency of the segmentation result under different modes.

6. The method for nuclear immunohistochemistry image segmentation based on SAM2 and HIPT models according to claim 1, characterized in that: In the Step 4, a phased training strategy is used in the joint optimization training process of the SAM2 and HIPT models. First, the SAM2 model is trained independently, and after the initial segmentation, the result is input into the HIPT model for multimodal feature optimization to ensure the collaborative optimization effect of the model.

Images