A method and system for automatic prompting-based pathological image cell nucleus segmentation
Patent Information
- Application Number
- CN202610717622.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-05-22
- Publication Date
- 2026-08-18
AI Technical Summary
[0004]（1）SAM面向大尺度自然物体或器官设计，对病理图像中稀疏、微小、且高度聚集的细胞核目标适配性差，易产生漏检和过分割
[0033] (1) Fully automatic segmentation: Through the automatic prompting auxiliary segmentation network, the dependence on manual point and box prompts is completely eliminated, and batch and fully automatic processing of large-scale pathological slides can be realized, which has extremely high clinical practical value.
Smart Images
Figure CN122598166A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of medical image processing and artificial intelligence technology, and particularly relates to a method and system for segmenting cell nuclei in pathological images based on automatic prompting. Background Technology
[0002] Cell nuclear segmentation is a core foundational task in computational pathology, cancer grading, and disease diagnosis, and its segmentation accuracy directly affects the reliability and automation level of pathological analysis. Traditional fully supervised segmentation models, such as U-Net, U-Net++, and TransUNet, while performing excellently on specific datasets, heavily rely on large-scale, high-quality, finely labeled data. Their performance deteriorates sharply in generalized scenarios involving cross-modalities, cross-staining conditions, and cross-tissue types, making it difficult to meet the diverse practical needs of clinical practice.
[0003] In recent years, the Segment Anything Model (SAM), as a general-purpose visual foundation model, has demonstrated strong zero-shot generalization ability and interactive segmentation potential, and has been extended to include medical image segmentation versions such as MedSAM and SAM-Med2D. However, directly applying SAM or its medical variants to pathological cell nucleus segmentation has the following significant drawbacks:
[0004] (1) SAM is designed for large-scale natural objects or organs. It is poorly adapted to sparse, small and highly clustered cell nuclei in pathological images, and is prone to missed detection and oversegmentation.
[0005] (2) The segmentation accuracy is highly dependent on manual prompts (such as points and boxes). When the number of cell nuclei is large, the number of prompts increases linearly, resulting in extremely low clinical efficiency and making it impossible to achieve fully automated batch processing.
[0006] (3) Without any manual prompts, SAM is prone to random sampling or incorrect segmentation, and lacks stable and reliable automatic segmentation capabilities.
[0007] (4) Existing fine-tuning methods are not well adapted to the characteristics of SAM in the cell nucleus and fail to effectively utilize the unique morphology and edge prior knowledge of the cell nucleus.
[0008] Therefore, there is an urgent need for a dedicated method for segmenting pathological cell nuclei that can achieve high precision, high generalization, and high stability without relying on manual interaction. Summary of the Invention
[0009] The purpose of this invention is to provide a method and system for segmenting cell nuclei in pathological images based on automatic prompting, in order to solve the problems mentioned in the background art.
[0010] This invention is implemented as follows: a method for segmenting cell nuclei in pathological images based on automatic prompting, the method being built on the SAN (Segment Any Nuclei) framework, and includes the following steps:
[0011] Step 1: Construct a multimodal cell nucleus-specific dataset, which contains multiple pathological images covering various tissues and organs, staining methods, and imaging modalities;
[0012] Step 2: Preprocess the input pathological image, input the preprocessed image into the frozen weights and insert the learnable Adapter layer of the SAM (Segment Anything Model) image encoder to extract cell nucleus-specific discrimination features;
[0013] Step 3: The nucleus-specific discrimination features are processed by an automatic prompting-assisted segmentation network to automatically generate nucleus segmentation prompts;
[0014] Step 4: The encoder prompts the automatic prompts to be encoded, and the encoded result is sent to the mask decoder to output the cell nucleus instance segmentation mask;
[0015] Step 5: During the training phase, a sparse-dense cueing combination strategy is used for model training. This strategy includes: using dense cueing only in the last round of training and a random intermediate round, and using sparse cueing in the remaining rounds, and selecting the prediction result with the highest intersection-over-union (IoU) with the ground truth mask to calculate the loss.
[0016] Step 6: Perform model evaluation on the test set and zero-sample dataset, and output high-precision cell nucleus segmentation results.
[0017] A further technical solution is that the Adapter layer includes a channel dimension adaptive module and a spatial dimension adaptive module:
[0018] Channel dimension adaptive module: The feature map is compressed through global average pooling, and the channel weights are obtained through two layers of linear transformation and ReLU activation. The weights are then multiplied with the input feature map channel by channel.
[0019] Spatial dimension adaptive module: Reduces the resolution of feature maps by convolution downsampling, restores the size by deconvolution upsampling, and adds skip connections after each Adapter layer.
[0020] In a further technical solution, the automatic prompting-assisted segmentation network is built based on the Swing Transformer and uses three independent output decoding paths, which are used to perform: cell nucleus body segmentation, ordinary edge segmentation, and clustered edge segmentation respectively; an attention sharing mechanism is used between the decoding paths.
[0021] In a further technical solution, the dense cue is obtained by mapping the low-resolution feature map generated in the previous round of training to the interval [0,1], which is used to provide spatial prior for the current round of training.
[0022] A further technical solution is that the mask decoder retains the lightweight structure of SAM and is trained from scratch to adapt to the sparse and dispersed cell nucleus instance segmentation task.
[0023] A further technical solution is that the method simultaneously supports automatic prompts and manual optional prompts; when there is no human interaction, the system operates fully automatically based on automatic prompts; when corrections are needed, users can interactively intervene through point prompts or box prompts.
[0024] A further technical solution employs the Adam optimizer with an initial learning rate of 1×10⁻⁶. -4 Training and inference were completed on an NVIDIA RTX 4070Ti GPU.
[0025] Another objective of this invention is to provide an automatic prompting-based pathological image cell nucleus segmentation system, based on the above-described method, comprising:
[0026] Data preprocessing module: used for normalization, size unification and enhancement of multimodal pathological images;
[0027] Image encoder module: contains SAM ViT encoder and a learnable adapter layer inserted therein, used to extract cell nucleus discrimination features;
[0028] Auto-suggestion module: Includes a three-branch auxiliary segmentation network based on Swing Transformer for automatically generating segmentation suggestions;
[0029] The prompt encoding and mask decoding module is used to encode the automatic prompts and output the instance segmentation results;
[0030] Post-processing module: used for screening, denoising, and boundary optimization of cell nucleus instances;
[0031] Training optimization module: used to implement sparse-dense cue combination, loss calculation and parameter update.
[0032] The present invention provides a method and system for cell nucleus segmentation in pathological images based on automatic prompting, the beneficial effects of which are as follows:
[0033] (1) Fully automatic segmentation: Through the automatic prompting auxiliary segmentation network, the dependence on manual point and box prompts is completely eliminated, and batch and fully automatic processing of large-scale pathological slides can be realized, which has extremely high clinical practical value.
[0034] (2) Strong generalization ability: It is trained on a large-scale dataset covering multiple tissues, multiple stainings and multiple modalities, and achieves performance far exceeding existing methods (such as SAM, MedSAM, etc.) on zero-shot test sets such as CPM17, proving its excellent cross-domain generalization ability.
[0035] (3) High-efficiency fine-tuning: Only the lightweight Adapter layer and the custom mask decoder are trained, the main encoder is frozen, the amount of computation and the amount of data required are small, and the training efficiency is high.
[0036] (4) Leading segmentation accuracy: On multiple public datasets such as PanNuke and MoNuSeg, the Dice, IoU and other metrics are comprehensively superior to the fully supervised model and the SAM series medical variant model.
[0037] (5) User-friendly interaction: While achieving full automation, it retains the support interface for human prompts, allowing professional doctors to make interactive corrections when necessary, thus balancing the flexibility required for automation and high precision. Attached Figure Description
[0038] Figure 1 The diagram shows the overall architecture and core modules of a pathological image cell nucleus segmentation method based on automatic prompting, as provided in an embodiment of the present invention.
[0039] Figure 2 Examples of segmentation results from different networks;
[0040] Figure 3 A comparison of SAN with existing similar SAM methods;
[0041] Figure 4 An overview of data distribution organized by cell pattern and imaging modality. Detailed Implementation
[0042] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.
[0043] The specific implementation of the present invention will be described in detail below with reference to specific embodiments.
[0044] An embodiment of the present invention provides a method for cell nucleus segmentation in pathological images based on automatic prompting. The method is built on the SAN framework and includes the following steps:
[0045] Step 1: Construct a multimodal cell nucleus-specific dataset, which contains multiple pathological images covering various tissues and organs, staining methods, and imaging modalities;
[0046] Step 2: Preprocess the input pathological image, input the preprocessed image into the frozen weights and insert the learnable Adapter layer of the SAM (Segment Anything Model) image encoder to extract cell nucleus-specific discrimination features;
[0047] Step 3: The nucleus-specific discrimination features are processed by an automatic prompting-assisted segmentation network to automatically generate nucleus segmentation prompts;
[0048] Step 4: The encoder prompts the automatic prompts to be encoded, and the encoded result is sent to the mask decoder to output the cell nucleus instance segmentation mask;
[0049] Step 5: During the training phase, a sparse-dense cueing combination strategy is used for model training. This strategy includes: using dense cueing only in the last round of training and a random intermediate round, and using sparse cueing in the remaining rounds, and selecting the prediction result with the highest intersection-over-union (IoU) with the ground truth mask to calculate the loss.
[0050] Step 6: Perform model evaluation on the test set and zero-sample dataset, and output high-precision cell nucleus segmentation results.
[0051] Another embodiment of the present invention provides a pathological image cell nucleus segmentation system based on automatic prompting, which, based on the above method, includes:
[0052] Data preprocessing module: used for normalization, size unification and enhancement of multimodal pathological images;
[0053] Image encoder module: contains SAM ViT encoder and a learnable adapter layer inserted therein, used to extract cell nucleus discrimination features;
[0054] Auto-suggestion module: Includes a three-branch auxiliary segmentation network based on Swing Transformer for automatically generating segmentation suggestions;
[0055] The prompt encoding and mask decoding module is used to encode the automatic prompts and output the instance segmentation results;
[0056] Post-processing module: used for screening, denoising, and boundary optimization of cell nucleus instances;
[0057] Training optimization module: used to implement sparse-dense cue combination, loss calculation and parameter update.
[0058] The following verification is based on several specific embodiments.
[0059] Example 1: Overall implementation of the SAN framework;
[0060] The SAN framework of this invention is as follows: Figure 1 As shown, this embodiment describes the complete network architecture implementation of SAN (Segment Any Nuclei). SAN is built on the encoder-decoder architecture of SAM, with its core innovation being an image encoder with an adapter and an automatic prompting-assisted segmentation network. The network input is a pathological cell nucleus image, which sequentially undergoes image encoding, automatic prompt generation, prompt encoding, and mask decoding, ultimately outputting pixel-level cell nucleus instance segmentation results.
[0061] The image encoder employs a SAM pre-trained ViT-B architecture with weights frozen throughout. A learnable Adapter layer is inserted only within each Transformer module to efficiently adapt to the cell nucleus segmentation domain without compromising general features. The auto-suggestion auxiliary segmentation network uses the Swing Transformer as its backbone, outputting cell nucleus region, ordinary edge, and cluster edge suggestions in parallel, eliminating the need for manual point / box input. The suggestion encoder receives and encodes the auto-suggestions, while the mask decoder uses a lightweight structure and is trained from scratch, specifically adapted for sparse and dispersed cell nucleus instance segmentation.
[0062] The SAN architecture maintains end-to-end inference, automatically completing segmentation without human intervention, while retaining a human prompt interface for interactive correction. The overall computational complexity of the model is controllable, significantly improving the segmentation accuracy and generalization ability of pathological cell nuclei while maintaining the advantages of the SAM architecture.
[0063] Example 2: Detailed implementation of an image encoder with an adapter;
[0064] This embodiment describes the detailed implementation of the image encoder and Adapter layer. SAN retains the original SAM image encoder structure and freezes the weights. A dual-branch Adapter module is inserted after each Transformer layer to perform adaptive changes in the channel dimension and spatial dimension, respectively.
[0065] (1) Channel dimension adaptive:
[0066] Global average pooling is performed on the input feature map to compress the spatial dimension to C×1×1 (C refers to the number of channels in the image) to obtain the channel description vector; it is then compressed and restored through two linear layers (compression ratio 1 / 4), and channel weights are generated by ReLU activation; the weights are multiplied with the original feature map channel by channel to enhance the channel response related to the cell nucleus.
[0067] (2) Spatial dimension adaptation:
[0068] The feature map is downsampled by convolutional layers to reduce the spatial scale and thus reduce computation; then it is upsampled back to the original size by deconvolutional layers to restore spatial details; skip connections are added inside the adapter to ensure information flow and stable gradient backpropagation.
[0069] After adapter adaptation, the feature map output by the encoder is more closely aligned with the cell nucleus morphology, edge and texture characteristics, providing a high-quality feature foundation for subsequent prompt generation and mask prediction.
[0070] Example 3: Detailed implementation of the auto-suggestion auxiliary segmentation network;
[0071] This embodiment describes the structure and implementation of an automatic suggestion-assisted segmentation network. The network is based on the TransNuSeg architecture, uses the Swing Transformer as the feature extraction backbone, employs a three-branch parallel decoding structure, and introduces an attention-sharing mechanism.
[0072] (1) First branch: used for segmentation of the main region of the cell nucleus, outputting a coarse segmentation mask as the basis for core prompts.
[0073] (2) Second branch: used for ordinary edge segmentation of cell nuclei, accurately locates cell nucleus outline, and improves boundary accuracy.
[0074] (3) Third branch: used for segmenting the edges of adherent / clustered cell nuclei, solving the problem of difficult segmentation of overlapping cell nuclei.
[0075] The three decoding paths share a portion of the self-attention head, enabling feature reuse, reducing the number of parameters, and improving output consistency. The network ultimately outputs a set of automatic prompts that can be directly used by the SAM prompt encoder, replacing traditional manual clicking / box selection, achieving completely interactive segmentation. Segmentation results from different networks are shown below. Figure 2 As shown.
[0076] Example 4: Implementation of a sparse-dense cue hybrid training strategy;
[0077] This embodiment describes the training process and optimization strategy of SAN. To balance segmentation accuracy and computational overhead, SAN employs a sparse-dense hybrid cueing training mechanism:
[0078] (1) In each round of training, the prediction result with the highest IoU with the ground truth mask is selected to calculate the loss, ensuring that the gradient direction is optimal.
[0079] (2) Normalize the low-resolution feature map generated in the previous round to [0,1] to provide spatial prior as dense cue, which helps the model learn fine-grained cell nuclear structure.
[0080] (3) To reduce computational cost, dense hints are used only in the last iteration and a random intermediate iteration, while sparse hints are used in the remaining iterations, which greatly reduces computational cost.
[0081] The training environment uses an NVIDIA RTX 4070 Ti GPU, with Adam as the optimizer and an initial learning rate of 1×10⁻⁶. -4 The training data consisted of 9244 integrated multimodal pathological cell nuclear images, divided into training and testing sets at an 8:2 ratio according to patient grade. The data distribution of cell patterns and imaging modalities is shown below. Figure 4 As shown.
[0082] Example 5: Experimental Verification and Result Analysis
[0083] This embodiment presents quantitative and qualitative experimental results of SAN on multiple datasets to verify the effectiveness of the invention. A comparison of SAN with existing similar SAM methods is shown below. Figure 3 As shown.
[0084] (1) Experimental setup:
[0085] A SAN was implemented in the PyTorch framework, and the comparison methods included U-Net, U-Net++, TransUNet, SwinUNet, TransNuSeg, SAM, MedSAM, and SAM-Med2D. Evaluation metrics included Dice, IoU, and pixel accuracy (Acc).
[0086] (2) Quantitative comparison results:
[0087] As shown in Table 1, SAN achieves best or near-best performance on all seven datasets. SAN achieves 86.83% Dice on the PanNuke dataset, 72.9% Dice on MoNuSeg, 25.18% Dice on CoNSeP, 32.91% Dice on GlandSegData, 83.69% Dice on Histology, 81.9% Dice on Fluorescence, and 95.61% Dice on Cytology. In most datasets, SAN consistently achieves the highest Dice score, outperforming CNN-based networks and hybrid networks such as TransNuSeg and previous SAM-based methods.
[0088] Table 1. Quantitative comparison of cell nucleus segmentation performance based on Dice score.
[0089]
[0090] As shown in Table 2, segmentation models based on fully supervised U-Net were not included in this comparison due to the lack of zero-shot capability. SAN demonstrated the best performance on the CPM17 dataset, with improvements of 10.72% in Dice score, 6.21% in accuracy, and 9.97% in IoU compared to the best-performing comparison method.
[0091] Table 2 Quantitative Comparison of Zero-Sample Generalization Performance
[0092]
[0093] (3) Computational efficiency analysis:
[0094] Table 3 compares the computational complexity of the models. SAN (without AP) strikes a balance between model complexity and performance, with a comparable number of parameters and FLOPs to SAM-Med2D, while improving segmentation accuracy. Including the Auto-Prompt (AP) assisted segmentation network in SAN (with AP) introduces additional parameters and computational overhead, but enables prompt-free inference and enhances the model's adaptability to various cell kernel segmentation tasks. Despite the increased complexity, SAN (with AP) maintains a reasonable inference time compared to other SAM-based methods, making it a practical choice for real-world applications.
[0095] Table 3 Comparison of Model Parameters, Computational Cost, and Inference Time
[0096]
[0097] It should be noted that the SAN framework referenced in this invention has been open-sourced on GitHub (https: / / github.com / Mrliujunwen / SAN), and the training code, pre-trained weights, and evaluation scripts are all publicly available, which facilitates reproduction and promotes subsequent research.
[0098] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A method for cell nucleus segmentation in pathological images based on automatic prompting, characterized in that, The method is built on a SAN framework and includes the following steps: Step 1: Construct a multimodal cell nucleus-specific dataset, which contains multiple pathological images covering various tissues and organs, staining methods, and imaging modalities; Step 2: Preprocess the input pathological image, input the preprocessed image into the frozen weights and insert the SAM image encoder with the learnable Adapter layer, and extract the nucleus-specific discrimination features; Step 3: The nucleus-specific discrimination features are processed by an automatic prompting-assisted segmentation network to automatically generate nucleus segmentation prompts; Step 4: The encoder prompts the automatic prompts to be encoded, and the encoded result is sent to the mask decoder to output the cell nucleus instance segmentation mask; Step 5: During the training phase, a sparse-dense cueing combination strategy is used for model training. This strategy includes: using dense cueing only in the last round of training and a random intermediate round, and using sparse cueing in the remaining rounds, and selecting the prediction result with the highest intersection-union comparison with the ground truth mask to calculate the loss. Step 6: Perform model evaluation on the test set and zero-sample dataset, and output high-precision cell nucleus segmentation results.
2. The method for cell nucleus segmentation in pathological images based on automatic prompting according to claim 1, characterized in that, The Adapter layer includes a channel dimension adaptive module and a spatial dimension adaptive module: Channel dimension adaptive module: The feature map is compressed through global average pooling, and the channel weights are obtained through two layers of linear transformation and ReLU activation. The weights are then multiplied with the input feature map channel by channel. Spatial dimension adaptive module: Reduces the resolution of feature maps by convolution downsampling, restores the size by deconvolution upsampling, and adds skip connections after each Adapter layer.
3. The method for cell nucleus segmentation in pathological images based on automatic prompting according to claim 2, characterized in that, The automatic prompting-assisted segmentation network is built based on the Swing Transformer and uses three independent output decoding paths, which are used to perform: cell nucleus body segmentation, ordinary edge segmentation, and clustered edge segmentation respectively; an attention sharing mechanism is used between the decoding paths.
4. The method for cell nucleus segmentation in pathological images based on automatic prompting according to claim 3, characterized in that, The dense cue is obtained by mapping the low-resolution feature map generated in the previous training round to the interval [0,1], and is used to provide spatial prior for the current training round.
5. The method for cell nucleus segmentation in pathological images based on automatic prompting according to claim 4, characterized in that, The mask decoder retains the lightweight structure of SAM and is trained from scratch to adapt to the sparsely distributed cell nucleus instance segmentation task.
6. The method for cell nucleus segmentation in pathological images based on automatic prompting according to claim 1, characterized in that, The method supports both automatic prompts and manually selectable prompts. When there is no human interaction, the system runs fully automatically based on automatic prompts; when corrections are needed, users can intervene interactively by clicking on prompts or using prompts in boxes.
7. A system for segmenting cell nuclei in pathological images based on automatic prompting, comprising the method for segmenting cell nuclei in pathological images based on automatic prompting as described in any one of claims 1-6, characterized in that, include: Data preprocessing module: used for normalization, size unification and enhancement of multimodal pathological images; Image encoder module: contains SAM ViT encoder and a learnable adapter layer inserted therein, used to extract cell nucleus discrimination features; Auto-suggestion module: Includes a three-branch auxiliary segmentation network based on Swing Transformer for automatically generating segmentation suggestions; The prompt encoding and mask decoding module is used to encode the automatic prompts and output the instance segmentation results; Post-processing module: used for screening, denoising, and boundary optimization of cell nucleus instances; Training optimization module: used to implement sparse-dense cue combination, loss calculation and parameter update.