Cell nucleus segmentation method based on large model guided cascade encoding segmentation network

By combining a cascaded coding segmentation network with a fine-tuned SAM encoder and a multi-level CNN encoder, the problem of insufficient local-global modeling in cell nucleus segmentation is solved, achieving efficient and accurate cell nucleus segmentation, which is suitable for medical image segmentation tasks.

CN118366153BActive Publication Date: 2024-11-12TIANJIN UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410587951.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-05-13
Publication Date
2024-11-12
Estimated Expiration
2044-05-13

AI Technical Summary

Technical Problem

Existing technologies for cell nucleus segmentation suffer from insufficient local-global modeling capabilities and excessively large training parameters, making it difficult to achieve efficient and accurate cell nucleus segmentation, especially under conditions of small datasets.

Method used

A cascaded encoding segmentation network based on a large model is adopted, which combines a fine-tuned SAM encoder and a multi-level CNN encoder. Through a multi-stage feature fusion module and a decoder module, the local-global information interaction is enhanced, the number of training parameters is reduced, and accurate segmentation of cell nuclei is achieved.

Benefits of technology

Without increasing the number of training parameters, it improves the accuracy and efficiency of cell nucleus segmentation, and can achieve high-quality cell nucleus segmentation under conditions of limited data.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118366153B_ABST
    Figure CN118366153B_ABST
Patent Text Reader

Abstract

The present invention discloses a cell nucleus segmentation method based on a cascade coding segmentation network guided by a large model. The method is implemented based on a segmentation system, wherein the segmentation system comprises an image acquisition module and a cascade encoder segmentation network. The original cell image is acquired and enhanced. A guided feature map with global information semantic features is extracted by using a fine-tuned SAM encoder. A feature map with local information semantic features is extracted step by step by using each level of CNN encoder, and the guided feature map is fused with the feature map extracted by each level of CNN encoder by using a first-stage feature fusion module for feature extraction by the next level of CNN encoder. The image resolution is restored layer by layer by decoders at each level to obtain a final segmentation result map with the same size and number of channels as the input. The method can effectively enhance the ability of the model to capture local-non-local information without introducing a large number of training parameters, thereby realizing accurate segmentation of the cell nucleus, and has broad application prospects.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a cell nucleus segmentation method based on a cascade coding segmentation network guided by a large model. Background Art

[0002] Pathological analysis of tissue samples is one of the key steps in diagnosing and treating diseases such as cancer. Cell nucleus segmentation involves identifying and segmenting cell nuclei from cell images, which can help pathologists quickly and accurately evaluate cell morphology, density and other characteristics, which is crucial for understanding cell behavior, disease diagnosis and biomedical research. However, cell nucleus segmentation is usually difficult due to uneven cell staining and different and possibly overlapping cell nuclei. Secondly, the ultra-high resolution of pathological images and the high density of cell nuclei make the pixel-level standard cost extremely high. Therefore, an accurate, fast and automatic cell nucleus segmentation method that does not require a large amount of training data is one of the important research goals in the field of medical image segmentation.

[0003] Before deep learning became mainstream, cell nucleus segmentation mainly relied on traditional image processing techniques such as thresholding, edge detection, morphological operations, etc. These methods usually require complex preprocessing and parameter adjustment, and are very sensitive to image quality and changes, and are easily affected by noise and image damage. In recent years, great progress has been made in the foundation of medical image processing based on deep learning, especially the application of convolutional neural networks (CNNs), which can automatically learn complex feature representations from training data, can handle highly variable biological morphologies and complex backgrounds, and overcome many limitations of traditional methods. The most mainstream network at present is the U-Net network structure composed of encoders and decoders, and its variant networks. Due to its symmetrical encoding and decoding structure and jump connections, it can effectively extract detailed features from cell images, especially in the boundary processing of cell structures. Although the former performs well in most cases, it shows limitations in explicitly modeling long-range dependencies due to the inherent locality of convolution operations. In order to solve the defects of CNN in global perception, researchers have proposed a structure that combines CNN and Transformer. For example, the TransUNet network effectively combines the long-distance dependency ability of Transformer and the local feature extraction ability of CNN through advanced feature fusion strategies, optimizes the information flow and CoTr network, and effectively utilizes the efficient feature extraction ability of convolutional networks and the global context modeling ability of Transformer by fusing deep separable convolution and Transformer structures. However, this structure that combines CNN and Transformer introduces a large number of training parameters due to the introduction of Transformer, which places high requirements on the amount of training data. However, the cost of obtaining medical image data such as labeled trainable cell nucleus segmentation is quite high, which makes the data size of most medical image segmentation data small. In this case, how to obtain local-global modeling capabilities without introducing a large number of training parameters has become our research goal.

[0004] Since the emergence of large models such as SegmentAnything, the encoders of these models trained on extremely large amounts of data have super strong feature extraction capabilities. A bunch of basic large models fine-tuned based on medical domain knowledge are directly applied to the medical field for downstream tasks such as medical image classification and segmentation. The most prominent one is MedicalSAMAdapter, or SAM adapter, which enables large models to obtain the ability to perform segmentation tasks in the field of medical images by fine-tuning SAM. However, since the overall size of the model is already very large, the amount of training parameters introduced by the overall fine-tuning of SAM is still very large. Therefore, it is of great research significance and application value to study how to enable the model to have a strong feature extraction capability like the SAM large model without increasing its size on a large scale. Summary of the invention

[0005] Therefore, the purpose of the present invention is to provide a cell nucleus segmentation method based on a cascaded encoding segmentation network guided by a large model, using a cascaded encoder segmentation network guided by a fine-tuned large model encoder, including a fine-tuned large model guided cascade encoder that can reduce the amount of training parameters and enhance information interaction between encoders, a multi-stage feature fusion module that filters useless information in jump connections, and a decoder. It can effectively enhance the model's ability to capture local-non-local information without introducing a large number of training parameters, and achieve accurate segmentation of cell nuclei, which has broad application prospects.

[0006] In order to achieve the above-mentioned object, the present invention provides a cell nucleus segmentation system based on a cascaded encoding segmentation network guided by a large model, an image acquisition module, a cascaded encoder segmentation network; the cascaded encoder segmentation network includes a multi-stage feature fusion module, a cascaded encoder module and a cascaded decoder module; the cascaded encoder module includes a fine-tuned SAM encoder and a plurality of cascaded CNN encoders; the multi-stage feature fusion module includes a first-stage feature fusion module and a second-stage feature fusion module;

[0007] The image acquisition module is used to acquire the original cell image and perform enhancement processing on the original cell image;

[0008] The cascade encoder segmentation network obtains the processed cell image, and uses the fine-tuned SAM encoder to extract the guide feature map with global information semantic features; each level of CNN encoder is used to extract the feature map with local information semantic features step by step, and the first stage feature fusion module is used to fuse the guide feature map with the feature map extracted by the CNN encoders at each level for feature extraction by the next level of CNN encoder;

[0009] The cascade decoder module inputs the feature map fused by the second-stage feature fusion module into the cascade decoder, and restores the image resolution layer by layer through each level of decoder to obtain the final segmentation result map with the same size and number of channels as the input.

[0010] Further preferably, the fine-tuned SAM encoder is a VisionTransformer network with frozen parameters after training with massive data.

[0011] Further preferably, the fine-tuning SAM encoder performs domain alignment through a domain alignment branch, and the domain alignment branch is a bottleneck structure consisting of a first linear layer connected to a GELU activation function connected to a second linear layer;

[0012] The first linear layer is used to perform dimensionality reduction operation on the extracted features and map the feature vectors to a low-dimensional feature space;

[0013] The GELU activation function is used to introduce nonlinearity; the second linear layer is used to restore the dimension of the feature vector processed by the activation function to the original dimension.

[0014] Further preferably, the first-stage feature fusion module includes a resampling module, a convolution module and a first channel attention module; after the guide feature map and the feature map extracted by each level of CNN encoder are dimensionally adjusted through the resampling module, the convolution module is used to extract features, and weights are added to each channel through the first channel attention module to reduce the impact of useless channels on model segmentation.

[0015] Further preferably, the second-stage feature fusion module includes a spatial attention module and a second channel attention module, which adopts spatial-channel dual attention, performs channel weighting on the basis of spatial weighting, and simultaneously adds features at all levels to the final cell nucleus segmentation result map.

[0016] Further preferably, the decoder module is composed of multiple transposed convolution modules connected in series, which are connected by a second-stage feature fusion module. The second-stage feature fusion module generates multi-level jump connections from the outputs of decoders at each level, and is concatenated and fused with the transposed convolution output features at each level in a cascade manner. Detailed information and real features are supplemented in the decoding stage to restore the final segmentation result of the cell nucleus.

[0017] The present application also provides a cell nucleus segmentation method based on a cascade coding segmentation network guided by a large model, which is implemented based on the above cell nucleus segmentation system and includes the following steps:

[0018] S1, obtaining the original cell image, and performing enhancement processing on the original cell image;

[0019] S2. Input the enhanced cell image into the constructed cascade encoder segmentation network guided by the fine-tuned large model encoder, use the fine-tuned SAM encoder to extract the guided feature map with global information semantic features, and use the cascade encoder to extract the feature map of local information semantic features step by step; fuse the guided feature map with the feature maps at each level and input them into the next level encoder for feature extraction;

[0020] S3. The feature map formed by fusing the feature map output by the last level encoder with the guide feature map is input into the cascaded decoder, and the image resolution is restored layer by layer through the decoder module to obtain the final segmentation result map with the same size and number of channels as the input.

[0021] Further preferably, in S1, the fine-tuned SAM encoder performs domain alignment through a domain alignment branch, and the domain alignment branch is a bottleneck structure consisting of a first linear layer connected to a GELU activation function and then connected to a second linear layer;

[0022] The first linear layer is used to perform dimensionality reduction operation on the extracted features and map the feature vectors to a low-dimensional feature space;

[0023] The GELU activation function is used to introduce nonlinearity; the second linear layer is used to restore the dimension of the feature vector processed by the activation function to the original dimension.

[0024] The present invention also provides an electronic device, comprising:

[0025] a memory storing computer program instructions;

[0026] A processor, when the computer program instructions are executed by the processor, implements the steps of the cell nucleus segmentation method based on the large model guided cascade encoding segmentation network as described above.

[0027] The present invention also provides a computer-readable storage medium, which is used to store instructions. When the stored instructions are executed on a computer, the computer executes the steps of the cell nucleus segmentation method based on the large model-guided cascade coding segmentation network.

[0028] The cell nucleus segmentation system and method based on the cascade coding segmentation network guided by a large model disclosed in the present application have at least the following advantages over the prior art:

[0029] The present invention proposes to apply an encoder with a powerful feature extraction capability of a large segmentation model to a U-shaped network architecture for cell nucleus segmentation, designs a cascade encoder to combine the global perception advantages and local perception advantages of Transformer and CNN, and uses dilated convolution to increase the encoder's receptive field, helping the model to learn effective features in cell nucleus images, solving the problem of difficult-to-distinguish boundaries between cell nucleus structures, while making up for the fact that traditional encoders ignore the excessive amount of model trainable parameters and insufficient computational cost in effectively combining the advantages of Transformer structure and convolutional network structure, making it difficult to achieve training efficiency and segmentation accuracy at the same time when performing cell nucleus segmentation. The network proposed by the present invention can effectively solve the problem of the size of model parameters, and obtain high-quality segmentation results with a small increase in parameters, thereby achieving accurate segmentation of cell nuclei. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a structural schematic diagram of a cell nucleus segmentation method based on a large model-guided cascade coding segmentation network of the present invention;

[0031] Figure 2It is a structural schematic diagram of a cascade encoder segmentation network guided by a fine-tuned large model encoder according to the present invention; wherein 1-feature fusion module and 2-feature fusion module represent a first-stage feature fusion module and a second-stage feature fusion module respectively;

[0032] Figure 3 This is a schematic diagram of the pre-trained SAM encoder structure after domain alignment, where the ice and snow symbol indicates that the parameters are frozen, and the flame symbol indicates that the parameters are trainable;

[0033] Figure 4 It is a schematic diagram of a domain alignment branch structure for domain alignment of a SAM encoder;

[0034] Figure 5 It is a schematic diagram of the structure of the multi-stage feature fusion module proposed in the present invention;

[0035] Figure 6 It is a comparison diagram of the visual effects obtained by using the network of the present invention to perform cell nucleus segmentation on the public dataset MoNuSeg. DETAILED DESCRIPTION

[0036] The present invention is further described in detail below through the accompanying drawings and specific embodiments.

[0037] like Figure 1 As shown, the present application provided by an embodiment of one aspect of the present invention further provides a cell nucleus segmentation method based on a cascade coding segmentation network guided by a large model, which is implemented based on the above cell nucleus segmentation system and includes the following steps:

[0038] S1, obtaining the original cell image, and performing enhancement processing on the original cell image;

[0039] S2. Input the enhanced cell image into the constructed cascade encoder segmentation network guided by the fine-tuned large model encoder, use the fine-tuned SAM encoder to extract the guided feature map with global information semantic features, and use the cascade encoder to extract the feature map of local information semantic features step by step; fuse the guided feature map with the feature maps at each level and input them into the next level encoder for feature extraction;

[0040] S3. The feature map formed by fusing the feature map output by the last level encoder with the guide feature map is input into the cascaded decoder, and the image resolution is restored layer by layer through the decoder module to obtain the final segmentation result map with the same size and number of channels as the input.

[0041] Further preferably, in S1, the fine-tuned SAM encoder performs domain alignment through a domain alignment branch, and the domain alignment branch is a bottleneck structure consisting of a first linear layer connected to a GELU activation function connected to a second linear layer;

[0042] The first linear layer is used to perform dimensionality reduction operation on the extracted features and map the feature vectors to a low-dimensional feature space;

[0043] The GELU activation function is used to introduce nonlinearity; the second linear layer is used to restore the dimension of the feature vector processed by the activation function to the original dimension.

[0044] When performing feature map fusion, the first stage feature fusion module includes a resampling module, a convolution module and a first channel attention module; after the dimension of the guided feature map and the feature map extracted by each level of CNN encoder is adjusted through the resampling module, the convolution module is used to extract features, and weights are added to each channel through the first channel attention module to reduce the impact of useless channels on model segmentation.

[0045] The second-stage feature fusion module includes a spatial attention module and a second channel attention module. It adopts spatial-channel dual attention, performs channel weighting on the basis of spatial weighting, and adds features at all levels to the final cell nucleus segmentation result map.

[0046] The decoder module is composed of multiple transposed convolution modules connected in series, which are connected by the second-stage feature fusion module. The second-stage feature fusion module generates multi-level skip connections from the outputs of decoders at each level, and concatenates and fuses them with the output features of transposed convolution at each level in a cascade manner. In the decoding stage, detailed information and real features are supplemented to restore the final segmentation result of the cell nucleus.

[0047] like Figure 2 As shown, the present invention also provides a cell nucleus segmentation system based on a cascaded encoding segmentation network guided by a large model, an image acquisition module, a cascaded encoder segmentation network; the network adopts a U-shaped encoder-decoder structure; the cascaded encoder segmentation network includes a multi-stage feature fusion module, a cascaded encoder module and a cascaded decoder module; the cascaded encoder module includes a fine-tuned SAM encoder and a plurality of cascaded CNN encoders; the multi-stage feature fusion module includes a first-stage feature fusion module and a second-stage feature fusion module;

[0048] The cascade encoder uses a domain-aligned SAM encoder followed by a first-stage feature fusion module and a multi-stage CNN encoder for cascading to effectively extract multi-stage image features with local-non-local information of the image. Each encoder of the cascade encoder is followed by a second-stage feature fusion module, and each second-stage feature fusion module is followed by a decoder at each level to ensure the transmission of local-non-local information in the decoder stage.

[0049] The image acquisition module is used to acquire the original cell image and perform enhancement processing on the original cell image; the enhancement processing includes horizontal flipping, vertical flipping, random rotation and scaling, and normalizing the cell nucleus image, normalizing the grayscale pixels of the input image to the range of [0, 255], and obtaining the preprocessed input image;

[0050] See also Figure 2 , Figure 3 , the cascade encoder segmentation network obtains the processed cell image, the fine-tuned SAM encoder is a fine-tuned large model encoder after domain alignment; the fine-tuned SAM encoder is used to extract the guide feature map with global information semantic features; each level of CNN encoder is used to extract the feature map with local information semantic features step by step, and the first stage feature fusion module is used to fuse the guide feature map with the feature map extracted by the CNN encoders at each level for the next level of CNN encoder to perform feature extraction;

[0051] The SAM encoder achieves domain knowledge alignment by adding a bottleneck structure (linear layer, GELU activation function, linear layer) after the multi-head attention. Without a large increase in model parameters, the SAM encoder with powerful feature extraction capabilities that has been pre-trained on a large dataset can demonstrate its ability on the medical image segmentation dataset. The SAM encoder is Figure 3 Shown is a VisionTransformer network with frozen parameters after training on massive data.

[0052] Wherein, the fine-tuned SAM encoder performs domain alignment through a domain alignment branch, and the domain alignment branch is a bottleneck structure consisting of a first linear layer connected to a GELU activation function connected to a second linear layer;

[0053] The first linear layer is used to perform dimensionality reduction operation on the extracted features and map the feature vectors to a low-dimensional feature space;

[0054] The GELU activation function is used to introduce nonlinearity; the second linear layer is used to restore the dimension of the feature vector after the activation function is processed to the original dimension. The Transformer-based network architecture has advantages in extracting global feature information; and the cascaded lower-level encoders 0 to 4 are multi-level CNN encoders formed by connecting multiple convolution blocks in series. Each convolution block extracts multi-level features of the image through convolution operations, making it easier to extract local information of the image. The two basic encoders are cascaded and fused to effectively combine the advantages of the two decoders.

[0055] The cascade decoder module inputs the feature map fused by the second-stage feature fusion module into the cascade decoder. The image resolution is restored layer by layer through each level of decoder to obtain the final segmentation result map with the same size and number of channels as the input. Figure 2 In the figure, 1 is the first stage feature fusion module, 2 is the second stage feature fusion module. In the encoding stage, the first stage feature fusion module is connected between the two encoders in the cascade encoder to perform information filtering; in the decoding stage, the multi-level features output by the encoder are fused with the features output by the multi-level decoder through the second stage feature fusion module.

[0056] The first-stage feature fusion module includes a resampling module, a convolution module and a first channel attention module; after the guide feature map and the feature map extracted by each level of CNN encoder are dimensionally adjusted through the resampling module, the convolution module is used to extract features, and weights are added to each channel through the first channel attention module to reduce the impact of useless channels on model segmentation.

[0057] The second-stage feature fusion module includes a spatial attention module and a second channel attention module, which adopts spatial-channel dual attention, performs channel weighting on the basis of spatial weighting, and adds features of each level to the final cell nucleus segmentation result map. The decoder module is composed of multiple transposed convolution modules in series, which are connected by the second-stage feature fusion module. The second-stage feature fusion module generates multi-level jump connections from the outputs of decoders at all levels, and splices and fuses them with the output features of transposed convolution at all levels in a cascade manner, supplements detailed information and real features in the decoding stage, and restores the final segmentation result of the cell nucleus.

[0058] The present invention also provides an electronic device, comprising:

[0059] a memory storing computer program instructions;

[0060] A processor, when the computer program instructions are executed by the processor, implements the steps of the cell nucleus segmentation method based on the large model guided cascade encoding segmentation network as described above.

[0061] The present invention also provides a computer-readable storage medium, which is used to store instructions. When the stored instructions are executed on a computer, the computer executes the steps of the cell nucleus segmentation method based on the large model-guided cascade coding segmentation network.

[0062] See also Figure 4The domain alignment branch is a bottleneck structure composed of a linear layer + GELU activation function + a linear layer. The bottleneck structure is connected to the multi-head attention layer in the upper encoder VisionTransformer encoder to introduce an encoder for the fine-tuning large model SAM with a small number of parameters, and forms a fine-tuning large model encoder module with the SAM encoder.

[0063] See also Figure 5 The multi-stage feature fusion module adopts a staged approach to perform feature fusion in different aspects. The structure includes a resampling module, a convolution module and a channel attention module in the first stage: channel angle attention is used to add weights to each channel to reduce the impact of useless channels on model segmentation; the second stage includes a spatial attention followed by a channel attention module, which uses spatial-channel dual attention and performs channel weighting operations on the basis of spatial weighting. In the decoding stage, the damage of useless information to the segmentation performance is suppressed, and at the same time, features at all levels are added to the final cell nucleus segmentation result map.

[0064] See also Figure 2 ,The decoder module is composed of multiple transposed convolution modules in series, which are connected by the second stage feature fusion module, which generates multi-level jump connections from the outputs of each level of decoders in the decoder stage, and is concatenated and fused with the output features of each level of transposed convolution in a cascaded manner, which supplements the detailed information and the real features in the decoding stage, and finally restores the final segmentation result of the nucleus.

[0065] The specific implementation process of the present invention is described below in conjunction with embodiments:

[0066] Firstly, the cell image is taken as input and the network operation parameters are set. The early stopping mechanism is used for the constructed cascade encoder segmentation network guided by the fine-tuned large model encoder to train the network until convergence.

[0067] The online enhancement processing methods include horizontal and vertical flipping, horizontal rotation, etc., and the input image pixels are normalized to the range of [0, 255], and the image size is fixed to 224 × 224. The network parameters are set to 3, the training batch size is set to 3, the Adam optimizer is used, and the initial learning rate is set to 0.001, and the cosine annealing strategy is used to adjust the learning rate.

[0068] like Figure 6Taking the cell image X as an example, its image resolution is 1000×1000 and the number of channels is 3. After a series of online enhancement and normalization preprocessing, the input size of the original image X is fixed to 224×224, and then it is sent to the upper encoder and the lower encoder in the cascade encoder at the same time. Then, the feature map with non-local information obtained in the upper encoder (i.e., the guide feature, whose resolution is 64×64 and the number of channels is 256) is spliced ​​and fused with the upper level output features of each level of CNN encoder layer by layer through the first stage feature fusion module and sent to the encoder of this level, and so on to obtain the final resolution of 14×14, 28×28, 56×56, 112×112, and the corresponding channel numbers are 512, 256, 128, 64. The multi-level feature map; the decoder has four corresponding levels of transposed convolution modules as a multi-level decoder, which restores the resolution layer by layer and outputs the final segmentation result with the resolution and the number of channels consistent with the input image.

[0069] The present invention is based on a cascade encoder segmentation network guided by a fine-tuned large model encoder, which is designed to efficiently and cost-effectively segment the nucleus in the nucleus image. Compared with the existing nucleus segmentation method, the present invention utilizes a SAM encoder with a strong feature extraction capability that has been trained on a data set containing 11 million images and more than 1 billion segmentation masks, and adds a domain alignment branch to the SAM encoder to make the encoder adaptable to the nucleus image data set, and can train a model with more competitive segmentation accuracy with a smaller amount of training data, and cascade the SAM encoder after the first domain alignment with a multi-level CNN encoder, taking into account the network's detail capture capability and global modeling capability, and training a small number of parameters to obtain a powerful feature extraction capability for extracting more comprehensive information, and designing a multi-stage feature fusion module, supplementing feature information in the encoder stage, combining local-non-local information, and filtering useless information in the decoder stage while retaining more details and real information, to achieve more complete detail information recovery, and to achieve a more effective and more accurate nucleus segmentation method.

[0070] In order to test the superiority and effectiveness of the present invention, the following experiments are further described.

[0071] Hardware experimental platform: NVIDIA GeForce RTX 3090 GPU with 24GB video memory; implementation framework: Pytorch.

[0072] The MoNuSeg public dataset is used to train and test the network. The dataset contains 44 images, 30 of which are used as training sets and 14 as test sets. This experiment uses the training set and test set given in the dataset.

[0073] The present invention uses the following index parameters to evaluate the segmentation performance:

[0074]

[0075]

[0076] Among them, TP, TN, FP and FN represent the number of true positive, true negative, false positive and false negative samples respectively. Dice is a set similarity metric, which measures the accuracy of segmentation by calculating the ratio of the intersection area of ​​the predicted result and the true result to their average value; IoU is the intersection-over-union ratio of the predicted sample and the actual sample, which measures the image segmentation performance by calculating the ratio of the intersection area of ​​the predicted segmentation result and the true segmentation result to their union. The value range of Dice and IoU is [0, 1]. The closer the value is to 1, the better the model segmentation effect is.

[0077] First, an ablation experiment is performed to verify the effects of the fine-tuning large model encoder module and the multi-stage feature fusion module HFM proposed in the present invention. The experimental results are shown in Table 1:

[0078] In Table 1, “+” indicates that the corresponding module is applied, HMF indicates the multi-stage feature fusion module, otherwise, ordinary skip links are used directly for feature concatenation, ASE indicates the fine-tuned large model encoder, that is, the SAM encoder after domain alignment, otherwise, only the multi-level CNN encoder is used. Among them, the first row represents the original UNet network structure, which is used as the baseline model.

[0079] It can be seen from Table 1 that both the HMF module and the fine-tuned large model encoder module can improve the segmentation accuracy of the model. After using the fine-tuned large model encoder module and the HMF module at the same time, the accuracy of the model's segmentation results has been greatly improved.

[0080] Table 1 Ablation experiments on components of the cascade encoder segmentation network guided by fine-tuning the large model encoder

[0081]

[0082] In order to comprehensively verify and intuitively demonstrate the performance advantage of the fine-tuned large model encoder-guided cascade encoder segmentation network proposed in this paper in cell nucleus segmentation, the fine-tuned large model encoder-guided cascade encoder segmentation network proposed in this paper, hereinafter referred to as SGDENet, is compared with several mainstream medical image segmentation networks. The selected comparison networks include UNet, UNet++ and AttentionUNet, whose encoder backbone is mainly CNN module.

[0083] (Oktay O, Schlemper J, Folgoc LL, et al. Attention U-net: Learning where to look for the panc reasons [J]. arXiv preprint arXiv: 1804.03999, 2018.), Swin-UNet with Transformer as the backbone (Cao H, Wang Y, Chen J, et al. Swin-unet: Unet-like pure transformer formed educational image segmentation [C] / / European conference on computer vision. Cham: Springer Nature Switzerland, 2022: 205-218.), TransUNet and UDTransNet, which focuses on skip connections and eliminates the semantic gap between the encoder and decoder by introducing Transformer into skip connections.

[0084] (Wang H, Cao P, Liu X, et al. Narrowing the semantic gaps in U-Net with learnable skip connections: The case of medical image segmentation [J]. arXiv preprint arXiv: 2312.15182, 2023.). The comparison indicators involved include the parameter quantity Params, the segmentation accuracy evaluation indicators Dice and IOU.

[0085] The quantification results of the segmentation accuracy of the above seven segmentation networks on the cell nucleus dataset are shown in Table 2. As can be seen from Table 2, SGDENet's Dice and IoU achieved good scores. After comprehensive comparison, the network SGDENet designed by the present invention has higher competitiveness. In summary, SGDENet's performance in cell nucleus segmentation is significantly better than other non-large model segmentation networks, and can effectively improve the accuracy of cell nucleus segmentation.

[0086] Table 2 Parameters and segmentation accuracy of SGDENet and the most advanced methods

[0087]

[0088] Through Figure 6The comprehensive analysis of Table 2 and Table 3 proves that the cascade encoder segmentation network guided by the fine-tuned large model encoder proposed in the present invention has a more accurate cell nucleus segmentation capability, and this method can be extended to other pathological image segmentation cases, and has broad application prospects in the fields of medical auxiliary diagnosis and so on.

[0089] This embodiment shows the implementation effect of cell nucleus segmentation, but the present invention is not limited to the segmentation of cell nucleus images. The present invention is also applicable to other forms of medical or non-medical image segmentation tasks. The present invention can be implemented in other specific forms without departing from its spirit or essential characteristics. The described embodiments are considered to be illustrative and non-restrictive in all aspects, for example:

[0090] 1) The supported carrier image types cover various forms of medical imaging, including but not limited to X-ray, computed tomography (CT), ultrasound, positron emission tomography (PET) and endoscopic images;

[0091] 2) The segmentation target range is not limited to cell nucleus segmentation, gland segmentation, lesion segmentation, organ segmentation, blood vessel segmentation, and bone segmentation;

[0092] 3) The type of segmentation is not limited to general semantic segmentation, but also includes instance segmentation, etc.

[0093] 4) The selection of parameter configuration is flexible, and technicians in this field can adjust and optimize it according to specific task requirements.

[0094] Obviously, the above embodiments are only examples for clear explanation, and are not intended to limit the implementation methods. For ordinary technicians in the relevant field, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived from them are still within the protection scope of the invention.

Claims

1. A cell nucleus segmentation system based on a large model-guided cascade encoding segmentation network, characterized in that: It includes an image acquisition module and a cascade encoder segmentation network; the cascade encoder segmentation network includes a multi-stage feature fusion module, a cascade encoder module and a cascade decoder module; the cascade encoder module includes a fine-tuned SAM encoder and multiple cascaded CNN encoders; The multi-stage feature fusion module includes a first-stage feature fusion module and a second-stage feature fusion module; The image acquisition module is used to acquire the original cell image and perform enhancement processing on the original cell image; The cascade encoder segmentation network obtains the processed cell image, and uses the fine-tuned SAM encoder to extract the guide feature map with global information semantic features; each level of CNN encoder is used to extract the feature map with local information semantic features step by step, and the first stage feature fusion module is used to fuse the guide feature map with the feature map extracted by the CNN encoders at each level for feature extraction by the next level of CNN encoder; The cascade decoder module inputs the feature map fused by the second-stage feature fusion module into the cascade decoder, and restores the image resolution layer by layer through each level of decoder to obtain the final segmentation result map with the same size and number of channels as the input.

2. The cell nucleus segmentation system based on the large model-guided cascade encoding segmentation network according to claim 1, characterized in that: The fine-tuned SAM encoder is a VisionTransformer network with frozen parameters after training with massive data.

3. The cell nucleus segmentation system based on the large model-guided cascade encoding segmentation network according to claim 1, characterized in that: The fine-tuned SAM encoder is domain aligned through a domain alignment branch, wherein the domain alignment branch is a bottleneck structure consisting of a first linear layer connected to a GELU activation function and then connected to a second linear layer; The first linear layer is used to perform dimensionality reduction operation on the extracted features and map the feature vectors to a low-dimensional feature space; The GELU activation function is used to introduce nonlinearity; the second linear layer is used to restore the dimension of the feature vector processed by the activation function to the original dimension.

4. The cell nucleus segmentation system based on the large model-guided cascade encoding segmentation network according to claim 1, characterized in that: The first-stage feature fusion module includes a resampling module, a convolution module and a first channel attention module; after the guide feature map and the feature map extracted by each level of CNN encoder are dimensionally adjusted through the resampling module, the convolution module is used to extract features, and weights are added to each channel through the first channel attention module to reduce the impact of useless channels on model segmentation.

5. The cell nucleus segmentation system based on the large model-guided cascade encoding segmentation network according to claim 1, characterized in that: The second-stage feature fusion module includes a spatial attention module and a second channel attention module, which adopts spatial-channel dual attention, performs channel weighting on the basis of spatial weighting, and adds features at all levels to the final cell nucleus segmentation result map.

6. The cell nucleus segmentation system based on the large model-guided cascade encoding segmentation network according to claim 1, characterized in that: The decoder module is composed of multiple transposed convolution modules connected in series, which are connected by the second-stage feature fusion module. The second-stage feature fusion module generates multi-level skip connections from the outputs of decoders at each level, and concatenates and fuses them with the output features of transposed convolution at each level in a cascade manner. In the decoding stage, detailed information and real features are supplemented to restore the final segmentation result of the cell nucleus.

7. A cell nucleus segmentation method based on a large model-guided cascade coding segmentation network, characterized in that: The cell nucleus segmentation system according to any one of claims 1 to 6 is implemented, comprising the following steps: S1, obtaining the original cell image, and performing enhancement processing on the original cell image; S2. Input the enhanced cell image into the constructed cascade encoder segmentation network guided by the fine-tuned large model encoder, use the fine-tuned SAM encoder to extract the guided feature map with global information semantic features, and use the cascade encoder to extract the feature map of local information semantic features step by step; fuse the guided feature map with the feature maps at each level and input them into the next level encoder for feature extraction; S3. The feature map formed by fusing the feature map output by the last level encoder with the guide feature map is input into the cascaded decoder, and the image resolution is restored layer by layer through the decoder module to obtain the final segmentation result map with the same size and number of channels as the input.

8. The cell nucleus segmentation method based on the large model-guided cascade coding segmentation network according to claim 7, characterized in that: In S1, the fine-tuned SAM encoder is domain aligned through a domain alignment branch, where the domain alignment branch is a bottleneck structure consisting of a first linear layer connected to a GELU activation function connected to a second linear layer; The first linear layer is used to perform dimensionality reduction operation on the extracted features and map the feature vectors to a low-dimensional feature space; The GELU activation function is used to introduce nonlinearity; the second linear layer is used to restore the dimension of the feature vector processed by the activation function to the original dimension.

9. An electronic device, characterized in that: include: a memory storing computer program instructions; A processor, when the computer program instructions are executed by the processor, implements the steps of the cell nucleus segmentation method based on the large model-guided cascade encoding segmentation network as described in any one of claims 7 to 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store instructions. When the stored instructions are executed on a computer, the computer is caused to execute the steps of the cell nucleus segmentation method based on a large model-guided cascade encoding segmentation network as described in any one of claims 7 to 8.