Pathological image cell segmentation method and device based on large model and knowledge distillation

Abstract

The invention relates to the technical field of medical image pathology analysis, in particular to a pathology image cell segmentation method and device based on a large model and knowledge distillation, and the method comprises the steps: generating a to-be-segmented image through employing a target slice image; generating a target prompt feature vector based on a target prompt encoder; extracting a first image feature of the to-be-segmented image by using a first target image encoder, and determining a second target image encoder based on a knowledge distillation technology to extract a second image feature of the to-be-segmented image; and inputting the target prompt feature vector, the first image feature and the second image feature into a target mask decoder to obtain a preliminary cell segmentation mask, performing feature fusion to obtain an enhanced cell segmentation mask, and further determining a cell segmentation result of the to-be-segmented image. Therefore, the problems that the model training time is increased, the calculation burden is increased, the cell boundary cannot be accurately described, and the accuracy and robustness of cell segmentation are reduced in a cell segmentation method in the related technology are solved.

Description

Technical Field

[0001] This application relates to the field of medical image pathology analysis technology, and in particular to a method and apparatus for cell segmentation of pathology images based on large models and knowledge distillation. Background Technology

[0002] Medical image segmentation is crucial in the field of medical imaging and plays a vital role in improving the sustainability of healthcare. However, this field faces several challenges, such as dataset limitations, variations in cell nucleus morphology across different organ and tissue types, the complexity of densely packed and color-varying cell nuclei, and the sparsity of cell nucleus distribution. Furthermore, differences in imaging equipment and slide processing procedures between hospitals and clinics also contribute to the diversity of image quality.

[0003] To address these issues, researchers have strived to improve the accuracy and stability of cell segmentation results. Current medical image segmentation methods include traditional approaches and deep learning-based algorithms. Deep learning algorithms, particularly fully convolutional neural networks (FCNs) and U-Net models, have become the mainstream choice for medical image segmentation, significantly improving segmentation performance through their encoder-decoder design and skip connections. However, these methods all place high demands on the training dataset. For large-scale model applications, SAM (Segment Anything Model) has been proposed as a fundamental image segmentation paradigm, supporting zero-shot image segmentation. To further enhance SAM's performance in medical imaging, MedSAM employs a simple fine-tuning strategy to optimize its medical image segmentation results. Furthermore, SAMUS increases SAM's generalization ability in medical image segmentation by introducing cross-branch attention from parallel CNN (Convolutional Neural Networks) branches and the ViT (Vision Transformer) encoder.

[0004] However, cell segmentation methods based on large models in related technologies require a large amount of computational resources for training and inference, which increases the training time and computational burden of the model, resulting in low cell segmentation efficiency and an inability to accurately depict cell boundaries, thus reducing the accuracy and robustness of cell segmentation. These issues urgently need to be addressed. Summary of the Invention

[0005] This application provides a method and apparatus for cell segmentation of pathological images based on large models and knowledge distillation, in order to solve the problems in related technologies where cell segmentation methods based on large models require a large amount of computing resources for training and inference, which increases the training time and computational burden of the model, resulting in low cell segmentation efficiency and inability to accurately depict cell boundaries, thus reducing the accuracy and robustness of cell segmentation.

[0006] The first aspect of this application provides a method for cell segmentation of pathological images based on large models and knowledge distillation, comprising the following steps: acquiring a target slice image that meets a first preset condition, and generating a segmentation image that meets a second preset condition using the target slice image; processing target input prompt information that meets the preset segmentation conditions based on a target prompt encoder to generate a target prompt feature vector; extracting a first image feature of the segmentation image using a first target image encoder, and determining a second target image encoder based on the first image feature, and extracting a second image feature of the segmentation image using the second target image encoder; inputting the target prompt feature vector, the first image feature, and the second image feature into a target mask decoder to obtain a preliminary cell segmentation mask, and fusing the preliminary cell segmentation mask, the first image feature, and the second image feature to obtain an enhanced cell segmentation mask, and using the enhanced cell segmentation mask to determine the pathological image cell segmentation result of the segmentation image.

[0007] Optionally, in one embodiment of this application, generating a segmented image that satisfies a second preset condition using the target slice image includes: cutting the target slice image to obtain a target sliced ​​image that meets a preset size; and normalizing the target sliced ​​image to obtain the segmented image that satisfies the second preset condition.

[0008] Optionally, in one embodiment of this application, the step of processing target input prompt information that meets preset segmentation conditions based on the target prompt encoder to generate a target prompt feature vector includes: determining multiple learnable vectors based on the input prompt information, and determining multiple target points corresponding to the multiple learnable vectors based on the label information in the input prompt information; calculating the multiple learnable vectors to obtain a feature vector of the input prompt information; and generating the target prompt feature vector based on the label of each target point, the multiple learnable vectors, and the feature vector.

[0009] Optionally, in one embodiment of this application, determining the second target image encoder based on the first image features includes: generating a target small image encoder using a target large image encoder based on a knowledge distillation technique and a general segmentation model SAM; and training the target small image encoder using the first image features to obtain the second target image encoder.

[0010] Optionally, in one embodiment of this application, the step of fusing the preliminary cell segmentation mask, the first image feature, and the second image feature to obtain an enhanced cell segmentation mask includes: extracting target local features from the preliminary cell segmentation mask, and determining a first local feature of the first image feature and a second local feature of the second image feature based on the target local features; concatenating the first local feature and the second local feature to generate a first fused feature map, and calculating the global features of the first fused feature map to obtain a global feature vector; extending the global feature vector to the target space dimension, and fusing the extended global feature vector with the first fused feature map to obtain a second fused feature map; convolving the second fused feature map to generate a third fused feature map, and using the third fused feature map to generate spatial attention weights, so as to obtain the enhanced cell segmentation mask using the spatial attention weights.

[0011] Optionally, in one embodiment of this application, determining the pathological image cell segmentation result of the image to be segmented using the enhanced cell segmentation mask includes: performing noise reduction processing on the enhanced cell segmentation mask to obtain a first target cell region that meets preset processing conditions; and performing false positive region processing on the first target cell region to obtain the pathological image cell segmentation result of the image to be segmented.

[0012] A second aspect of this application provides a pathological image cell segmentation device based on a large model and knowledge distillation, comprising: an acquisition module for acquiring a target slice image that meets a first preset condition and generating a segmentation image that meets a second preset condition using the target slice image; a generation module for processing target input prompt information that meets the preset segmentation conditions based on a target prompt encoder to generate a target prompt feature vector; an extraction module for extracting a first image feature of the segmentation image using a first target image encoder, determining a second target image encoder based on the first image feature, and extracting a second image feature of the segmentation image using the second target image encoder; and a processing module for inputting the target prompt feature vector, the first image feature, and the second image feature into a target mask decoder to obtain a preliminary cell segmentation mask, and performing feature fusion on the preliminary cell segmentation mask, the first image feature, and the second image feature to obtain an enhanced cell segmentation mask, thereby using the enhanced cell segmentation mask to determine the pathological image cell segmentation result of the segmentation image.

[0013] Optionally, in one embodiment of this application, the acquisition module includes: a first acquisition unit, configured to cut the target slice image to obtain a target slice image that meets a preset size; and a first determination unit, configured to normalize the target slice image to obtain the image to be segmented that meets the second preset condition.

[0014] Optionally, in one embodiment of this application, the generation module includes: a second determining unit, configured to determine multiple learnable vectors based on the input prompt information, and determine multiple target points corresponding to the multiple learnable vectors based on the label information in the input prompt information; a second obtaining unit, configured to calculate the multiple learnable vectors to obtain the feature vector of the input prompt information; and a first generating unit, configured to generate the target prompt feature vector based on the label of each target point, the multiple learnable vectors, and the feature vector.

[0015] Optionally, in one embodiment of this application, the extraction module includes: a second generation unit, configured to generate a target small image encoder using a target large image encoder based on knowledge distillation technology and a general segmentation model SAM; and a third acquisition unit, configured to train the target small image encoder using the first image features to obtain the second target image encoder.

[0016] Optionally, in one embodiment of this application, the processing module includes: a third determining unit, configured to extract target local features of the preliminary cell segmentation mask, and determine a first local feature of the first image feature and a second local feature of the second image feature based on the target local features; a first processing unit, configured to concatenate the first local feature and the second local feature to generate a first fused feature map, and calculate the global features of the first fused feature map to obtain a global feature vector; a second processing unit, configured to extend the global feature vector to the target space dimension, and perform feature fusion with the extended global feature vector and the first fused feature map to obtain a second fused feature map; and a fourth determining unit, configured to convolve the second fused feature map to generate a third fused feature map, and generate spatial attention weights using the third fused feature map, so as to obtain the enhanced cell segmentation mask using the spatial attention weights.

[0017] Optionally, in one embodiment of this application, the processing module includes: a fourth acquisition unit, used to perform noise reduction processing on the enhanced cell segmentation mask to obtain a first target cell region that meets preset processing conditions; and a third processing unit, used to perform false positive region processing on the first target cell region to obtain the pathological image cell segmentation result of the image to be segmented.

[0018] A third aspect of this application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the pathological image cell segmentation method based on large models and knowledge distillation as described in the above embodiments.

[0019] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for pathological image cell segmentation based on large models and knowledge distillation.

[0020] A fifth aspect of this application provides a computer program product, including a computer program that, when executed, is used to implement the above-described method for segmenting pathological images based on large models and knowledge distillation.

[0021] This application embodiment can generate an image to be segmented using a target slice image, and process target input prompt information that meets the segmentation conditions based on a target prompt encoder to generate a target prompt feature vector. Then, a first target image encoder is used to extract the first image features of the image to be segmented, thereby determining a second target image encoder to extract the second image features of the image to be segmented. The target prompt feature vector, the first image features, and the second image features are input into a target mask decoder to obtain a preliminary cell segmentation mask, and feature fusion is performed to obtain an enhanced cell segmentation mask, thereby determining the cell segmentation result of the image to be segmented. This effectively improves cell segmentation efficiency, accuracy, and robustness. Therefore, it solves the problems in related technologies where cell segmentation methods based on large models require a large amount of computational resources for training and inference, increasing model training time and computational burden, resulting in low cell segmentation efficiency and inability to accurately depict cell boundaries, thus reducing the accuracy and robustness of cell segmentation.

[0022] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0023] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0024] Figure 1 This is a flowchart of a pathological image cell segmentation method based on large models and knowledge distillation, according to an embodiment of this application.

[0025] Figure 2 This is a schematic diagram of the original pathological image of a specific embodiment of this application;

[0026] Figure 3 This is a schematic diagram of the prompt information processing algorithm framework of a specific embodiment of this application;

[0027] Figure 4 This is a schematic diagram of a knowledge distillation algorithm according to a specific embodiment of this application;

[0028] Figure 5 This is a schematic diagram of a dual-channel mask fusion algorithm according to a specific embodiment of this application;

[0029] Figure 6 This is a schematic diagram of a global information integration and spatial selection convolution algorithm according to a specific embodiment of this application;

[0030] Figure 7 This is a schematic diagram of a pathological image cell segmentation device based on a large model and knowledge distillation, according to an embodiment of this application.

[0031] Figure 8 This is a schematic diagram of the structure of an electronic device provided according to an embodiment of this application. Detailed Implementation

[0032] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0033] The following description, with reference to the accompanying drawings, illustrates a method and apparatus for cell segmentation of pathological images based on large models and knowledge distillation, according to embodiments of this application. Addressing the issues mentioned in the background section regarding cell segmentation methods based on large models requiring substantial computational resources for training and inference, which increases model training time and computational burden, resulting in low cell segmentation efficiency and an inability to accurately depict cell boundaries, thus reducing the accuracy and robustness of cell segmentation, this application provides a method for cell segmentation of pathological images based on large models and knowledge distillation. In this method, a target slice image is used to generate an image to be segmented. Based on a target cue encoder, target input cue information that meets segmentation conditions is processed to generate a target cue feature vector. Then, a first target image encoder is used to extract first image features from the image to be segmented, thereby determining a second target image encoder to extract second image features from the image to be segmented. The target cue feature vector, the first image features, and the second image features are input into a target mask decoder to obtain a preliminary cell segmentation mask. Feature fusion is then performed to obtain an enhanced cell segmentation mask, thereby determining the cell segmentation result of the image to be segmented. This effectively improves cell segmentation efficiency and enhances the accuracy and robustness of cell segmentation. This solves the problems in related technologies, such as the need for large computational resources for training and inference in cell segmentation methods based on large models, which increases the training time and computational burden of the model, resulting in low cell segmentation efficiency and inability to accurately depict cell boundaries, thus reducing the accuracy and robustness of cell segmentation.

[0034] Specifically, Figure 1 This is a flowchart illustrating a pathological image cell segmentation method based on a large model and knowledge distillation, provided in an embodiment of this application.

[0035] like Figure 1 As shown, this pathological image cell segmentation method based on large models and knowledge distillation includes the following steps:

[0036] In step S101, a target slice image that meets the first preset condition is acquired, and a segmentation image that meets the second preset condition is generated using the target slice image.

[0037] It is understood that embodiments of this application can acquire target slice images that meet the first condition, for example, such as Figure 2As shown, this embodiment of the application can acquire stained whole-slice images. The whole-slice images are obtained by H&E (hematoxylin-eosin staining) staining of tumor tissue slices. The target slice image in the following steps is used to generate a segmentation image that meets the second condition. For example, this embodiment of the application can first divide a dataset containing multiple large-size HE images into a training image set, a validation image set, and a test image set. Then, during training, the images are preprocessed, that is, the whole image is segmented and resized to obtain the segmentation image, which effectively improves the feasibility of cell segmentation.

[0038] In one embodiment of this application, generating a segmented image that meets a second preset condition using a target slice image includes: cutting the target slice image to obtain a target sliced ​​image that meets a preset size; and normalizing the target sliced ​​image to obtain a segmented image that meets the second preset condition.

[0039] For example, in this embodiment of the application, the target slice image, i.e. the whole slice image, can be cut and resized to obtain a target slice image of a certain size, which can be compatible with the pre-trained neural network model. The target slice image is also adjusted by data format standardization and intensity normalization to obtain the image to be segmented, and it is ensured that all images to be segmented have consistent conditions during the training process. The image to be segmented contains multiple types of tumor tissue slices, which effectively improves the applicability of cell segmentation.

[0040] It should be noted that the preset size is set by those skilled in the art based on the actual situation, and no specific limitation is made here.

[0041] In step S102, based on the target prompt encoder, the target input prompt information that meets the preset segmentation conditions is processed to generate the target prompt feature vector.

[0042] It is understood that the embodiments of this application can process target input prompt information that meets certain segmentation conditions based on the target prompt encoder in the following steps, such as manually labeled cell boundary points, known cell features, etc., thereby generating prompt feature vectors. This ensures that the prompt feature vectors can help the model better understand the cell structure in the image to be segmented, effectively improving the accuracy of cell segmentation.

[0043] It should be noted that the preset segmentation conditions are set by those skilled in the art based on the actual situation, and are not specifically limited here.

[0044] In one embodiment of this application, a target prompt encoder processes target input prompt information that meets preset segmentation conditions to generate a target prompt feature vector. This includes: determining multiple learnable vectors based on the input prompt information, and determining multiple target points corresponding to the multiple learnable vectors based on the label information in the input prompt information; calculating the multiple learnable vectors to obtain the feature vector of the input prompt information; and generating a target prompt feature vector based on the label of each target point, the multiple learnable vectors, and the feature vector.

[0045] For example, such as Figure 3 As shown, in this embodiment of the application, a set of learnable vectors with a size of 4×1×256 can be generated based on the input prompt information. Then, another learnable vector with a size of 1×256 is generated to indicate that the position is not a point. The set of learnable vectors includes four special vectors with a size of 1×256 to represent different types of points.

[0046] Next, this embodiment of the application can perform detection. If there is no box in the input prompt, an additional point is added with coordinates [0,0] and label -1 to maintain the consistency of the data format. If there is a box in the input prompt, the size of the point is N×1×2 and the label can be 0 or 1 to distinguish whether the additional point needs to be processed in subsequent steps. The size of the label is N×1.

[0047] Secondly, the embodiments of this application can calculate learnable vectors. Specifically, the coordinates of the points are normalized to the range [0,1]. The normalized points are multiplied by a random Gaussian matrix to obtain a new matrix. The sine and cosine values ​​of the matrix are calculated separately, and the results are concatenated to obtain the feature vector of the input prompt information. The above calculation method can be quantified as follows:

[0048] c N =2×c-1

[0049] c T =c N ×G

[0050] c o =concat[sin(2π×c T ),cos(2π×c T )]

[0051] Wherein, the input coordinate c is a coordinate in [0,1] 2 Tensors within the range, with shape d1×…×d n ×2, G is a Gaussian matrix, concat(...) means concatenating the sine and cosine results along the last dimension, c N For the normalized tensor, c TLet c be the tensor resulting from multiplying with the Gaussian matrix. o This is the final output tensor.

[0052] Furthermore, the aforementioned finely matched landmarks simultaneously satisfy the conditions of both the highest level of biomarker similarity and the consistency of keypoint spatial distribution, ultimately outputting c. o The shape is d1×…×d n ×c, where c = 256.

[0053] Finally, in this embodiment, based on the feature vector, the previously generated learnable vector can be added to the corresponding point according to the label of each point, so that each point has a complete vector containing its location information and type information.

[0054] In step S103, the first image features of the image to be segmented are extracted using the first target image encoder, and based on the first image features, the second target image encoder is determined, and the second image features of the image to be segmented are extracted using the second target image encoder.

[0055] In the embodiments of this application, the first target image encoder is a large-scale image encoder; the second target image encoder is a small-scale image encoder.

[0056] It is understood that the embodiments of this application may use knowledge distillation technology to extract the first image features of the image to be segmented from the first target image encoder, i.e., the large-scale image encoder, and determine the second target image encoder, i.e. the small image encoder, based on the first image features. The small image encoder is then used to extract the second image features of the image to be segmented, thereby reducing the computational burden, effectively reducing the complexity of model computation, and improving the efficiency of cell segmentation.

[0057] In one embodiment of this application, determining a second target image encoder based on a first image feature includes: generating a target small image encoder using a target large image encoder based on knowledge distillation technology and the SAM; and training the target small image encoder using the first image feature to obtain the second target image encoder.

[0058] For example, such as Figure 4 As shown, the embodiments of this application can use knowledge distillation technology to enable the large-scale image encoder in the teacher model to acquire knowledge and fine-tune the mask decoder. Information is obtained by directly training the SAM model instead of starting from scratch. A small image encoder is extracted directly from the large image encoder of the fine-tuned SAM. Then, the small image encoder is trained using the first image features to obtain the small-scale image encoder in the trained student model. The training process does not involve combining the mask decoder, thus completing the knowledge distillation task and effectively reducing the computational complexity of the model.

[0059] Among them, large-scale image encoders can be large-scale ViT encoders with deep network structures and many parameters, enabling them to extract high-level image features and output high-level feature maps F. L Small encoders have shallower network structures and fewer parameters. Their main advantage lies in faster inference speed and output of low-level feature maps F. S ,

[0060] In step S104, the target cue feature vector, the first image feature, and the second image feature are input into the target mask decoder to obtain a preliminary cell segmentation mask. The preliminary cell segmentation mask, the first image feature, and the second image feature are then fused to obtain an enhanced cell segmentation mask, which is used to determine the pathological image cell segmentation result of the image to be segmented.

[0061] It is understood that, in the embodiments of this application, the target cue feature vector, the first image feature, and the second image feature in the following steps can be input into the target mask decoder to obtain a preliminary cell segmentation mask. The preliminary cell segmentation mask, the first image feature, and the second image feature are then fused together. That is, the preliminary cell segmentation mask is integrated through the fusion module to obtain an enhanced cell segmentation mask, which enhances the cell boundary features. Thus, the enhanced cell segmentation mask can be used to determine the pathological image cell segmentation result of the image to be segmented. Therefore, the embodiments of this application can effectively process the cell structure in complex pathological images and improve the accuracy and robustness of cell segmentation.

[0062] In actual implementation, such as Figure 5 As shown, embodiments of this application can use a mask decoder to generate a preliminary cell segmentation mask. The mask decoder can receive high-level feature maps from a large image encoder, basic feature maps from a small image encoder, and feature information from a cue encoder, and output a preliminary cell segmentation mask containing possible cell regions in the image. Secondly, embodiments of this application can integrate the preliminary cell segmentation mask through a fusion module. The fusion module combines the preliminary cell segmentation mask from the mask decoder with original image features, such as first image features and second image features, to enhance cell boundary features, thereby improving cell segmentation accuracy.

[0063] Optionally, in one embodiment of this application, feature fusion is performed on the preliminary cell segmentation mask, the first image feature, and the second image feature to obtain an enhanced cell segmentation mask, including: extracting target local features from the preliminary cell segmentation mask, and based on the target local features, determining the first local features of the first image feature and the second local features of the second image feature; concatenating the first local features and the second local features to generate a first fused feature map, and calculating the global features of the first fused feature map to obtain a global feature vector; extending the global feature vector to the target space dimension, and fusing the extended global feature vector with the first fused feature map to obtain a second fused feature map; convolving the second fused feature map to generate a third fused feature map, and using the third fused feature map to generate spatial attention weights, so as to obtain the enhanced cell segmentation mask using the spatial attention weights.

[0064] For example, such as Figure 5 As shown, embodiments of this application can extract and fuse local features from the preliminary cell segmentation mask, and respectively process the aforementioned high-level feature map F. L and basic feature map F S Local features are extracted by applying independent convolution operations.

[0065] Next, in this embodiment, two independent convolutional layers can be used to generate local feature maps C corresponding to the high-level feature maps. L The local feature map C corresponding to the low-level feature map S The above calculation method can be quantified as follows:

[0066] C L =Conv L (F L )

[0067] C S =Conv S (F S )

[0068] Among them, C L C represents the local feature map corresponding to the high-level feature map. S This is the local feature map corresponding to the low-level feature map.

[0069] Secondly, for the local feature map C L and C S Perform a stitching operation to generate a preliminary fused feature map C. F The above calculation method can be quantified as follows:

[0070] C F =concat(C L C S )

[0071] Among them, CF This is for the initial fusion of feature maps.

[0072] Furthermore, such as Figure 6 As shown, embodiments of this application can use global information integration and spatial selection convolution algorithms to fuse feature map C. F Perform global feature calculation to obtain the global feature vector G. F Among them, global features are obtained by analyzing C. F The spatial dimension is obtained by average pooling; G is then used to obtain G. F Extended to C via broadcast mechanism F Matching spatial dimensions and with C F Feature fusion is performed to obtain an enhanced fused feature map C. F G, the above calculation method can be quantified as follows:

[0073]

[0074] C F G=C F +G F

[0075] Among them, G F C is the global feature vector. F G represents the enhanced fused feature map, H represents the length of the feature map, and W represents the width of the feature map.

[0076] Next, regarding C F G applies convolutional operations, further processing the fused features through convolutional layers to enhance the feature correlation between channels and generate a fused feature map C. Fused And introduce spatially selected convolutional layers to fuse feature maps C Fused Spatial attention weights are generated to adjust the contributions of features from different channels, and the optimized feature map C is finally output. O The above calculation method can be quantified as follows:

[0077] C O =conv spatial (C Fused )

[0078] Among them, C O For the optimized feature map, C Fused To fuse feature maps.

[0079] Therefore, the embodiments of this application can utilize the optimized feature map to obtain an enhanced cell segmentation mask, enhance cell boundary features, and thus improve the accuracy of cell segmentation.

[0080] Optionally, in one embodiment of this application, determining the pathological image cell segmentation result of the image to be segmented using an enhanced cell segmentation mask includes: performing noise reduction processing on the enhanced cell segmentation mask to obtain a first target cell region that meets preset processing conditions; and performing false positive region processing on the first target cell region to obtain the pathological image cell segmentation result of the image to be segmented.

[0081] In this embodiment of the application, the preset processing conditions are those that retain a large area and meet the cell characteristics.

[0082] For example, embodiments of this application can perform noise reduction processing on the enhanced cell segmentation mask, that is, use the opening operation to remove isolated small noise regions. The opening operation removes small false positive regions by first eroding and then dilating. By analyzing each independently connected region, small regions whose area or shape does not conform to the actual cell characteristics are filtered out, and larger regions that conform to cell characteristics are retained.

[0083] Furthermore, for larger areas that conform to cell characteristics, false positive regions are processed. This involves analyzing the contrast differences between the inside and outside of the cell area to further enhance the salience of the real cell area, thereby weakening or eliminating false positive regions. Shape-based feature analysis, such as the roundness of the contour and the ratio of area to perimeter, is used to identify and remove false positive regions that are inconsistent with the typical cell shape. Finally, high-quality cell segmentation results are generated, ensuring the accuracy and robustness of cell segmentation.

[0084] Finally, the embodiments of this application can evaluate and post-process the final segmentation results, using various evaluation metrics such as Dice coefficient and IoU to evaluate the segmentation results, and perform post-processing according to actual needs, such as cell contour smoothing and connectivity checks of segmented regions, to ensure the accuracy and practicality of the segmentation results.

[0085] Therefore, this application embodiment achieves accurate cell segmentation of large-scale pathological images by introducing a large model and knowledge distillation technology, training the large model and transferring its knowledge to a smaller model. The network structure adopts a dual encoder model, which can effectively learn and fuse cell features, thereby significantly improving the accuracy and precision of cell segmentation in pathological images.

[0086] In other words, the embodiments of this application effectively reduce the computational burden and complexity of the model by distilling the knowledge of the large image encoder into the small image encoder, and transfer the semantic information, complex features and subtle information in the large model to the small model, thereby enhancing the performance of the small model in high-precision medical image segmentation tasks. Furthermore, the use of a fusion module to integrate local and global information enhances the model's understanding of the correlation between masks, effectively improving the segmentation accuracy and robustness of the model, and providing a useful reference for clinical pathologists.

[0087] The pathological image cell segmentation method based on large models and knowledge distillation proposed in this application can generate an image to be segmented using a target slice image. Based on a target cue encoder, target input cue information that meets the segmentation conditions is processed to generate a target cue feature vector. Then, a first target image encoder is used to extract the first image features of the image to be segmented, thereby determining a second target image encoder to extract the second image features of the image to be segmented. The target cue feature vector, the first image features, and the second image features are input into a target mask decoder to obtain a preliminary cell segmentation mask. Feature fusion is then performed to obtain an enhanced cell segmentation mask, thereby determining the cell segmentation result of the image to be segmented. This effectively improves cell segmentation efficiency, accuracy, and robustness. Therefore, it solves the problems of large model-based cell segmentation methods in related technologies, which require a large amount of computational resources for training and inference, increasing model training time and computational burden, resulting in low cell segmentation efficiency and inability to accurately depict cell boundaries, thus reducing the accuracy and robustness of cell segmentation.

[0088] Next, referring to the accompanying drawings, a pathological image cell segmentation device based on large models and knowledge distillation, according to an embodiment of this application, is described.

[0089] Figure 7 This is a block diagram of a pathological image cell segmentation device based on large models and knowledge distillation, according to an embodiment of this application.

[0090] like Figure 7 As shown, the pathological image cell segmentation device 10 based on large model and knowledge distillation includes: acquisition module 100, generation module 200, extraction module 300 and processing module 400.

[0091] Specifically, the acquisition module 100 is used to acquire a target slice image that meets the first preset condition, and to generate a segmentation image that meets the second preset condition using the target slice image.

[0092] The generation module 200 is used to process target input prompt information that meets preset segmentation conditions based on the target prompt encoder, so as to generate target prompt feature vectors.

[0093] The extraction module 300 is used to extract the first image features of the image to be segmented using the first target image encoder, and based on the first image features, determine the second target image encoder, and use the second target image encoder to extract the second image features of the image to be segmented.

[0094] The processing module 400 is used to input the target prompt feature vector, the first image feature, and the second image feature into the target mask decoder to obtain a preliminary cell segmentation mask, and to fuse the preliminary cell segmentation mask, the first image feature, and the second image feature to obtain an enhanced cell segmentation mask, so as to use the enhanced cell segmentation mask to determine the pathological image cell segmentation result of the image to be segmented.

[0095] Optionally, in one embodiment of this application, the acquisition module 100 includes: a first acquisition unit and a first determination unit.

[0096] The first acquisition unit is used to cut the target slice image to obtain a target cut image that meets the preset size.

[0097] The first determining unit is used to normalize the target segmentation image to obtain a segmentation image that meets the second preset condition.

[0098] Optionally, in one embodiment of this application, the generation module 200 includes: a second determining unit, a second obtaining unit, and a first generating unit.

[0099] The second determining unit is used to determine multiple learnable vectors based on the input prompt information, and to determine multiple target points corresponding to the multiple learnable vectors based on the label information in the input prompt information.

[0100] The second acquisition unit is used to calculate multiple learnable vectors to obtain the feature vector of the input prompt information.

[0101] The first generation unit is used to generate target cue feature vectors based on the label of each target point, multiple learnable vectors, and feature vectors.

[0102] Optionally, in one embodiment of this application, the extraction module 300 includes a second generation unit and a third acquisition unit.

[0103] The second generation unit is used to generate a target small image encoder using a target large image encoder based on knowledge distillation technology and SAM.

[0104] The third acquisition unit is used to train a target small image encoder using the first image features to obtain a second target image encoder.

[0105] Optionally, in one embodiment of this application, the processing module 400 includes: a third determining unit, a first processing unit, a second processing unit, and a fourth determining unit.

[0106] The third determining unit is used to extract the target local features of the preliminary cell segmentation mask, and based on the target local features, determine the first local features of the first image features and the second local features of the second image features.

[0107] The first processing unit is used to splice the first local features and the second local features to generate a first fused feature map, and to calculate the global features of the first fused feature map to obtain a global feature vector.

[0108] The second processing unit is used to extend the global feature vector to the target space dimension, and to fuse the extended global feature vector with the first fused feature map to obtain the second fused feature map.

[0109] The fourth determining unit is used to convolve the second fusion feature map to generate the third fusion feature map, and to generate spatial attention weights using the third fusion feature map, so as to obtain an enhanced cell segmentation mask using the spatial attention weights.

[0110] Optionally, in one embodiment of this application, the processing module 400 includes a fourth acquisition unit and a third processing unit.

[0111] The fourth acquisition unit is used to perform noise reduction processing on the enhanced cell segmentation mask in order to obtain a first target cell region that meets the preset processing conditions.

[0112] The third processing unit is used to process the first target cell region as a false positive region to obtain the pathological image cell segmentation result of the image to be segmented.

[0113] It should be noted that the foregoing explanation of the embodiment of the pathological image cell segmentation method based on large model and knowledge distillation also applies to the pathological image cell segmentation device based on large model and knowledge distillation in this embodiment, and will not be repeated here.

[0114] The pathological image cell segmentation device based on large models and knowledge distillation proposed in this application can generate an image to be segmented using a target slice image. Based on a target cue encoder, it processes target input cue information that meets segmentation conditions to generate a target cue feature vector. Then, a first target image encoder extracts the first image features of the image to be segmented, thereby determining a second target image encoder to extract the second image features of the image to be segmented. The target cue feature vector, the first image features, and the second image features are input into a target mask decoder to obtain a preliminary cell segmentation mask. Feature fusion is then performed to obtain an enhanced cell segmentation mask, thereby determining the cell segmentation result of the image to be segmented. This effectively improves cell segmentation efficiency, accuracy, and robustness. Therefore, it solves the problems in related technologies where large model-based cell segmentation methods require large amounts of computational resources for training and inference, increasing model training time and computational burden, resulting in low cell segmentation efficiency and inability to accurately depict cell boundaries, thus reducing the accuracy and robustness of cell segmentation.

[0115] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may include:

[0116] The memory 801, the processor 802, and the computer program stored on the memory 801 and capable of running on the processor 802.

[0117] When the processor 802 executes the program, it implements the pathological image cell segmentation method based on large model and knowledge distillation provided in the above embodiments.

[0118] Furthermore, electronic devices also include:

[0119] Communication interface 803 is used for communication between memory 801 and processor 802.

[0120] The memory 801 is used to store computer programs that can run on the processor 802.

[0121] The memory 801 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0122] If the memory 801, processor 802, and communication interface 803 are implemented independently, then the communication interface 803, memory 801, and processor 802 can be interconnected via a bus to complete communication between them. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be divided into address buses, data buses, control buses, etc. For ease of representation, Figure 8 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0123] Optionally, in a specific implementation, if the memory 801, processor 802, and communication interface 803 are integrated on a single chip, then the memory 801, processor 802, and communication interface 803 can communicate with each other through an internal interface.

[0124] The processor 802 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.

[0125] This embodiment also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for pathological image cell segmentation based on large models and knowledge distillation.

[0126] This embodiment also provides a computer program product, including a computer program that, when executed, is used to implement the above-described method for segmenting pathological images based on large models and knowledge distillation.

[0127] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0128] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0129] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0130] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0131] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0132] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0133] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0134] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A pathological image cell segmentation method based on a large model and knowledge distillation, characterized by, The method comprises the following steps: Collecting a target slice image meeting a first preset condition, and generating a to-be-segmented image meeting a second preset condition by using the target slice image; Processing target input prompt information meeting a preset segmentation condition by using a target prompt encoder to generate a target prompt feature vector; Extracting a first image feature of the to-be-segmented image by using a first target image encoder, and determining a second target image encoder based on the first image feature, and extracting a second image feature of the to-be-segmented image by using the second target image encoder; Inputting the target prompt feature vector, the first image feature and the second image feature into a target mask decoder to obtain a preliminary cell segmentation mask, and performing feature fusion on the preliminary cell segmentation mask, the first image feature and the second image feature to obtain an enhanced cell segmentation mask, and determining a pathological image cell segmentation result of the to-be-segmented image by using the enhanced cell segmentation mask.

2. The method of claim 1, wherein, The generating of the to-be-segmented image meeting the second preset condition by using the target slice image comprises: Cutting the target slice image to obtain a target cut image meeting a preset size; Performing normalization processing on the target cut image to obtain the to-be-segmented image meeting the second preset condition.

3. The method of claim 1, wherein, The processing of the target input prompt information meeting the preset segmentation condition by using the target prompt encoder to generate the target prompt feature vector comprises: Determining a plurality of learnable vectors according to the input prompt information, and determining a plurality of target points corresponding to the plurality of learnable vectors according to label information in the input prompt information; Calculating the plurality of learnable vectors to obtain a feature vector of the input prompt information; Generating the target prompt feature vector based on the label of each target point, the plurality of learnable vectors and the feature vector.

4. The method of claim 1, wherein, The determining of the second target image encoder based on the first image feature comprises: Generating a target small image encoder based on a target large image encoder of a general segmentation model SAM by using a knowledge distillation technology; Training the target small image encoder by using the first image feature to obtain the second target image encoder.

5. The method of claim 1, wherein, The feature fusion of the preliminary cell segmentation mask, the first image feature and the second image feature to obtain the enhanced cell segmentation mask comprises: Extracting a target local feature of the preliminary cell segmentation mask, and determining a first local feature of the first image feature and a second local feature of the second image feature based on the target local feature; Splicing the first local feature and the second local feature to generate a first fusion feature map, and calculating a global feature of the first fusion feature map to obtain a global feature vector; Expanding the global feature vector to a target space dimension, and performing feature fusion on the expanded global feature vector and the first fusion feature map to obtain a second fusion feature map; ​ The second fusion feature map is convolved to generate a third fusion feature map, and a spatial attention weight is generated using the third fusion feature map, so that the enhanced cell segmentation mask is obtained using the spatial attention weight.

6. The method of claim 1, wherein, The pathological image cell segmentation result of the image to be segmented is determined using the enhanced cell segmentation mask, including: The enhanced cell segmentation mask is denoised to obtain a first target cell region that satisfies a preset processing condition; The first target cell region is subjected to pseudo-positive region processing to obtain the pathological image cell segmentation result of the image to be segmented. 7.A pathological image cell segmentation device based on a large model and knowledge distillation, characterized by, Including: The acquisition module is configured to acquire a target slice image that satisfies a first preset condition, and generate an image to be segmented that satisfies a second preset condition using the target slice image; The generation module is configured to process target input prompt information that satisfies a preset segmentation condition based on a target prompt encoder to generate a target prompt feature vector; The extraction module is configured to extract a first image feature of the image to be segmented using a first target image encoder, determine a second target image encoder based on the first image feature, and extract a second image feature of the image to be segmented using the second target image encoder; The processing module is configured to input the target prompt feature vector, the first image feature, and the second image feature into a target mask decoder to obtain a preliminary cell segmentation mask, and perform feature fusion on the preliminary cell segmentation mask, the first image feature, and the second image feature to obtain an enhanced cell segmentation mask, and determine a pathological image cell segmentation result of the image to be segmented using the enhanced cell segmentation mask.

8. An electronic device, comprising: Including: A memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the program to implement the pathological image cell segmentation method based on a large model and knowledge distillation according to any one of claims 1-5.

9. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the pathological image cell segmentation method based on a large model and knowledge distillation according to any one of claims 1-5.

10. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the pathological image cell segmentation method based on a large model and knowledge distillation according to any one of claims 1-5.

Images