SAM-based method and system for segmentation and classification of pleural and peritoneal effusion cells

Through the SAM-based HSV-SAM model and Transformer model, the abnormal cells in the pathological smear of thoracic ascites are automatically segmented and classified, which solves the problem of time-consuming and subjectiveness of manual interpretation in the prior art, and improves the accuracy of diagnosis and efficiency.

CN118781594BActive Publication Date: 2024-11-12JIANGXI YIZHICHU MEDICAL PATHOLOGICAL DIAGNOSIS MANAGEMENT CO LTD

Patent Information

Application Number
CN202411270505.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-11
Publication Date
2024-11-12
Estimated Expiration
2044-09-11

AI Technical Summary

Technical Problem

In the prior art, the analysis of abnormal cells of thoracic ascites smears relies on manual interpretation, which consumes time, is inefficient and has a high subjectivity, resulting in poor diagnostic accuracy.

Method used

Using SAM-based method, the pathological smear of thoracic ascites was segmented through the HSV-SAM model, and abnormal cells were identified and classified using the Transformer model. Combined with the HSV prompt encoder and mask decoder, the segmentation threshold was automatically calculated and nucleated cells were screened out. .

Benefits of technology

It improves the accuracy and efficiency of cell segmentation in pathological smears of thoracic ascites, reduces the subjectivity of manual interpretation, meets the clinical needs of pathologists, and achieves faster and more accurate diagnosis.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118781594B_ABST
    Figure CN118781594B_ABST
Patent Text Reader

Abstract

The present invention relates to a method and system for segmenting and classifying pleural and ascites cells based on SAM, the method comprising: performing Diff fast staining on collected pleural and ascites samples to obtain pleural and ascites pathological smears; segmenting the pleural and ascites pathological smears using an HSV-SAM model to screen out nucleated cell images; the HSV-SAM model comprises an image encoder, an HSV prompt encoder and a mask decoder; and classifying nucleated cell images based on a Transformer model. The present invention uses SAM improved by HSV color space for pleural and ascites pathological smears, and deeply identifies and marks the nucleated cell images segmented by the Transformer model to ensure the accuracy of the classification results.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of image target detection, and in particular to a method and system for segmenting and classifying pleural and ascites cells based on SAM. Background Art

[0002] Pleural effusion and ascites are common signs of progression and recurrence of malignant tumors, and are also important parameters for determining tumor stage and progression. Malignant pleural effusion and ascites are common complications in patients with advanced malignant tumors. The main feature is the rapid and abnormal increase of body cavity fluid, which can even be life-threatening in severe cases. Pleural effusion and ascites samples have the advantages of less sampling damage, repeatable acquisition, and short examination time, and have been widely used in the examination of advanced malignant tumors. However, conventional cytological smears are difficult to provide information such as tumor pathological histological type and site of onset, and the accuracy of diagnosis is poor.

[0003] Currently, the analysis of abnormal cells in pleural and peritoneal effusion smears is mainly based on manual interpretation by pathologists under a microscope. However, manual interpretation by pathologists contains the following problems:

[0004] (1) Manual interpretation is time-consuming and inefficient, and long-term diagnostic work can easily cause fatigue;

[0005] (2) Manual interpretation is subjective to a certain extent, and different doctors may have different interpretations of the same pleural effusion sample.

[0006] In recent years, the rapid development of artificial intelligence and its wide application in the field of medical imaging have provided a new direction for the identification of tumor cells in pleural and peritoneal effusion smears. Summary of the invention

[0007] In view of the above-mentioned state of the prior art, the purpose of the present invention is to provide a SAM-based pleural effusion and ascites cell segmentation and classification method and system, which segments pleural effusion pathological smears through the HSV-SAM model, screens out nucleated cells through the HSV prompt encoder, and finally identifies and classifies abnormal cells in pleural effusion pathological smears through the Transformer model to meet the needs of pathologists in clinical work.

[0008] The present invention is implemented by the following technical scheme: A method for segmenting and classifying pleural and peritoneal effusion cells based on SAM, comprising the following steps:

[0009] Step S1: Perform Diff's rapid staining on the collected pleural and ascites samples to obtain pleural and ascites pathological smears;

[0010] Step S2: segment the pleural and ascites pathological smears using the HSV-SAM model to screen out nucleated cell images; the HSV-SAM model includes an image encoder, an HSV hint encoder, and a mask decoder;

[0011] Step S2-1: Use the image encoder to calculate image embedding and extract image features;

[0012] Step S2-2: Obtain a pixel value histogram from the pleural effusion image and extract key pixel points from it; construct a convolutional neural network model by distinguishing the pixel points of nucleated cells from the pixel points outside the nucleated cells. The model, as a prompt generation model, can automatically calculate the optimal segmentation threshold on the saturation component;

[0013] Step S2-3: The trained hint generation model is used as the HSV hint encoder to replace the original hint encoder to improve the segmentation mask output; according to the optimal segmentation threshold, the HSV hint encoder regards the area with grayscale value greater than the optimal segmentation threshold as the cell nucleus, and removes the area with grayscale value less than the optimal segmentation threshold to obtain nucleated cells;

[0014] Step S2-4: Use the mask decoder to output the mask image of the region of interest, that is, the nucleated cell image, calculate the centroid of each mask image, intercept a 128×128 pixel color block with the centroid as the center point, and evaluate the segmentation effect of the nucleated cells;

[0015] Step S3: Classify nucleated cell images based on the Transformer model.

[0016] Further optimization suggests that the generative model divides the pleural and ascites pathological smears into three levels: H, S, and V, where H is hue, S is saturation, and V is tone; the difference between the cell nucleus and the background is most obvious in the saturation component, where the cell nucleus is white (the grayscale value tends to 255) and the background is black (the grayscale value tends to 0); the generative model is prompted to generate a pixel intensity histogram on the saturation component; the area occupied by the cell nucleus is small, and the area occupied by the background is large, so the cell nucleus occupies fewer pixels.

[0017] Further preferably, the pleural and ascites pathological smears are manually annotated to obtain a pleural and ascites pathological smear dataset, and the dataset is divided into a training set and a validation set for HSV-SAM model and Transformer model training.

[0018] Further preferably, the classifying of nucleated cell images based on the Transformer model comprises the following steps:

[0019] Step S3-1: Acquire a color block data set of nucleated cells;

[0020] Step S3-2: performing data enhancement on the color block;

[0021] Step S3-3: performing slice prediction training on the color block slices after data enhancement based on the Transformer model to obtain a nucleated cell classification prediction model;

[0022] Step S3-4: Evaluate the classification results of the Transformer model for nucleated cells.

[0023] Further preferably, the Transformer model consists of an encoder, a decoder, a multi-head self-attention mechanism, position encoding, residual connection, layer normalization, a feedforward neural network, a linear transformation and a Softmax layer.

[0024] The present invention also provides a system for the segmentation and classification of pleural and ascites cells based on SAM, comprising:

[0025] A sample acquisition module is used to perform Diff's rapid staining on the collected pleural and ascites samples to obtain pleural and ascites pathological smears;

[0026] An image segmentation module is used to segment the pathological smears of pleural and ascites using the HSV-SAM model and filter out nucleated cell images;

[0027] The classification module is used to classify nucleated cell images based on the Transformer model.

[0028] The beneficial effects of the present invention are as follows: the SAM model can segment all objects in the pathological smear of pleural and ascites, but cannot distinguish the region of interest (i.e., cells). Therefore, the HSV prompt encoder is used to replace the original prompt encoder to obtain the HSV-SAM model. The HSV-SAM model normalizes the pathological smear of pleural and ascites in the HSV space, makes a pixel histogram of the stained image of pleural and ascites, and automatically determines the segmentation threshold according to the Otsu method to distinguish the cell nucleus from the background in the saturation component; the difference between the cell nucleus and the background in the saturation component is the most obvious. According to the pixel histogram, the optimal segmentation threshold is determined to be a gray value of 120, that is, the area with a gray value greater than 120 is the cell nucleus, and the area with a gray value less than 120 is removed. Finally, the abnormal cells in the pathological smear of pleural and ascites are identified and classified by the Transformer model to meet the needs of pathologists in clinical work. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is the HSV-SAM model structure diagram.

[0030] Figure 2 are examples of results from various types of segmentation models.

[0031] Figure 3 This is the comparison result between the SAM model and other segmentation models.

[0032] Figure 4 It is the comparison of negative / positive cell recall rate.

[0033] Figure 5 It is a comparison of the results of various classification models.

[0034] Figure 6 It is the Grad-CAM result of various classification models. DETAILED DESCRIPTION

[0035] The present invention is further described below in conjunction with the embodiments, but the protection scope of the present invention is not limited thereto. Example 1

[0036] A method for segmenting and classifying pleural and peritoneal effusion cells based on SAM comprises the following steps:

[0037] Step S1: Performing Diff's rapid staining on the collected pleural and ascites samples to obtain pleural and ascites pathological smears; in this embodiment, pleural and ascites pathological sections of 400 patients with respiratory and gastrointestinal tract related cancers were collected, and the pleural and ascites pathological sections were photographed as pleural and ascites pathological smears, and the resolution of each pleural and ascites pathological smear was 1024×1024 pixels;

[0038] Step S2: Use the HSV-SAM model to segment the pathological smear of pleural and ascites to screen out the nucleated cell images; Figure 1 As shown in the figure, the HSV-SAM model includes an image encoder, an HSV hint encoder and a mask decoder; the image encoder is a trained big data model used to identify image information; the HSV hint encoder replaces the original hint encoder, normalizes the pleural and ascites pathological smear in the HSV space, makes a pixel histogram of the pleural and ascites staining image, extracts key pixels from it, and constructs a convolutional neural network (CNN) model, which is used as a hint generation model to automatically calculate the optimal segmentation threshold on the saturation component; the mask decoder outputs a mask image of the region of interest through decoding; the specific process is as follows:

[0039] Step S2-1: Use the image encoder to calculate image embedding and extract image features;

[0040] Step S2-2: Obtain a pixel value histogram from the pleural effusion image and extract key pixel points from it. By distinguishing the pixel points of nucleated cells from the pixel points outside the nucleated cells, a convolutional neural network (CNN) model is constructed. This model, as a prompt generation model, can automatically calculate the optimal segmentation threshold on the saturation component;

[0041] Step S2-3: The trained hint generation model is used as the HSV hint encoder to replace the original hint encoder to improve the segmentation mask output; according to the optimal segmentation threshold, the HSV hint encoder regards the area with grayscale value greater than the optimal segmentation threshold as the cell nucleus, and removes the area with grayscale value less than the optimal segmentation threshold to obtain nucleated cells;

[0042] Step S2-4: Use the mask decoder to output the mask image of the region of interest, that is, the nucleated cell image, calculate the centroid of each mask image, intercept a 128×128 pixel color block with the centroid as the center point, and evaluate the segmentation effect of the nucleated cells;

[0043] Step S3: Classify the nucleated cell images based on the Transformer model. The process includes:

[0044] Step S3-1: Acquire a color block data set of nucleated cells;

[0045] Step S3-2: performing data enhancement on the color block;

[0046] Step S3-3: performing slice prediction training on the color block slices after data enhancement based on the Transformer model to obtain a nucleated cell classification prediction model;

[0047] Step S3-4: Evaluate the classification results of the Transformer model for nucleated cells.

[0048] In this embodiment, the Transformer model consists of an encoder, a decoder, a multi-head self-attention mechanism, position encoding, residual connection, layer normalization, a feedforward neural network, a linear transformation, and a Softmax layer. The encoder and the decoder are the main structures of the Transformer model, and deep learning is achieved by stacking multiple identical layers; the encoder is responsible for reading the input sequence and extracting features, while the decoder generates the target sequence based on the output of the encoder and the generated output sequence. The multi-head self-attention mechanism allows the Transformer model to focus on multiple positions in the sequence at the same time, enhancing the ability to capture different types of relationships. Positional Encoding provides the Transformer model with the order information of words in the sequence, solving the limitation of the Transformer's lack of sequence processing capabilities. Residual Connection helps alleviate the gradient vanishing problem in deep networks by adding inputs to the outputs of each layer. Layer Normalization normalizes the activation values ​​of each layer to stabilize and accelerate the training process. The Feed Forward Neural Network further performs nonlinear transformation on the data after the self-attention mechanism, enhancing the expressive power of the Transformer model. Finally, the linear transformation and Softmax layer in the decoder convert the output into a probability distribution for predicting the next word in the sequence. The collaborative work of these components makes the Transformer model perform well in processing data.

[0049] The nucleated cell images of pleural and ascites pathological smears were manually annotated to obtain a pleural and ascites pathological smear dataset, which was then divided into a training set and a validation set for HSV-SAM model and Transformer model training.

[0050] Furthermore, the training process of the Transformer model is as follows: the nucleated cell images obtained after segmentation by the HSV-SAM model are used as the training set, and the training ratio is adjusted to 0.8 for testing. For each cell, we select one of the feature points as the key point and extract a cell image with a pixel size of 128×128 pixels centered on the key point. In order to avoid extracting as many identical cell areas as possible, the distance between different feature points should be greater than half of the cell radius. Finally, the number of data (batch-size) passed to the program for training at a single time is adjusted to 40, and it is completed after 20 iterations.

[0051] In order to further identify the cells in the pathological smears of pleural and ascites, the pathological smears of pleural and ascites were normalized in the HSV color space. Through analysis, it was found that the difference between the cell nucleus and the background was the most significant in the saturation component: the cell nucleus appeared white (grayscale value close to 255), while the background appeared black (grayscale value close to 0). Based on this finding, the saturation component was chosen to distinguish the cell nucleus from the background in the image.

[0052] Subsequently, by distinguishing the pixels of nucleated cells from those outside nucleated cells, a convolutional neural network (CNN) model was constructed. Using this model as a prompt encoder, it was found that the pixels were mainly concentrated below the grayscale value of 120, indicating that the area below this segmentation threshold was mainly the background. Therefore, 120 was determined as the segmentation threshold for distinguishing cell nuclei from the background. Through this segmentation threshold, we identified the area with a grayscale value greater than 120 as the cell nucleus, and the area with a grayscale value less than 120 as the background and removed it. Through this series of processing, the nucleated cell images in the region of interest were successfully extracted from the pleural and ascites pathological smear data, totaling 31,304 images.

[0053] In order to verify the effectiveness of image segmentation technology, 50 high-resolution (1024×1024 pixels) pleural and peritoneal effusion pathological smears were selected, and experienced pathologists were invited to annotate the abnormal cells (positive cells) and normal cells (negative cells) in these sections. Figure 2 a, b, c, d) marked in A, and the segmentation results of two types of segmentation models, SAML, Mask-RCNN, and Groud Truth (manually labeled) are given.

[0054] Several key evaluation indicators are used to measure the quality of the segmentation results, including mean intersection over union (MioU), precision, and recall.

[0055] Next, three Segment Anything techniques based on the SAM model and the HSV screening method (labeled as SAML, SAMB, and SAMH, respectively) were compared with the Mask-RCNN segmentation method to evaluate their advantages and disadvantages in segmentation effects ( Figure 3). Among them, SAML (Low) refers to the low-resource version of the SAM model, which has fewer parameters and can reduce memory and storage requirements; SAMB (Base) refers to the basic version of the SAM model, which has a moderate number of parameters and network complexity, balances the requirements of performance and computing resources, and is the most common model version; SAMH (High) refers to the high-performance version of the SAM model, which has an optimized network structure and algorithm to improve operating efficiency and reduce computing resource consumption. The comparison analysis results show that after replacing the original prompt encoder with the HSV prompt encoder, the MioU (mean intersection over union) and accuracy of the HSV-SAM model have been significantly improved. This shows that through the screening process, the HSV-SAM model has become more accurate than the SAM model in identifying regions of interest in image data, thereby improving the overall segmentation quality. Specifically, the improvement in the mean intersection over union means that the model has enhanced the recognition accuracy of each type of target when predicting. The significant improvement in accuracy reflects that the model has reduced the number of false positives in the recognition process, that is, it has reduced the number of cases where the background or other non-target areas are mistakenly identified as targets. It is noteworthy that despite the HSV selection, the recall rate remained almost unchanged, indicating that the selection process did not result in missing any real target cells, that is, the SAM model combined with HSV selection remained highly efficient in identifying all actual positive cells.

[0056] Recall is a key metric that measures the model's ability to identify all actual positive examples (positive cells). A higher value indicates a lower missed diagnosis rate. In the evaluation of the three SAM models, their recall rates did not exceed 0.9, suggesting room for further improvement. Therefore, the recall rates of positive cells and negative cells were calculated separately ( Figure 4 ). In-depth analysis of the recognition performance of the three SAM models for positive and negative cells shows that the recall rate of positive cells has been significantly improved compared with the previous one, which indicates that the model's ability to detect positive cells has been enhanced. At the same time, the recall rate of negative cells has slightly decreased, which may be attributed to the high proportion of negative cells in the population. Nevertheless, this small decrease does not affect the positive evaluation of the overall performance of the model, as it reflects that the model is more accurate in distinguishing positive cells. Overall, the three SAM models showed higher accuracy and reliability in identifying key positive cells.

[0057] After completing the reliability verification of the segmentation results, senior pathologists were invited to carefully classify the 31,304 nucleated cell images obtained through the segmentation process, and clearly divide them into two categories: abnormal cells (positive cells) and normal cells (negative cells). In order to improve the generalization and robustness of the model, data augmentation technology was used to expand the original nucleated cell images to 125,216. This step not only increases the size of the dataset, but also helps to train a more robust model by introducing image diversity. Subsequently, the expanded dataset was randomly divided into training and test sets according to a ratio of 8:2. This division method ensures that the model can be exposed to enough samples during the training process, while retaining a portion of the data for evaluating the performance of the model on unseen data.

[0058] In order to verify the classification effect, ROC curve, AUC value, confusion matrix and Grad-CAM are used as indicators to evaluate the classification effect. Among them, the surface area formed by the ROC curve and the diagonal line (area under the curve AUC) can also reflect the authenticity of the detection method. The closer the AUC value is to 1.0, the higher the authenticity of the detection method; when it is equal to 0.5, the authenticity is the lowest and has no practical value; the confusion matrix is ​​a performance evaluation tool commonly used in classification problems, which can help us understand the performance of the model in various categories, and can also reveal the possible deviations or error types of the model, thereby providing direction for the improvement of the model; Grad-CAM can visually interpret the image visually, making it easier to find areas of interest in the picture.

[0059] In this study, an in-depth comparative analysis was conducted to evaluate the performance of different deep learning models on specific tasks. For this purpose, three widely recognized convolutional neural network (CNN) architectures were selected - VGG-16, ResNet (residual network) and AlexNet, and their performance was compared with the Transformer model ( Figure 5). It can be found that the overall effect of Alex Net and Transformer is better than that of VGG16 and ResNet. Among Alex Net and Transformer, Transformer has a 3.22% improvement in accuracy compared with Alex Net. This result suggests the high accuracy of the Transformer model in identifying abnormal cells; in addition, the analysis of the confusion matrix further confirms the advantages of the Transformer model. The accuracy of the Transformer model in identifying normal cells reached 0.955, and the accuracy in identifying abnormal cells was also as high as 0.897. These data not only highlight the reliability of the Transformer model in cell recognition tasks, but also show its potential in improving diagnostic accuracy. In summary, the Transformer model has advantages in accuracy and robust performance shown in the confusion matrix.

[0060] In Grad-CAM, two main regions can be clearly seen: the region marked “a” represents positive cells, while the region marked “b” represents negative cells. In order to gain a deeper understanding of the recognition capabilities of different models for these cell types, the Grad-CAM (gradient weighted class activation mapping) results of the four classification models are further shown ( Figure 6 ).

[0061] Significant differences can be observed through Grad-CAM images, especially in emphasizing positive and negative cell features. The feature response of the positive cell region in the Grad-CAM image is more prominent and distinct, which indicates that the adopted model shows high sensitivity and accuracy in identifying positive cells. After comparing the Grad-CAM results of the four models, the Transformer model performs particularly well in identifying positive and negative cells. It not only provides clearer feature mapping in the positive cell region, but also shows good discrimination ability in the negative cell region, which further proves the superiority of the Transformer model in the cell recognition task. Example 2

[0062] In order to implement the SAM-based pleural effusion cell segmentation and classification method of Example 1, a system of the SAM-based pleural effusion cell segmentation and classification method is provided on the basis of Example 1, including:

[0063] A sample acquisition module is used to perform Diff's rapid staining on the collected pleural and ascites samples to obtain pleural and ascites pathological smears;

[0064] An image segmentation module is used to segment the pathological smears of pleural and ascites using the HSV-SAM model and filter out nucleated cell images;

[0065] The classification module is used to classify nucleated cell images based on the Transformer model.

[0066] The above only expresses the preferred embodiments of the present invention, and the description is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that for ordinary technicians in this field, without departing from the concept of the present invention, several modifications, improvements and substitutions can be made, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be based on the attached claims.

Claims

1. A method for segmenting and classifying pleural and peritoneal effusion cells based on SAM, characterized in that: The steps include: Step S1: Perform Diff's rapid staining on the collected pleural and ascites samples to obtain pleural and ascites pathological smears; Step S2: segment the pleural and ascites pathological smears using the HSV-SAM model to screen out nucleated cell images; the HSV-SAM model includes an image encoder, an HSV hint encoder, and a mask decoder; Step S2-1: Use the image encoder to calculate image embedding and extract image features; Step S2-2: Obtain a pixel value histogram from the pleural effusion image and extract key pixel points from it; construct a convolutional neural network model by distinguishing the pixel points of nucleated cells from the pixel points outside the nucleated cells. The model, as a prompt generation model, can automatically calculate the optimal segmentation threshold on the saturation component; Step S2-3: The trained cue generation model is used as an HSV cue encoder. According to the optimal segmentation threshold, the HSV cue encoder regards the area with a grayscale value greater than the optimal segmentation threshold as a cell nucleus, and removes the area with a grayscale value less than the optimal segmentation threshold to obtain nucleated cells; Step S2-4: Use the mask decoder to output the mask image of the region of interest, that is, the nucleated cell image, calculate the centroid of each mask image, intercept the color block with the centroid as the center point, and evaluate the segmentation effect of the nucleated cells; Step S3: Classify nucleated cell images based on the Transformer model.

2. The SAM-based pleural effusion cell segmentation and classification method according to claim 1, characterized in that: The generative model is prompted to divide the pleural and ascites pathological smears into three levels: H, S, and V. H is hue, S is saturation, and V is tone. In the saturation component, the cell nucleus is white and the background is black, which prompts the generative model to generate a pixel intensity histogram on the saturation component.

3. The SAM-based pleural effusion cell segmentation and classification method as claimed in claim 1, characterized in that: The pleural and ascites pathological smears were manually annotated to obtain the pleural and ascites pathological smear dataset, which was then divided into training set and validation set for HSV-SAM model and Transformer model training.

4. The SAM-based pleural effusion cell segmentation and classification method according to claim 1, characterized in that: Classifying nucleated cell images based on the Transformer model includes the following steps: Step S3-1: Acquire a color block data set of nucleated cells; Step S3-2: performing data enhancement on the color block; Step S3-3: performing slice prediction training on the color block slices after data enhancement based on the Transformer model to obtain a nucleated cell classification prediction model; Step S3-4: Evaluate the classification results of the Transformer model for nucleated cells.

5. A system for implementing the SAM-based pleural effusion and ascites cell segmentation and classification method according to any one of claims 1 to 4, characterized in that: include: A sample acquisition module is used to perform Diff's rapid staining on the collected pleural and ascites samples to obtain pleural and ascites pathological smears; An image segmentation module is used to segment the pathological smears of pleural and ascites using the HSV-SAM model and filter out nucleated cell images; The classification module is used to classify nucleated cell images based on the Transformer model.

Citation Information

Patent Citations

  • Cargo segmentation method and system based on SAM and YOLOV8n

    CN117475148A

  • Cervical liquid-based thin-layer cell image segmentation method based on SAM segmentation model

    CN117876401A

  • Digital image classification method for cervical fluid-based cells based on a deep learning detection model

    US20220083762A1

Cited By

  • Emulsion microscopic image droplet identification method based on multistage segmentation strategy

    CN121259817A