Weakly supervised pathological image segmentation method and system based on class activation map and arbitrary segmentation model

Abstract

The present invention discloses a weakly supervised pathological image segmentation method and system based on a class activation map and an arbitrary segmentation model. The method comprises: selecting a pathological image to be processed, and using a CAM generator with nerve tissue slices as a negative class to generate a significant area of ​​tumor tissue; defining and calculating three prior knowledge of basic appearance, spatial distribution, and feature difference to achieve adaptive point selection, thereby providing a more reliable initial prompt; and inputting the obtained initial prompt into the SAM to obtain the final segmentation result. The present invention effectively combines the CAM and SAM by designing an adaptive point selection module, which can significantly improve the segmentation accuracy of tumor tissue in the absence of pixel-level supervision. Experiments show that the tumor segmentation accuracy of this method on two independent data sets is improved by nearly 25%, and compared with the existing weakly supervised segmentation method, the performance is improved by more than 15%.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical image processing, and in particular relates to a weakly supervised pathological image segmentation method and system based on a class activation map and an arbitrary segmentation model. Background Art

[0002] Tissue pathology images are the gold standard for disease detection, diagnosis, and prognosis, and image segmentation is a fundamental task in pathology image analysis. Accurate tumor tissue segmentation is crucial for many clinical applications, including disease diagnosis, treatment planning, and monitoring disease progression. However, these images are often large and contain numerous elements, making analysis of them a challenging task.

[0003] In recent years, deep learning technology has made tremendous progress in the field of medical image analysis. Through techniques such as convolutional neural networks (CNNs) and the Transformer architecture, deep learning has significantly improved the accuracy and efficiency of medical image segmentation. Fully supervised learning uses manual annotation as the gold standard for delineating anatomical structures and pathological regions, but such methods rely heavily on extensive training and precise pixel-level labeling. Due to the large size of pathological images and the greater complexity of accurately annotating them compared to other medical images, only experienced pathologists can accurately segment these tissues, making such methods time-consuming and labor-intensive.

[0004] To address these challenges, weakly supervised learning methods are becoming an inevitable trend in pathology image processing. Weakly supervised semantic segmentation aims to learn the concept of segmentation using image-level category labels and has shown great potential in pathology image analysis. For example, Class Activation Maps (CAMs) help identify representative regions in an image by visualizing the decision-making process of a neural network, thereby providing a deeper understanding of the neural network's response to different categories and revealing the underlying mechanisms behind these activations. However, due to the lack of pixel-level supervision, CAMs are often unable to accurately outline the boundaries of the identified objects.

[0005] Meanwhile, the Segment Anything Model (SAM), a foundational model for segmentation, has demonstrated remarkable effectiveness and efficiency in semantic segmentation tasks for natural images. Combining an image encoder, a hint encoder, and a mask decoder, the SAM model rapidly generates segmentation results based on user-provided hints (such as points, boxes, and text). Its zero-shot generalization capability enables it to handle image types or segmentation tasks not previously seen in the training set. However, SAM also exhibits limited robustness due to noise introduced by inaccurate hints.

[0006] In summary, deep learning technology in the field of medical image segmentation still faces challenges such as complex data annotation and insufficient model robustness. Further optimization of weakly supervised learning methods, improvement of zero-shot generalization and robustness of models, and exploration of more efficient segmentation frameworks are needed to better meet the needs of clinical applications. Summary of the Invention

[0007] The purpose of the present invention is to provide a weakly supervised pathological image segmentation method and system based on class activation map and arbitrary segmentation model, combining CAM and SAM to obtain a simple but effective weakly supervised segmentation network for medical images to achieve accurate segmentation of tumor tissue in pathological images.

[0008] The technical solution for achieving the purpose of the present invention is: a weakly supervised pathological image segmentation method based on class activation map and arbitrary segmentation model, comprising the following steps:

[0009] (1) Select the pathological image to be processed and use the CAM generator with nerve tissue slices as negative class to generate the salient area of ​​tumor tissue;

[0010] (2) Define and calculate the three prior knowledge of basic appearance, spatial distribution and feature difference to achieve adaptive point selection and obtain initial hints;

[0011] (3) Input the obtained initial hint into SAM to obtain the final segmentation result.

[0012] Furthermore, in step (1), the image classifier G used for tumor tissue slice recognition (with nerve tissue slices as negative class) is used as the class activation map generator, and the input image Processed by encoder G, W and H are the width and height of the image respectively, and the feature map is obtained , as follows:

[0013]

[0014] Among them, C=2 means there are two categories;

[0015] The image-level class prediction log-odds value The calculation formula is:

[0016]

[0017] Where GAP is the global average pooling layer along the spatial axis;

[0018] Using binary cross entropy loss function To train the encoder G, the formula of the binary cross entropy loss function is:

[0019]

[0020] in is the predicted tumor identification confidence, and is the image-level classification label.

[0021] Furthermore, in step (2), three prior knowledges, namely basic appearance, spatial distribution and feature difference, are defined and calculated to achieve adaptive point selection, thus providing more reliable initial hints;

[0022] In terms of basic appearance, and They represent the average appearance of tumor tissue and background respectively, and their specific expressions are:

[0023]

[0024] Among them, for the test image set , represents the test image, Represents the true label, L represents the number of test images, and x, y are the coordinates of the pixel points.

[0025] The basic appearance prior calculation formula is as follows:

[0026]

[0027]

[0028] in, It means that T is the center. is the neighborhood radius, represents the parameter controlling the radius,

[0029] and Indicates the maximum intensity of the pathological image.

[0030] The spatial distribution is calculated as follows:

[0031]

[0032] in, is the confidence threshold of the predicted CAM; the point set J of the predicted foreground category is obtained:

[0033]

[0034] The K-Means++ (KMPP) algorithm is used to generate the cluster centers of these foreground points. The number of clusters is set to 3, and three spatial distribution centers are obtained:

[0035]

[0036] The spatial distribution prior calculation formula is as follows:

[0037]

[0038]

[0039] in, Indicates As the center, is the neighborhood radius, represents the parameter controlling the radius,

[0040] represents the Euclidean distance between p and q.

[0041] The feature difference prior calculation formula is as follows:

[0042]

[0043]

[0044] is the cluster center generated by KMPP; AP and DP are appearance prior and distribution prior respectively; Is an arbitrarily selected reverse point, that is, a background point, which satisfies the following conditions: It is the point where S is equal to 0, where S is the spatial distribution.

[0045] Furthermore, in step (3), the forward point obtained is and negative points It is input into SAM as a hint to obtain the final segmentation result. The specific process is as follows:

[0046]

[0047] Where I represents the input image, Indicates a hint. represents the segmentation result, , and They represent the image encoder, hint encoder, and mask decoder in the SAM module respectively.

[0048] Based on the same inventive concept, the present invention also provides a weakly supervised pathological image segmentation system based on class activation maps and arbitrary segmentation models, which is used to implement the above method. The system includes:

[0049] The first module is used to select the pathological image to be processed and generate the significant area of ​​the tumor tissue through the CAM generator with the nerve tissue slice as the negative class;

[0050] The second module is used to define and calculate the three prior knowledge of basic appearance, spatial distribution and feature difference to achieve adaptive point selection and obtain the initial prompt;

[0051] The third module is used to input the obtained initial hint into SAM to obtain the final segmentation result.

[0052] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the steps of the above method are implemented when the processor executes the program.

[0053] Compared with the existing technology, the beneficial effects of the present invention are as follows: the present invention proposes a simple but effective weakly supervised segmentation network for medical images, which realizes the accurate segmentation of tumor tissue in pathological images by designing an adaptive point selection method to combine class activation mapping (CAM) and arbitrary segmentation model (SAM); it mainly includes three modules: CAM, adaptive point selection and SAM; the CAM module is adjusted to generate significant areas of tumor tissue in pathological images; adaptive point selection is achieved by integrating three prior knowledges of basic appearance, spatial distribution and feature differences, providing more reliable initial hints for subsequent SAM; the SAM module is used to predict the final segmentation result; the present invention can realize accurate segmentation of tumor tissue in pathological images without the need for extensive training and precise pixel-level labels. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 is a flow chart of a method according to an embodiment of the present invention.

[0055] Figure 2 1 is a comparison of three prior knowledge ablation experimental results in adaptive point selection according to an embodiment of the present invention.

[0056] Figure 3 This is a comparison of the segmentation results of the embodiment of the present invention with other CAM-based models.

[0057] Figure 4 This is a comparison of the segmentation results of other weakly supervised segmentation networks according to an embodiment of the present invention.

[0058] Figure 5 2 is a comparison of performance distribution of experiments according to embodiments of the present invention. DETAILED DESCRIPTION

[0059] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0060] Reference Figure 1The present invention proposes a weakly supervised pathological image segmentation method based on class activation map and arbitrary segmentation model, comprising the following steps:

[0061] Step (1) selects the pathological image to be processed and uses the CAM generator with nerve tissue slices as negative class to generate the significant area of ​​tumor tissue.

[0062] The image classifier G is designed as a CAM generator, which uses neural tissue blocks as negative classes to identify tumor tissue blocks from input pathological images. ResNet-18 is used as the backbone network of the image classifier G.

[0063] Given an input image , where W and H are the width and height of the image respectively, and 3 represents the RGB channels of the image. The feature map is obtained by processing the classifier G. , the process can be expressed as:

[0064]

[0065] Among them, C=2 means there are two categories, and F contains the activation intensity of each category at each position in the image.

[0066] Next, the feature map F is processed by the Global Average Pooling (GAP) layer to obtain the image-level category prediction logarithmic probability value Global average pooling is a feature simplification method that averages feature maps along the spatial axes (i.e. width and height) to obtain a single prediction value for each category. The process can be expressed as:

[0067]

[0068] where GAP is a global average pooling layer along the spatial axis.

[0069] Using binary cross entropy loss function To train the classifier G, the formula for the binary cross entropy loss function is:

[0070]

[0071] in is the predicted tumor identification confidence, and is the image-level classification label. This loss function is used to measure and By minimizing this loss function, the model can learn how to accurately identify tumor tissue blocks from input images. This process involves adjusting the parameters of the model to maximize the probability of correct classification on a given image.

[0072] In step (2), three prior knowledge, namely basic appearance, spatial distribution and feature difference, are defined and calculated to achieve adaptive point selection, thus providing more reliable initial hints.

[0073] Regarding the basic appearance, since there are obvious differences in appearance features between tumors and surrounding areas (such as background) in pathological images, such as intensity and texture, a reliable positive point hint should meet certain appearance requirements. This means that in order to generate effective segmentation hints, it is necessary to select points that are similar in appearance to tumor tissue. ,in represents the test image, Represents the true label, and L represents the number of test images. The average appearance of tumor tissue and background is calculated, and the specific expressions are:

[0074]

[0075] in, and Represents the color or intensity characteristics of tumor tissue and background, respectively.

[0076] For any punctuality tips candidate , which should be located at the mean tumor representation within the adjacent exterior area and as far away as possible Therefore, the basic appearance prior calculation formula is defined as follows:

[0077]

[0078]

[0079] in, It means that T is the center. is the neighborhood radius, represents the parameter controlling the radius,

[0080] and Indicates the maximum intensity of the pathological image.

[0081] The distribution prior is to ensure that the punctuality cues are representative and evenly distributed. The spatial distribution is calculated as follows:

[0082]

[0083] in, Is the confidence threshold of the predicted CAM. Get the point set J of the predicted foreground category:

[0084]

[0085] The K-Means++ (KMPP) algorithm is used to generate the cluster centers of these foreground points. The number of clusters is set to 3, and three spatial distribution centers are obtained:

[0086]

[0087] The spatial distribution prior calculation formula is as follows:

[0088]

[0089]

[0090] in, Indicates As the center, is the neighborhood radius, represents the parameter controlling the radius,

[0091] represents the Euclidean distance between p and q.

[0092] Thus, points close to the cluster centers were selected as cue points, which enhanced the representativeness. This approach not only supported the systematic exploration of spatial features, but also ensured a diverse sampling of the study area.

[0093] By introducing negative point hints, the model can better understand the feature differences between foreground and background, thereby improving the accuracy of segmentation. The feature difference prior calculation formula is as follows:

[0094]

[0095]

[0096] in, are three randomly selected punctual prompts that meet the basic appearance prior and spatial distribution prior requirements, A negative point hint is randomly sampled from the background area.

[0097] Step (3), get the forward point and negative points It is input into SAM as a hint to obtain the final segmentation result.

[0098] The specific process is as follows:

[0099]

[0100] Where I represents the input image, Indicates a hint. represents the segmentation result, , and They represent the image encoder, hint encoder, and mask decoder in the SAM module respectively.

[0101] For SAM, the present invention uses the officially released version: ViT-H model. In the specific implementation, the parameters of the control radius are and Set to 0.5, and set the confidence threshold of the predicted CAM to Set to 0.5.

[0102] The following three specific examples are used to verify the implementation effect of the method of the present invention.

[0103] Example 1

[0104] This example evaluates the proposed weakly supervised tumor pathology image segmentation method on three independent datasets.

[0105] Training set: We collected 1,000 pathological H&E images from the public PAIP dataset to train the CAM module. Each image is 512x512 pixels in size. 500 of these images contain tumor tissue and are labeled as positive, while the remaining 500 contain neural tissue and are labeled as negative.

[0106] Test Set: The proposed method was evaluated on two independent datasets, one from Nanjing Drum Tower Hospital (NDTH) and the other from Jiangsu Provincial Hospital of Traditional Chinese Medicine (JHCM). The NDTH dataset contains 200 512x512 H&E images with pixel-level tumor tissue annotations. The JHCM dataset contains 100 512x512 H&E images with pixel-level tumor tissue annotations. All tumor tissues were manually annotated by two experienced pathologists.

[0107] This example uses the Dice similarity coefficient (Dice), Hausdorff distance (HD), false positive segmentation rate (FPSR), and false negative segmentation rate (FNSR) to evaluate the performance of tumor tissue segmentation. The Dice metric measures the degree of overlap between the segmentation result and the true label. The HD metric measures the accuracy of boundary segmentation. Its calculation method is as follows:

[0108]

[0109]

[0110]

[0111]

[0112]

[0113] Among them, M represents the predicted value and N represents the true value.

[0114] We conduct ablation experiments to verify the effectiveness of the three priors proposed in this paper. Table 1 shows the experimental results of the present invention on two datasets, where CP, AP, and DP represent contrast prior, appearance prior, and distribution prior, respectively.

[0115] Table 1 Ablation study results of the present invention on the NDTH training set and JHCM test set

[0116]

[0117] Figure 2 This figure shows a visual comparison of three ablation studies of prior knowledge in the proposed Adaptive Point Selection (APS) module. The top four test images are from the NDTH dataset, and the bottom four test images are from the JHCM dataset. CP, AP, and DP represent contrastive prior, appearance prior, and distribution prior, respectively. Figure 2 In the figure, column a is the initial image, column b is the segmentation results without CP, AP, and DP, column c is the segmentation results with CP, column d is the segmentation results with AP, column e is the segmentation results with DP, column f is the segmentation results with CP and AP, column g is the segmentation results with CP and DP, column h is the segmentation results with AP and DP, column i is the segmentation results of the present invention, that is, all three priors are present, and column j is the gold standard. It can be seen that the method of the present invention performs well in terms of overall segmentation accuracy.

[0118] Example 2

[0119] The effectiveness of the present invention is verified by comparing the experimental results of the present invention with those of CAM and its enhanced algorithms Grad-CAM and Grad-CAM++. Other experimental conditions are the same as those in Example 1.

[0120] Table 2 shows the segmentation experiment comparison results of the present invention, CAM and its enhanced algorithms Grad-CAM and Grad-CAM++. Figure 3 Visual comparison results of tumor segmentation between the present invention and other CAM-based methods on the NDTH dataset are shown.

[0121] Table 2 Experimental comparison results of the present invention and other segmentation methods

[0122]

[0123] Experiments show that compared with the CAM-based method, the model of the present invention achieves a Dice value improvement of more than 20% in tumor tissue segmentation.

[0124] Example 3

[0125] The present invention verifies the effectiveness of the present invention by comparing the experimental results of the present invention with other state-of-the-art weakly supervised segmentation networks, and other experimental conditions are the same as those in Example 1.

[0126] Table 3 shows the quantitative comparison results of the proposed method with other state-of-the-art weakly supervised segmentation networks (DeepLabV3+, Swin Transformer, and U-Net) on two independent test datasets. Figure 4 The visual comparison results of tumor segmentation between the present invention and these three weakly supervised segmentation networks on the JHCM dataset are shown. Figure 5 The performance distribution of the comparative experiments is shown. (1) is the NDTH dataset, (2) is the JHCM dataset; (a) CAM; (b) Grad-CAM; (c) Grad-CAM++; (d) DeeplabV3+; (e) Swin Transformer; (f) U-Net; (g) the proposed method.

[0127] Table 3 Experimental comparison results of the proposed method and other state-of-the-art weakly supervised segmentation networks

[0128]

[0129] Experiments show that compared with the previous state-of-the-art weakly supervised segmentation method, the present invention improves the performance by more than 15%.

[0130] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the invention can be embodied in other specific forms without departing from the spirit or essential characteristics of the invention. The embodiments should therefore be considered illustrative and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations that come within the meaning and range of equivalents of the claims be embraced herein.

[0131] In addition, it should be understood that although this specification is described in terms of implementation methods, not every implementation method contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

Claims

1. A weakly supervised pathological image segmentation method based on class activation map and arbitrary segmentation model, characterized in that: The following steps are involved: (1) Select the pathological image to be processed and use the CAM generator with nerve tissue slices as negative class to generate the salient area of ​​tumor tissue; The image classifier G for tumor tissue slice recognition is used as the class activation map generator, and the input image Processed by encoder G, W and H are the width and height of the image respectively, and the feature map is obtained , as follows: ； Among them, C=2 means there are two categories; Image-level class prediction log odds The calculation formula is: ； Where GAP is the global average pooling layer along the spatial axis; Using binary cross entropy loss function To train the encoder G, the formula of the binary cross entropy loss function is: ； in is the predicted tumor identification confidence, is the image-level classification label; (2) Define and calculate the three prior knowledge of basic appearance, spatial distribution and feature difference to achieve adaptive point selection and obtain the initial prompt, specifically: In terms of basic appearance, and They represent the average appearance of tumor tissue and background respectively, and their expressions are: ； Among them, for the test image set , represents the test image, represents the true label, L represents the number of test images, and x, y are the coordinates of the pixel points; (3) Input the obtained initial hint into SAM to obtain the final segmentation result.

2. The weakly supervised pathological image segmentation method based on class activation map and arbitrary segmentation model according to claim 1, characterized in that The basic appearance prior calculation formula is as follows: ， ； in, It means that T is the center. is the neighborhood radius, represents the parameter controlling the radius, Indicates the maximum intensity of the pathological image.

3. The weakly supervised pathological image segmentation method based on class activation map and arbitrary segmentation model according to claim 2, characterized in that The spatial distribution is calculated as follows: ； in, is the confidence threshold of the predicted CAM; the point set J of the predicted foreground category is obtained: ； The K-Means++ algorithm is used to generate the cluster centers of these foreground points; the number of clusters is set to 3, and three spatial distribution centers are obtained: 。 4. The weakly supervised pathological image segmentation method based on class activation map and arbitrary segmentation model according to claim 3, characterized in that The spatial distribution prior calculation formula is as follows: ， ； in, Indicates As the center, is the neighborhood radius, represents the parameter controlling the radius, represents the Euclidean distance between p and q.

5. The weakly supervised pathological image segmentation method based on class activation map and arbitrary segmentation model according to claim 4 is characterized in that the feature The difference prior calculation formula is as follows: ； ； is the cluster center generated by KMPP; AP and DP are appearance prior and distribution prior respectively; Is an arbitrarily selected reverse point, that is, a background point, which satisfies the following conditions: It is the point where S is equal to 0, where S is the spatial distribution.

6. The weakly supervised pathological image segmentation method based on class activation map and arbitrary segmentation model according to claim 5, characterized in that In step (3), the forward point obtained and negative points Input it into SAM as a prompt to obtain the final segmentation result; the specific process is as follows: ； Where I represents the input image, Indicates a hint. represents the segmentation result, , and They represent the image encoder, hint encoder, and mask decoder in the SAM module respectively.

7. A weakly supervised pathological image segmentation system based on class activation maps and arbitrary segmentation models, characterized in that: For implementing the method described in any one of claims 1 to 6, the system comprises: The first module is used to select the pathological image to be processed and generate the significant area of ​​the tumor tissue through the CAM generator with the nerve tissue slice as the negative class; The second module is used to define and calculate the three prior knowledge of basic appearance, spatial distribution and feature difference to achieve adaptive point selection and obtain the initial prompt; The third module is used to input the obtained initial hint into SAM to obtain the final segmentation result.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the method according to any one of claims 1 to 6 are implemented.

Images