A Weakly Supervised Tissue Segmentation Method and Device for Pathological Images Based on Inaccurate Point Annotation of Fully Digital Pathological Sections
By using an inaccurate point annotation method for fully digital pathological slides, and pre-training single-label and multi-label classification networks to generate pseudo-labels, a segmentation network is constructed, solving the problems of high cost and low efficiency in pathological image segmentation and achieving fast and efficient semantic segmentation.
Patent Information
- Application Number
- CN202311038530.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-08-17
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2043-08-17
AI Technical Summary
Existing technologies for pathological image segmentation are costly and inefficient due to the need for extensive pixel-level annotations. In particular, the heterogeneity of tumors in fully digital pathological sections increases the annotation difficulty, and existing weakly supervised methods rely on image-level annotations, which increases economic and human costs.
We employ an inaccurate point annotation method based on fully digital pathological slides. By rapidly expanding point annotations to generate a single-label set, we use single-label and multi-label classification networks for pre-training to generate pseudo-labels. We then construct a segmentation network for semantic segmentation, reducing our reliance on pixel-level annotations.
It enables rapid and efficient tissue segmentation of pathological images under low magnification and large field of view, reduces annotation costs, improves work efficiency, and reduces reliance on image-level multi-label noise-free annotation.
Smart Images

Figure CN117152068B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method and apparatus for tissue segmentation of weakly supervised pathological images based on inaccurate point annotation of fully digital pathological slides. Background Technology
[0002] The tumor microenvironment (TME) plays a crucial role not only in tumor initiation and progression but also influencing treatment outcomes and prognosis in cancer patients. The TME is composed of different tissue types, including tumor epithelial cells, tumor-infiltrating lymphocytes (TILs), and tumor-associated stroma. Previous studies have demonstrated their close clinical relevance to further tumor development. Therefore, accurately distinguishing and segmenting these different tissue types within the TME is essential. Traditional segmentation methods utilize manually annotated pixel-level features for tissue segmentation, such as texture, morphological characteristics, and color.
[0003] Recently, deep learning has achieved great success in medical image segmentation tasks. However, obtaining large amounts of pixel-level annotations requires significant human and financial resources, especially for pathological images. Due to their diversity and complexity, the annotation of pathological images requires personnel with a clinical pathology background to process them. Furthermore, due to the heterogeneity of tumors, they can have various morphological appearances. At the same time, gigapixel-level fully digital pathological sections (WSI) also increase the difficulty of manual annotation.
[0004] In past research, researchers have explored various methods to alleviate the burden of image annotation during medical image segmentation. These typically employ weak annotation methods, which contain less information than pixel-level annotations, and utilize networks to extract richer semantic information from the annotations. This approach aims to improve annotation efficiency while maintaining segmentation performance, thereby reducing the annotation burden. Current weak annotation methods include image-level annotation, line annotation, bounding box annotation, and point annotation. Among these, image-level annotation is the mainstream annotation method for image classification tasks, and related segmentation tasks have seen the most research, while research on point annotation is relatively limited. Summary of the Invention
[0005] The main objective of this invention is to overcome the shortcomings and deficiencies of the prior art and provide a weakly supervised method and apparatus for tissue segmentation of pathological images based on inaccurate point annotation of fully digital pathological slides. This method enables rapid and inaccurate point annotation of pathological slides under low magnification and large field of view. Through the action of the weak supervision mechanism, it can achieve rapid and efficient tissue segmentation of pathological images while saving economic and labor costs.
[0006] To achieve the above objectives, the present invention adopts the following technical solution:
[0007] In a first aspect, the present invention provides a weakly supervised tissue segmentation method for pathological images based on inaccurate point annotations in fully digital pathological slides, comprising the following steps:
[0008] Pathological slides are acquired and rapid inaccurate point annotations are performed to accumulate inaccurate point annotations and form an inaccurate point annotation set. Any inaccurate point annotation in the inaccurate point annotation set is then expanded outwards to surrounding areas, forming sample image patch x and sample image set X. After feature filtering, image-level single-label p and single-label set with abstract features are generated. ;
[0009] Using a single tag set Pre-train a pre-defined single-label classification network to obtain an image-level pseudo-multi-class label set D. cls ;
[0010] Using image-level pseudo-multi-class label set D cls A pre-trained multi-label classification network is used to obtain pseudo-masks for the three intermediate layers and generate pseudo-labels for training the segmentation network. ;
[0011] Constructing a segmentation network Meanwhile, the segmentation network is trained under the supervision of the pseudo-labels. The segmentation result of the segmentation network and the loss of the pseudomask are calculated separately. The segmentation result map is obtained by weighted summation of the segmentation result and the loss of the pseudomask.
[0012] As a preferred technical solution, the step of expanding any inaccurate point annotation from the inaccurate point annotation set to the surrounding area to form a sample image block x and a sample image set X specifically involves:
[0013] Based on the location of the arbitrary inaccurate point label, each is extended by d / 2 along the direction of the coordinate axis to form a sample image block x with a width and height of d, generating a sample image block set X containing mixed multi-class inaccurate point labels;
[0014] Filter out sample image patches generated by expanding the category of inaccurate point annotations.
[0015]
[0016] in Indicates the category of inaccurate point markings;
[0017] Generate image-level single-label p with abstract features:
[0018]
[0019] Where n is the number of categories. The vector parameters representing a single label p;
[0020] Generate a single tag set .
[0021] As a preferred technical solution, the single-label classification network is pre-trained, specifically as follows:
[0022] A single-label classification network is built based on residual networks, using a single-label set. This serves as a supervisory signal for training a single-label classification network, causing the network to be biased towards learning single-label classes with real labels.
[0023] The input of any sample in a single-label set involves loss, according to the loss function L. EM Calculate the predicted output of the sample image patch x and the loss of the label p; the loss function is as follows:
[0024]
[0025]
[0026] Where f single (x) represents the predicted output for sample x, f i Let H(·) represent the output probability of the model for sample x in the i-th class, H(·) represent the loss function for the class with label 0, n represent the number of label classes, and p i This represents the label of the i-th class of the sample;
[0027] Compare the predicted value of any sample image patch with a set threshold. Comparative judgments are made to generate corresponding image-level pseudo-multi-class labels. Obtain the image-level pseudo-multi-class label set D cls .
[0028] As a preferred technical solution, the pre-training of the multi-label classification network is characterized by specifically including the following steps:
[0029] From the pseudo-multi-label training set D cls Extract sample image patch x as input and extract deep feature map. ;
[0030] A gradual dropout attention mechanism is employed to optimize the attention of deep feature maps; the average value of activation maps for all classes is obtained to generate an attention map A based on gradual dropout, which is then compared with the feature maps. Rich feature representations are obtained after positional multiplication. ;
[0031] The probability of the k-th organization category is calculated using global average pooling and a fully connected layer. :
[0032]
[0033] Where GAP(·) represents the global average pool. This represents the weight of the k-th category;
[0034] Calculate the value of the classification network loss function based on the classification labels:
[0035]
[0036] in This indicates a pseudo-multi-category label. Indicates the predicted classification result. This represents the loss function for multi-label classification.
[0037] Iterate through the above steps until the set training stop condition is met.
[0038] As a preferred technical solution, the gradual discarding attention mechanism comprises the following steps:
[0039] Based on feature map The weighted sum generates a class activation graph for each category:
[0040]
[0041] in This represents the class activation graph of the k-th class. This indicates the weights taken from the fully connected layer of the output layer;
[0042] A gradual discarding mechanism is set for each category activation graph to invalidate the most prominent regions of the class activation graph in order to update the class activation graph; the gradual discarding mechanism is as follows:
[0043]
[0044] Where i and j represent the corresponding class activation graphs. and coordinates This represents the updated class activation graph. Related to the maximum value of the class activation graph;
[0045]
[0046] in Indicates the gradual discard coefficient; The numerical calculation method is as follows:
[0047]
[0048] Where t represents the number of iterations. This represents the attenuation rate, and l represents the lowest boundary value.
[0049] Obtain the average value of the activation graphs for all classes and generate an attention graph A based on gradual dropout.
[0050] As a preferred technical solution, the method of using a multi-label classification network to obtain pseudo-masks for the three intermediate layers and generating pseudo-labels for segmentation network training specifically involves:
[0051] Using the feature maps of three intermediate layers in a multi-label classification network, a class activation map is generated through a gradient-weighted class activation map function. The pseudo-masks of the three intermediate layers are obtained by taking the maximum value of the class activation map, and used as pseudo-labels for training the segmentation network. .
[0052] As a preferred technical solution, generating the segmentation result image includes the following specific steps:
[0053] Given a sample image patch x, the predicted segmentation result P is obtained using the following formula:
[0054]
[0055] in The parameters representing the predicted segmentation;
[0056] Calculate the cross-entropy loss function of the predicted segmentation result and the pseudomasks of the three intermediate layers respectively. The weighted sum of the pseudomask loss is calculated as follows:
[0057]
[0058]
[0059] in, This represents the pixel point on sample x; The hyperparameters represent the loss functions of the three intermediate layers; the pseudo-mask is obtained by bilinear interpolation to achieve the resolution of the original image.
[0060] The process is repeated in reverse and the above steps are executed until the set stopping condition is reached. Finally, the segmentation result is obtained by weighted summation with the pseudo-mask.
[0061] Secondly, the present invention provides a weakly supervised pathological image tissue segmentation system based on inaccurate point annotation of fully digital pathological slides, which is applied to the weakly supervised pathological image tissue segmentation method based on inaccurate point annotation of fully digital pathological slides, including a fast point annotation module, a single-label classification network pre-training module, a multi-label classification network pre-training module, and a segmentation network training module.
[0062] The rapid point annotation module is used to acquire pathological slides and perform rapid inaccurate point annotation, accumulate inaccurate point annotations to form an inaccurate point annotation set, and expand any inaccurate point annotation in the inaccurate point annotation set outward to the surrounding area to form sample image block x and sample image set X. After feature filtering, image-level single labels p and single label sets with abstract features are generated. ;
[0063] The single-label classification network pre-training module is used to utilize single-label sets. Pre-train a pre-defined single-label classification network to obtain an image-level pseudo-multi-class label set D. cls ;
[0064] The multi-label classification network pre-training module is used to utilize the image-level pseudo-multi-class label set D. cls A pre-trained multi-label classification network is used to obtain pseudo-masks for the three intermediate layers and generate pseudo-labels for training the segmentation network. ;
[0065] The segmentation network training module is used to construct the segmentation network. Meanwhile, the segmentation network is trained under the supervision of the pseudo-labels. The segmentation result of the segmentation network and the loss of the pseudomask are calculated separately. The segmentation result map is obtained by weighted summation of the segmentation result and the loss of the pseudomask.
[0066] Thirdly, the present invention provides an electronic device, the electronic device comprising:
[0067] At least one processor; and,
[0068] A memory communicatively connected to the at least one processor; wherein,
[0069] The memory stores computer program instructions that can be executed by the at least one processor, which enables the at least one processor to perform the weakly supervised pathological image tissue segmentation method for inaccurate annotation of fully digital pathological slides.
[0070] Fourthly, the present invention provides a computer-readable storage medium storing a program, which, when executed by a processor, implements the described weakly supervised pathological image tissue segmentation method based on inaccurate point annotation of fully digital pathological slides.
[0071] Compared with the prior art, the present invention has the following advantages and beneficial effects:
[0072] (1) The present invention achieves semantic segmentation in the segmentation network, thereby reducing the cost of annotating tissue pathology images. At the same time, the present invention uses a single label set generated by fast point annotation with less information than pixel-level annotation as a supervision signal for weak supervision semantics. It does not require pre-cutting a large number of sample image blocks of pathology images, avoiding a large amount of screening input caused by covering tumor heterogeneity, thereby improving work efficiency.
[0073] (2) This invention utilizes low magnification and large field of view for rapid point annotation and reduces the reliance on image-level multi-label noise-free annotation, thereby reducing economic and manpower costs. Attached Figure Description
[0074] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0075] Figure 1 This diagram illustrates a weakly supervised pathological image tissue segmentation method based on inaccurate point annotation of fully digital pathological slides, as described in an embodiment of the present invention.
[0076] Figure 2 This is a schematic diagram of the tissue segmentation system for weakly supervised pathological images based on inaccurate point annotation of fully digital pathological slides, according to an embodiment of the present invention.
[0077] Figure 3 This is a structural diagram of an electronic device according to an embodiment of the present invention.
[0078] Explanation of icon numbers:
[0079] 10-Weakly supervised pathological image tissue segmentation system based on inaccurate point annotation of fully digital pathological slides; 11-Fast point annotation module; 12-Single-label classification network pre-training module; 13-Multi-label classification network pre-training module; 14-Segmentation network training module; 20-Electronic device; 21-First processor; 22-First memory; 23-Program of weakly supervised pathological image tissue segmentation method based on inaccurate point annotation of fully digital pathological slides. Detailed Implementation
[0080] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort are within the scope of protection of the present application.
[0081] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application can be combined with other embodiments.
[0082] All-digital pathology slides: a superpixel pathology image storage format. This data is pyramid-shaped and contains pathology slide images at different magnifications.
[0083] Fully digital pathological slide point annotation: Point annotation in the large field of view of fully digital pathological slides is much less difficult than pixel-level annotation methods, and can represent the main category of the surrounding image area.
[0084] Inaccurate Point Annotation Method for All-Digital Pathological Slides: Weakly Supervised Semantic Segmentation of Pathological Images utilizes weak annotations (including line annotations, bounding box annotations, image-level annotations, and point annotations) with limited information to accomplish pixel-level semantic segmentation tasks. Existing point annotation methods involve pre-cutting a large number of image blocks and then annotating the objects within these blocks using points. Due to the heterogeneity of tumor tissue, segmentation tasks require the annotation dataset to cover various tissue regions, prompting the model to learn sample features more comprehensively. Existing annotation methods collect a large number of image blocks from pathological images, which can be divided into many subsets with different tissue structures. Creating an annotation dataset requires selecting suitable samples from all image blocks and annotating them to cover all heterogeneous subsets. Therefore, this invention relates to an inaccurate point annotation method for pathological tissues under low-resolution, large-field-of-view conditions in all-digital pathological slides. Through this inaccurate point annotation method for all-digital pathological slides, this invention can simply, quickly, and efficiently obtain sufficient samples to cover the tumor heterogeneity of all-digital pathological slides.
[0085] A Weakly Supervised Semantic Segmentation Process Based on Inaccurate Point Labeling: Existing weakly supervised semantic segmentation of pathological tissues mainly involves the following steps: ① Training a multi-label classification network using multi-label samples; ② Generating a Class Activation Map (CAM) and obtaining pixel-level pseudo-labels; ③ Training a fully supervised semantic segmentation network based on the pixel-level pseudo-labels. However, this existing method relies on image-level multi-label noise-free annotations. Therefore, how to effectively utilize the aforementioned inaccurate point labels is what distinguishes this invention from existing technologies. This invention first expands from point labels outwards to surrounding areas to form small image patches, converting the point labels into image-level single labels. Then, a single-label classification network is trained, assuming that categories with predicted values exceeding a set threshold exist within the small image patches, forming image-level multi-labels. Finally, the process proceeds to the existing weakly supervised semantic segmentation steps.
[0086] Please see Figure 1 This embodiment proposes a weakly supervised pathological image tissue segmentation method based on inaccurate point annotation of fully digital pathological sections, which includes the following steps:
[0087] Step 1: Quickly annotate inaccurate points and generate image-level single tags, which includes the following steps:
[0088] (1-1) Obtain pathological slide data;
[0089] (1-2) Establish a coordinate system on the acquired pathological slide data;
[0090] (1-3) The original input WSI is used to quickly annotate multiple inaccurate points, and the inaccurate point annotations are accumulated to generate an inaccurate point annotation set;
[0091] (1-4) Generate sample image blocks by expanding any inaccurate point annotation from the inaccurate point annotation set to the surrounding area. Generate a sample image set X1; where inaccurate points of the main category in the sample image blocks are labeled as Points;
[0092] Optionally, the size of the sample image patch is
[0093] Optionally, the relative position of the sample image patch is characterized by the coordinates of the top-left and bottom-right corners being (p... x -d / 2,p y -d / 2) and (p x +d / 2, p y +d / 2);
[0094] Optional, the expanded sample image patch Generate a sample image set X1, specifically as follows:
[0095] (1-4-1) Based on the position of the arbitrary inaccurate point, extend each of the points along the coordinate axes by d / 2 to form a sample image block with width and height d. Generate a sample image block set X1 containing mixed multi-class inaccurate point annotations;
[0096] (1-4-2) Filter out the sample image blocks generated by expanding the inaccurate point labeling category.
[0097]
[0098] in Indicates the category of inaccurate point markings;
[0099] (1-4-3) Generate image-level single-label p with abstract features:
[0100]
[0101] Where n is the number of categories. The vector parameters representing a single label p;
[0102] (1-4-4) Generate a single tag set :
[0103]
[0104] Where P represents the set of image-level tags with a single tag of type p.
[0105] (1-5) Generate a set of sample image blocks, and filter out inaccurate points by labeling them as points and expanding the generated sample image blocks;
[0106] (1-6) Generate image-level single labels for abstract features and form a single label set.
[0107] Step 2: Pre-training of the single-label classification network, which includes the following steps:
[0108] (2-1) Constructing a single-label classification network based on residual networks ;
[0109] Furthermore, the residual network uses ResNet38 or ResNet50;
[0110] Furthermore, the residual network employs other residual networks that can enable the construction of single-label classification networks;
[0111] (2-2) Use a single label set as supervision to train a single label classification network, and use a loss function to make the network biased towards learning the single label class of the real label;
[0112] Preferably, the loss function is,
[0113]
[0114]
[0115] Where f single (x) represents the predicted output for sample x, f i Let H(·) represent the output probability of the model for sample x in the i-th class, H(·) represent the loss function for the class with label 0, n represent the number of label classes, and p i This represents the label of the i-th class of the sample;
[0116] (2-3) Set threshold The predicted output pred for any sample image patch is compared with a threshold. A comparison is made, and the predicted output `pred` for any sample image patch is greater than the set threshold. Generate corresponding image-level pseudo-multi-class labels;
[0117] ;
[0118] Optionally, set a threshold. =0.7, when the predicted output pred of the sample image patch is greater than the threshold. Generate corresponding pseudo-image-level multi-tags ; Obtain the image-level pseudo-multi-classification tag set D cls ;
[0119] ;
[0120]
[0121] in This represents the one-hot encoded vector of the image-level pseudo-multi-class label.
[0122] Step 3: Pre-training the multi-label classification network, which includes the following steps:
[0123] (3-1) Building a multi-classification network based on ResNet38 Taking the sample image patch x as input, the pseudo-image-level multi-label obtained in step 2... As a one-hot encoded vector As parameters for multi-label classification networks;
[0124] (3-2) Extracting deep feature maps :
[0125] ;
[0126] (3-3) Attention optimization is performed on the deep feature map based on the gradual discarding attention mechanism, the average value of the activation maps of all classes is obtained, and an attention map A based on gradual discarding is generated and compared with the feature map. Rich feature representations are obtained after positional multiplication. ;
[0127] Preferably, the gradual discarding mechanism described in step (3-3) includes the following steps:
[0128] (3-3-1) Based on the feature map The weighted sum generates a class activation graph for any category. ;
[0129] (3-3-2) Set the discard boundary The attention map is obtained by averaging the activation maps of different classes. ;
[0130] Furthermore, during the iterative training process, a gradual dropout coefficient is set. Set discard boundaries attenuation rate =0.985, The initial value is 1, and the lower bound is reached after T iterations. ;
[0131] (3-4) Attention Map With feature map After multiplying by digits, we get Then, global average pooling is used to obtain the probability graph of the classification results. ,
[0132]
[0133] Where GAP(·) represents the global average pool. This represents the weight of the k-th category;
[0134] (3-5) Calculate the value of the classification network loss function based on the classification labels. ; This represents the pseudo-multi-label obtained from step 2. Indicates the predicted classification result. This represents the Multi-Label Soft Margin Loss function;
[0135] (3-6) Repeat steps (3-1) to (3-5) until the set stopping condition is met, and the pre-training of the multi-label classification network terminates.
[0136] Preferably, after completing step 3, the present invention further includes the step of obtaining pseudo-masks for the three intermediate layers using a multi-label classification network and generating pseudo-labels for training the segmentation network, as follows:
[0137] Using the feature maps of three intermediate layers in a multi-label classification network, a class activation map is generated through a gradient-weighted class activation map function. The pseudo-masks of the three intermediate layers are obtained by taking the maximum value of the class activation map, and used as pseudo-labels for training the segmentation network. ;
[0138] Furthermore, the feature maps of the three intermediate layers in the multi-label classification network are... ;
[0139] Furthermore, the gradient-weighted class activation graph function generates a class activation graph, specifically by using a gradient-weighted class activation graph. Generate class activation graphs:
[0140]
[0141] in This represents the index of the category. Indicates the index of the intermediate layer of the classification network;
[0142] Furthermore, the Grad-CAM method is used to obtain the pseudo-masks of the three intermediate layers. .
[0143] Step 4. Training the segmentation network, which includes the following steps:
[0144] (4-1) Constructing a segmentation network Simultaneously, the three-layer pseudo-labels generated by the classification network in step 3 are utilized. Under supervision, the segmentation network is trained;
[0145] Preferred segmentation network The specific training method is as follows:
[0146] (4-1-1) Input image block , fake mask and segmentation parameters The data is fed into a segmentation network to obtain the predicted segmentation results. ;
[0147] (4-1-2) In order for the CNN model to learn feature maps at different levels, and also to reduce the information difference between pixel-level and image-level annotations, this invention supervises the training of the segmentation network in three layers of pseudomasks, and calculates the cross-entropy loss function between the segmentation result and the pseudomasks of the three intermediate layers. The formula is as follows, and a weighted sum is calculated for the losses of multiple pseudomasks;
[0148] Preferably, the weighted sum of the losses from multiple pseudomasks is calculated as follows:
[0149]
[0150] ;
[0151] in This represents the pixel point on sample x;
[0152] Preferably, the loss of the pseudomask is based on Calculate the weighted sum proportionally;
[0153] (4-1-3) Propagate in reverse and repeat steps (4-1-1) to (4-1-2) until the set stopping condition is reached and terminated. Finally, the segmentation result is obtained by weighted summation with the pseudo-mask.
[0154] In a more specific embodiment, the present invention provides a weakly supervised pathological image tissue segmentation method based on inaccurate point annotation of fully digital pathological slides, comprising the following methods:
[0155] S1. The original input WSI is annotated with multiple inaccurate points, and the resulting image patch of size dxd=224x224 is generated by expanding outwards to the surrounding area. The sample image blocks contain single labels for the main categories. The corresponding single tag is Image blocks After passing through a single-label classification network, the predicted output is greater than the threshold. =0.7 corresponds to pseudo-image-level multi-annotation .
[0156] S2, Image-level Multiple Annotations It is a one-hot encoded vector that indicates whether each type of organization appears in the image patch. These are parameters related to the organization's classification category. (Image block) It is fed into a CNN network to extract deep feature maps. .
[0157] S3. Perform weighted summation on the feature maps to obtain the class activation map. Set discard boundaries The attention map is obtained by averaging the activation maps of different classes. The feature map is obtained by multiplying the feature maps bitwise. Then, global average pooling is used to obtain the probability graph of the classification results. .
[0158] S4. During the iterative training process, set a gradual dropout coefficient. , The initial value is 1, until after The next loop reaches the lower bound. .
[0159] S5. Input the original image patch into the trained classification network to obtain the feature maps of the intermediate layers. Three independent pseudomasks are obtained using the Grad-CAM method. .
[0160] S6, Input image block , fake mask and segmentation parameters The segmentation network is used to obtain the segmentation results. .
[0161] S7. Calculate the segmentation result of the segmentation network and the loss of the pseudomask separately, and calculate the weighted sum in a ratio of 8:1:1 to obtain the segmentation result image.
[0162] It should be noted that, for the sake of simplicity, the aforementioned method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously.
[0163] Based on the same idea as the weakly supervised pathological image tissue segmentation method based on inaccurate point annotation of fully digital pathological slides in the above embodiments, the present invention also provides a weakly supervised pathological image tissue segmentation system based on inaccurate point annotation of fully digital pathological slides. This system can be used to execute the aforementioned weakly supervised pathological image tissue segmentation method based on inaccurate point annotation of fully digital pathological slides. For ease of explanation, the structural schematic diagram of the embodiment of the weakly supervised pathological image tissue segmentation system based on inaccurate point annotation of fully digital pathological slides only shows the parts related to the embodiments of the present invention. Those skilled in the art will understand that the illustrated structure does not constitute a limitation on the device, and may include more or fewer components than illustrated, or combine certain components, or have different component arrangements.
[0164] Please see Figure 2 In another embodiment of this application, a weakly supervised pathological image tissue segmentation method system 10 based on inaccurate point annotation of fully digital pathological slides is provided. The system includes a fast point annotation module 11, a single-label classification network pre-training module 12, a multi-label classification network pre-training module 13, and a segmentation network training module 14.
[0165] The rapid point annotation module 11 is used to acquire pathological slides and perform rapid inaccurate point annotation, complete the accumulation of inaccurate point annotations, form an inaccurate point annotation set, expand any inaccurate point annotation in the inaccurate point annotation set to the surrounding area, forming sample image block x and sample image set X. After feature filtering, it generates image-level single labels p and single label sets with abstract features. ;
[0166] The single-label classification network pre-training module 12 is used to utilize single-label sets Pre-train a pre-defined single-label classification network to obtain an image-level pseudo-multi-class label set D. cls ;
[0167] The multi-label classification network pre-training module 13 is used to utilize the image-level pseudo-multi-class label set D. cls A pre-trained multi-label classification network is used to obtain pseudo-masks for the three intermediate layers and generate pseudo-labels for training the segmentation network. ;
[0168] The segmentation network training module 14 is used to construct a segmentation network. Meanwhile, the segmentation network is trained under the supervision of the pseudo-labels. The segmentation result of the segmentation network and the loss of the pseudomask are calculated separately. The segmentation result map is obtained by weighted summation of the segmentation result and the loss of the pseudomask.
[0169] It should be noted that the tissue segmentation system for weakly supervised pathological images based on inaccurate point annotation of fully digital pathological slides of the present invention corresponds one-to-one with the tissue segmentation method for weakly supervised pathological images based on inaccurate point annotation of fully digital pathological slides of the present invention. The technical features and beneficial effects described in the embodiments of the tissue segmentation method for weakly supervised pathological images based on inaccurate point annotation of fully digital pathological slides are all applicable to the embodiments of the tissue segmentation method for weakly supervised pathological images based on inaccurate point annotation of fully digital pathological slides. For details, please refer to the description in the embodiments of the method of the present invention, which will not be repeated here.
[0170] Furthermore, in the embodiments of the weakly supervised pathological image tissue segmentation system based on inaccurate annotation of fully digital pathological slides described above, the logical division of each program module is merely illustrative. In practical applications, the above functions can be assigned to different program modules as needed, for example, for the sake of corresponding hardware configuration requirements or the convenience of software implementation. That is, the internal structure of the weakly supervised pathological image tissue segmentation system based on inaccurate annotation of fully digital pathological slides can be divided into different program modules to complete all or part of the functions described above.
[0171] Please see Figure 3In one embodiment, an electronic device is provided for implementing a weakly supervised pathological image tissue segmentation method based on inaccurate point annotation of fully digital pathological slides. The electronic device 20 may include a first processor 21, a first memory 22, and a bus, and may also include a computer program stored in the first memory 22 and executable on the first processor 21, such as program 23 for a weakly supervised pathological image tissue segmentation method based on inaccurate point annotation of fully digital pathological slides.
[0172] The first memory 22 includes at least one type of readable storage medium, including flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the first memory 22 can be an internal storage unit of the electronic device 20, such as the portable hard drive of the electronic device 200. In other embodiments, the first memory 22 can be an external storage device of the electronic device 20, such as a plug-in portable hard drive, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the electronic device 20. Furthermore, the first memory 22 can include both internal and external storage units of the electronic device 20. The first memory 22 can be used not only to store application software and various types of data installed on the electronic device 20, such as the code of the weakly supervised pathological image tissue segmentation method program 23 based on inaccurate point annotation of fully digital pathological slides, but also to temporarily store data that has been output or will be output.
[0173] In some embodiments, the first processor 21 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The first processor 21 is the control unit of the electronic device, connecting various components of the entire electronic device through various interfaces and lines. It executes programs or modules stored in the first memory 22 and calls data stored in the first memory 22 to perform various functions of the electronic device 20 and process data.
[0174] Figure 3 Only electronic devices with components are shown; those skilled in the art will understand that... Figure 3The structure shown does not constitute a limitation on the electronic device 20, and may include fewer or more components than shown, or combine certain components, or have different component arrangements.
[0175] The program 23 of the weakly supervised pathological image tissue segmentation method based on inaccurate point annotation of fully digital pathological slides, stored in the first memory 22 of the electronic device 20, is a combination of multiple instructions. When run in the first processor 21, it can achieve the following:
[0176] Pathological slides are acquired and rapid inaccurate point annotations are performed to accumulate inaccurate point annotations and form an inaccurate point annotation set. Any inaccurate point annotation in the inaccurate point annotation set is then expanded outwards to surrounding areas, forming sample image patch x and sample image set X. After feature filtering, image-level single-label p and single-label set with abstract features are generated. ;
[0177] Using a single tag set Pre-train a pre-defined single-label classification network to obtain an image-level pseudo-multi-class label set D. cls ;
[0178] Using image-level pseudo-multi-class label set D cls A pre-trained multi-label classification network is used to obtain pseudo-masks for the three intermediate layers and generate pseudo-labels for training the segmentation network. ;
[0179] Constructing a segmentation network Meanwhile, the segmentation network is trained under the supervision of the pseudo-labels. The segmentation result of the segmentation network and the loss of the pseudomask are calculated separately. The segmentation result map is obtained by weighted summation of the segmentation result and the loss of the pseudomask.
[0180] Furthermore, if the modules / units integrated in the electronic device 20 are implemented as software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium. The computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).
[0181] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.
[0182] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0183] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.
Claims
1. A weakly supervised tissue segmentation method for pathological images based on inaccurate point annotation of fully digital pathological sections, characterized in that, Includes the following steps: Pathological slides are acquired and rapid inaccurate point annotations are performed to accumulate inaccurate point annotations and form an inaccurate point annotation set. Any inaccurate point annotation in the inaccurate point annotation set is then expanded outwards to surrounding areas, forming sample image patch x and sample image set X. After feature filtering, image-level single-label p and single-label set with abstract features are generated. ; Using a single tag set Pre-train a pre-defined single-label classification network to obtain an image-level pseudo-multi-class label set D. cls ; Using image-level pseudo-multi-class label set D cls A pre-trained multi-label classification network is used to obtain pseudo-masks for the three intermediate layers and generate pseudo-labels for training the segmentation network. ; Constructing a segmentation network Meanwhile, the segmentation network is trained under the supervision of the pseudo-labels. The segmentation result of the segmentation network and the loss of the pseudomask are calculated separately, and the segmentation result image is obtained by weighted summation of the segmentation result and the loss of the pseudomask. The step of expanding any inaccurate point annotation from the inaccurate point annotation set to the surrounding area to form a sample image block x and a sample image set X specifically involves: Based on the location of the arbitrary inaccurate point label, each is extended by d / 2 along the direction of the coordinate axis to form a sample image block x with a width and height of d, generating a sample image block set X containing mixed multi-class inaccurate point labels; Filter out sample image patches generated by expanding the category of inaccurate point annotations. in Indicates the category of inaccurate point markings; Generate image-level single-label p with abstract features: Where n is the number of categories. The vector parameters representing a single label p; Generate a single tag set ; The single-label classification network is pre-trained, specifically as follows: A single-label classification network is built based on residual networks, using a single-label set. This serves as a supervisory signal for training a single-label classification network, causing the network to be biased towards learning single-label classes with real labels. The input of any sample in a single-label set involves loss, according to the loss function L. EM Calculate the predicted output of the sample image patch x and the loss of the label p; the loss function is as follows: Where f single (x) represents the predicted output for sample x, f i Let H(·) represent the output probability of the model for sample x in the i-th class, H(·) represent the loss function for the class with label 0, n represent the number of label classes, and p i This represents the label of the i-th class of the sample; Compare the predicted value of any sample image patch with a set threshold. Comparative judgments are made to generate corresponding image-level pseudo-multi-class labels. Obtain the image-level pseudo-multi-class label set D cls .
2. The method for tissue segmentation of pathological images based on inaccurate point annotation of fully digital pathological sections according to claim 1, characterized in that, The pre-training of the multi-label classification network specifically includes the following steps: From the pseudo-multi-label training set D cls Extract sample image patch x as input and extract deep feature map. ; A gradual discarding attention mechanism is used to optimize the attention of deep feature maps; Obtain the average of all class activation maps, generate an attention map A based on gradual dropout, and compare it with the feature maps. Rich feature representations are obtained after positional multiplication. ; The probability of the k-th organization category is calculated using global average pooling and a fully connected layer. : Where GAP(·) represents the global average pool. This represents the weight of the k-th category; Calculate the value of the classification network loss function based on the classification labels: in This indicates a pseudo-multi-category label. Indicates the predicted classification result. This represents the loss function for multi-label classification. Iterate through the above steps until the set training stop condition is met.
3. The method for tissue segmentation of pathological images based on inaccurate point annotation of fully digital pathological sections according to claim 2, characterized in that, The gradual discarding attention mechanism comprises the following steps: Based on feature map The weighted sum generates a class activation graph for each category: in This represents the class activation graph of the k-th class. This indicates the weights taken from the fully connected layer of the output layer; A gradual discarding mechanism is set for each class activation graph to invalidate the most prominent region of the class activation graph in order to update the class activation graph; the gradual discarding mechanism is as follows: Where i and j represent the corresponding class activation graphs. and coordinates This represents the updated class activation graph. Related to the maximum value of the class activation graph; in Indicates the gradual discard coefficient; The numerical calculation method is as follows: Where t represents the number of iterations. This represents the attenuation rate, and l represents the lowest boundary value. Obtain the average value of the activation graphs for all classes and generate an attention graph A based on gradual dropout.
4. The method for tissue segmentation of pathological images based on inaccurate point annotation of fully digital pathological sections according to claim 1, characterized in that, The process of using a multi-label classification network to obtain pseudo-masks for the three intermediate layers and generating pseudo-labels for training the segmentation network is as follows: Using the feature maps of three intermediate layers in a multi-label classification network, a class activation map is generated through a gradient-weighted class activation map function. The pseudo-masks of the three intermediate layers are obtained by taking the maximum value of the class activation map, and used as pseudo-labels for training the segmentation network. .
5. The method for tissue segmentation of pathological images based on inaccurate point annotation of fully digital pathological sections according to claim 4, characterized in that, The generation of the segmentation result image includes the following specific steps: Given a sample image patch x, the predicted segmentation result P is obtained using the following formula: in This represents the parameters of the prediction segmentation network; Calculate the cross-entropy loss function of the predicted segmentation result and the pseudomasks of the three intermediate layers respectively. The weighted sum of the pseudomask loss is calculated as follows: in, This represents the pixel point on sample x; The hyperparameters represent the loss functions of the three intermediate layers; the pseudo-mask is obtained by bilinear interpolation to achieve the resolution of the original image. The process is repeated in reverse and the above steps are executed until the set stopping condition is reached. Finally, the segmentation result is obtained by weighted summation with the pseudo-mask.
6. A weakly supervised pathological image tissue segmentation system based on inaccurate point annotation of fully digital pathological slides, characterized in that... The tissue segmentation method for weakly supervised pathological images based on inaccurate point annotation of fully digital pathological slides, as described in any one of claims 1-5, includes a fast point annotation module, a single-label classification network pre-training module, a multi-label classification network pre-training module, and a segmentation network training module. The rapid point annotation module is used to acquire pathological slides and perform rapid inaccurate point annotation, accumulate inaccurate point annotations to form an inaccurate point annotation set, and expand any inaccurate point annotation in the inaccurate point annotation set outward to the surrounding area to form sample image block x and sample image set X. After feature filtering, image-level single labels p and single label sets with abstract features are generated. ; The single-label classification network pre-training module is used to utilize single-label sets. Pre-train a pre-defined single-label classification network to obtain an image-level pseudo-multi-class label set D. cls ; The multi-label classification network pre-training module is used to utilize the image-level pseudo-multi-class label set D. cls A pre-trained multi-label classification network is used to obtain pseudo-masks for the three intermediate layers and generate pseudo-labels for training the segmentation network. ; The segmentation network training module is used to construct the segmentation network. Meanwhile, the segmentation network is trained under the supervision of the pseudo-labels. The segmentation result of the segmentation network and the loss of the pseudomask are calculated separately. The segmentation result map is obtained by weighted summation of the segmentation result and the loss of the pseudomask.
7. An electronic device, characterized in that, The electronic device includes: At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores computer program instructions executable by the at least one processor, the computer program instructions being executed by the at least one processor to enable the at least one processor to perform the weakly supervised pathological image tissue segmentation method based on inaccurate point annotation of fully digital pathological slides as described in any one of claims 1-5.
8. A computer-readable storage medium storing a program, characterized in that, When the program is executed by the processor, it implements the weakly supervised pathological image tissue segmentation method based on inaccurate point annotation of fully digital pathological slides as described in any one of claims 1-5.
