Colon histology image gland segmentation method based on instance-aware diffusion model
By employing an instance-aware diffusion model, combining an image encoder, an FPN network, and Gaussian noise, and introducing an instance-aware filter and a Mask FCN Head network, the problems of accuracy in gland segmentation and background classification errors in colon histology images are solved, achieving more accurate gland segmentation.
Patent Information
- Application Number
- CN202311282053.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-09-28
- Publication Date
- 2026-01-09
- Estimated Expiration
- 2043-09-28
AI Technical Summary
Existing technologies for gland segmentation in colon histological images suffer from insufficient accuracy in target location identification and misclassification of background tissue, resulting in low segmentation accuracy.
An instance-aware diffusion model-based approach is adopted, which acquires features through an image encoder and an FPN network, combines Gaussian noise and conditional coding, and uses a diffusion model for gland instance segmentation. An instance-aware filter and a Mask FCN Head network are introduced to achieve accurate segmentation.
It improves the accuracy of gland segmentation and the refinement of segmentation edges, enhances the distinction between the target and the background, and improves the precision and reliability of gland segmentation.
Smart Images
Figure CN117237371B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of colon histology image gland segmentation, and particularly relates to a colon histology image gland segmentation method based on an instance perception diffusion model. BACKGROUND
[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute the prior art.
[0003] Colorectal cancer is a cancer that is prevalent worldwide, and it mainly manifests as colorectal adenocarcinoma, which develops in the inner layer of the colon or rectum and is identified through the formation of glandular structures that play a crucial role in the secretion of proteins and carbohydrates in various organ systems. In order to evaluate the differentiation of colorectal adenocarcinoma, pathologists typically perform gland histology examination using hematoxylin and eosin staining. The degree of gland formation is an important factor used by pathologists to determine the tumor grade or degree of differentiation, and accurate segmentation of glands on histology images is an important basis for evaluating gland morphology and determining the malignancy degree of colorectal adenocarcinoma.
[0004] Accurate segmentation of glands is not only important on histopathology slides of colorectal glands, but also on histology slides of other organs such as the prostate. Manually annotating gland instances is a time-consuming process that requires a high level of expertise and dedication. Therefore, automated gland segmentation methods are of great significance in clinical practice.
[0005] Automatic image segmentation has become the subject of several deep learning-based techniques, including segmentation networks such as U-Net, FCN, Siamese Network and its variants; another approach to this work is a two-stage instance segmentation method, which includes methods such as Mask RCNN and BlendMask, these techniques mainly integrate two sub-networks for object detection and segmentation in a sequential manner. However, these methods may encounter challenges in capturing the location of cells with different shapes and distinguishing very close gland boundaries.
[0006] In recent years, diffusion models have received extensive attention as a highly generalizable category of generative models, and they have proven to have good generative ability compared to GAN networks. In addition, diffusion models have also been used for various other tasks, and in the task of object detection, object detection is considered as a generation task in the bounding box space of an image, thereby addressing the problem of occlusion detection. In addition, some research has studied the feasibility of using diffusion models in medical image segmentation, but existing diffusion models for medical image segmentation are one-step segmentation, with the diffusion model directly outputting the segmentation result, and in this process, there is no specific location of the glands, and the global mask is always being reconstructed, which is prone to errors.
[0007] In addition, the colon histology image mainly has the characteristics of diverse and irregular glandular morphology, blurred boundary, uneven staining and the like, resulting in that the image segmentation is difficult to accurately cut the edge position. SUMMARY
[0008] In order to solve the problems in the prior art, the application provides a colon histology image gland segmentation method and system based on an instance perception diffusion model, an electronic device and a computer readable storage medium, which can accurately identify the target position and prevent the false classification of background tissues.
[0009] In the first aspect, the application provides a colon histology image gland segmentation method based on an instance perception diffusion model.
[0010] The colon histology image gland segmentation method based on the instance perception diffusion model comprises the following steps.
[0011] Obtaining a colon histology image to be segmented;
[0012] Inputting the colon histology image to be segmented into a trained instance perception diffusion model for processing to obtain a gland segmentation result; specifically comprising the following steps.
[0013] Inputting the colon histology image to be segmented into a trained image encoder and FPN network respectively to obtain original image features and multi-scale features;
[0014] Fusing the original image features and Gaussian noise to generate noise image features, and inputting the noise image features and the original image features into a segmentation network to obtain predicted gland instance features and filtering;
[0015] Obtaining a gland instance segmentation result according to the multi-scale features and the filtered gland instance features.
[0016] Further, the step of fusing the original image features and Gaussian noise to generate noise image features specifically comprises the following steps.
[0017] Randomly generating Gaussian noise, and generating a plurality of noise boxes based on the original image features.
[0018] Further, the step of obtaining a gland instance segmentation result according to the filtered gland instance features and the multi-scale features specifically comprises the following steps.
[0019] Inputting the multi-scale features and the filtered gland instance features into an instance segmentation model to obtain a bounding box and a mask of each gland instance.
[0020] Preferably, the instance segmentation model is a mask head network.
[0021] Further, the noise image and the original image features are input into the segmentation network to obtain predicted gland instance features, including:
[0022] The noise image features and the original image features are correlated to obtain fused extraction features.
[0023] The fused extraction features are input into the image decoder for processing to obtain the predicted gland instance features.
[0024] The image decoder is constructed based on a diffusion model.
[0025] Preferably, the input of the fused extraction features into the image decoder for processing includes:
[0026] The fused extraction features are converted into gland instance features by reverse diffusion of the fused extraction features through a Markov chain.
[0027] Further, the predicted gland instance features are input into an instance-aware filter for filtering.
[0028] In a second aspect, the present application provides a colon histology image gland segmentation system based on an instance-aware diffusion model.
[0029] A colon histology image gland segmentation system based on an instance-aware diffusion model includes:
[0030] An acquisition module configured to acquire a colon histology image to be segmented.
[0031] A colon histology image gland segmentation module configured to input the colon histology image to be segmented into a trained instance-aware diffusion model for processing to obtain a gland segmentation result, specifically including:
[0032] The colon histology image to be segmented is input into a trained image encoder and FPN network respectively to obtain original image features and multi-scale features.
[0033] The original image features and Gaussian noise are fused to generate noise image features, and the noise image features and the original image features are input into a segmentation network to obtain predicted gland instance features and filtering.
[0034] The gland instance segmentation result is obtained according to the filtered gland instance features and the multi-scale features.
[0035] In a third aspect, the present application provides an electronic device.
[0036] An electronic device includes a memory and a processor, and computer instructions stored on the memory and running on the processor, when the computer instructions are run by the processor, the steps of the above colon histology image gland segmentation method based on an instance-aware diffusion model are completed.
[0037] In a fourth aspect, the present application provides a computer readable storage medium;
[0038] A computer readable storage medium for storing computer instructions, which when executed by a processor, complete the steps of the colon histology image gland segmentation method based on the instance perception diffusion model.
[0039] Compared with the prior art, the present application has the following beneficial effects:
[0040] 1. The technical scheme provided by the present application aims at the problem of insufficient accuracy of target position recognition and incorrect classification of background tissues in the network trained in the prior art, and proposes a colon histology image gland segmentation method based on an instance perception diffusion model, which introduces a diffusion model into gland segmentation and proposes a segmentation network containing an instance perception filter, a Mask FCN Head and a conditional coding, to accurately identify the target position and prevent incorrect classification of background tissues.
[0041] 2. The technical scheme provided by the present application uses a diffusion model to model the process of gland instance segmentation in a colon histology image as a denoising process, which is a new method for colon histology image gland segmentation, and regards the segmentation process as a diffusion process of generating noise to a filter; in order to enhance the segmentation process, an instance perception technology is used to restore the details that may be lost in the denoising process, specifically, a filter and a multi-scale mask branch are used to construct a global mask and refine finer details of segmentation. In order to enhance the difference between the target and the background, conditional coding is used to enhance the intermediate features through original image coding. This method effectively integrates the rich information of the original image, thereby improving the difference between the object of interest and the surrounding background. The method is trained and tested on the 2015 MICCAI gland segmentation (GlaS) challenge dataset and the colorectal adenocarcinoma gland (CRAG) dataset, and the experimental results prove the effectiveness of the method. BRIEF DESCRIPTION OF DRAWINGS
[0042] The drawings accompanying the specification of the present application form a part thereof and serve to provide further understanding of the present application, the illustrative embodiments of the present application and their descriptions serve to explain the present application, and do not constitute an improper limitation on the present application.
[0043] Figure 1 The flowchart provided for the embodiments of the present application;
[0044] Figure 2 The diffusion model running schematic diagram provided for the embodiments of the present application;
[0045] Figure 3 The colon histology image dataset example diagram provided for the embodiments of the present application;
[0046] Figure 4 An example diagram of a final result provided by an embodiment of the present application is shown in FIG. 1.
[0047] Figure 5 Another example diagram of a final result provided by an embodiment of the present application is shown in FIG. 2. DETAILED DESCRIPTION
[0048] It should be noted that the following detailed description is exemplary in nature and is intended to provide further description of the application. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.
[0049] It is also important to note that the use of the term "example" herein is merely meant to illustrate an example embodiment of the application, and is not meant to limit the application. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof.
[0050] The embodiments in the present application and the features in the embodiments can be combined with each other without conflict.
[0051] Embodiment One
[0052] The medical image segmentation technology in the prior art is not accurate enough in identifying the target position when segmenting the gland of a colon histology image, and is prone to misclassification of the background tissue, resulting in low accuracy of gland segmentation. Therefore, the present application provides a colon histology image gland segmentation method based on an instance perception diffusion model.
[0053] Next, the colon histology image gland segmentation method based on the instance perception diffusion model will be described in detail. Figures 1-5 The colon histology image gland segmentation method based on the instance perception diffusion model comprises the following steps:
[0054] S1, obtaining a colon histology image to be segmented.
[0055] S2, inputting the colon histology image to be segmented into a preset instance perception diffusion model for processing to obtain a gland segmentation result. The specific steps include:
[0056] S201, input the colon histology image to be segmented into the trained image encoder and FPN network respectively, and obtain the original image feature and the multi-scale feature. The image encoder is a ResNet network, which includes five layers of convolutional networks connected in sequence
[0057] Further, the colon histology image to be segmented is input into the trained ResNet network, and the output of the last layer of the network is obtained as the original image feature F R . The FPN network fuses the image features output by the second, third, fourth and fifth layers of convolutional networks in the ResNet network to obtain the multi-scale feature.
[0058] S202, fuse the original image feature and the Gaussian noise to generate a noise image feature, input the noise image and the original image feature into the segmentation network, obtain the predicted gland instance feature and filter.
[0059] Specifically, the specific process of generating a noise image feature is to randomly generate Gaussian noise, generate a plurality of noise boxes based on the original image feature, and take the noise boxes as the noise image feature b t .
[0060] Illustrative, the process of inputting the noise image and the original image feature into the segmentation network to obtain the predicted gland instance feature and filtering is further described:
[0061] (1) Perform correlation operation on the noise image feature b t and the original image feature F R to obtain the fusion extraction feature to make up for the lost information in the diffusion process, and the fusion extraction feature is represented as:
[0062] F=Concat(b t ,F R )
[0063] Wherein, F represents the fusion extraction feature, b t is the noise image feature, and F R is the original image feature.
[0064] (2) Input the fusion extraction feature into the image decoder based on the diffusion model to obtain the predicted gland instance feature, and the gland instance feature is represented as:
[0065] f t =D(Concat(E(F),F R ),t)
[0066] Wherein, t represents the current time, E represents the encoder, D represents the decoder, and f t represents the output feature of the diffusion model, i.e. the predicted gland instance feature.
[0067] (3) The instance-aware filter is inputted with the above features, and after information filtering, the final predicted gland instance features are obtained, denoted as:
[0068]
[0069] wherein IAF represents the instance-aware filter, denotes the final predicted instance features at the current time.
[0070] Further, as shown in Figure 1 , wherein Image represents the original image, Ground Truth represents the true value label, Image Encoder represents the image encoder, Image Decoder represents the image decoder, Diffusion Model represents the diffusion model, Conditional Encoding represents the conditional encoding, Mask Branch represents the mask branch, InstanceAware Filters represents the instance-aware filter, and Gaussian Noise represents the Gaussian noise.
[0071] The image decoder is constructed based on the diffusion model. The diffusion model usually uses two Markov chains, which are divided into two stages: forward diffusion stage and backward diffusion stage. The diffusion model is mainly trained on the training data with T time steps. The components of the diffusion model are a learning reverse process called p θ (z t-1 |z t ) that creates samples by converting noise from q(z0) to samples, and a forward diffusion process called q(z t |z t-1 ) that gradually destroys data from a certain target distribution into a normal distribution. In the forward diffusion stage, Gaussian noise is gradually added to the input image until the image is completely destroyed and becomes a completely noisy image with Gaussian distribution. Since the forward process is a non-homogeneous Markov chain, its dynamics can be simulated by a one-step transition density:
[0072]
[0073] The image decoder in the embodiment is based on the diffusion model, and using the reverse diffusion process, it can be regarded as a noise-to-Ground Truth denoising sampling process. A set of random boxes sampled from the Gaussian distribution is used as input to crop the ROI features from the feature map generated by the image encoder. In this process, the data samples are composed of a set of bounding boxes.
[0074] For example, the features obtained during the diffusion process are input using conditional encoding to supplement some information lost during the diffusion process; the fused features are input into the trained diffusion model to obtain the predicted gland instance features.
[0075] The fusion-extracted features are noisy image features, and the diffusion model neural network extracts feature b from the fusion. t The gland instance feature b0 is predicted, where b0 is the Ground Truth. The denoising process is as follows:
[0076]
[0077] S203. Obtain the gland instance segmentation result based on the multi-scale features and the filtered gland instance features.
[0078] Specifically, the final predicted gland instance features and multi-scale features F mask Inputting into the Mask FCN Header yields the bounding box and mask for each instance, represented as follows:
[0079]
[0080] Where s is the final output gland segmentation result, that is, the final output gland instance features, including gland instance bounding boxes and masks.
[0081] Furthermore, in some embodiments, before applying this method, the instance-aware model is trained using an existing dataset through the above logic. The training process is as follows: the goal of training the proposed network is to accurately determine the instance location based on the generative reasoning process of the diffusion model, and the training process follows the training process of the standard diffusion model.
[0082] During training, the process for acquiring features from noisy images is as follows:
[0083] (1) Based on the features of the original image, obtain the baseline true value corresponding to the colon histology image; based on the baseline true value, obtain the bounding box and mask of the gland instance.
[0084] (2) By using Gaussian noise, multiple noise boxes are generated and fused with the bounding boxes of gland instances in the baseline true value to obtain noise boxes that approximate the noise, so as to simulate the noise in the real world.
[0085] Unlike the established diffusion model, the network model proposed in this embodiment requires the further development of an instance-aware filter and a Mask FCN Head to effectively obtain instance information.
[0086] Next, to further validate the effectiveness of the method described in this embodiment, the data sets of Glas and CRAG were used for testing and compared with other state-of-the-art methods on this data set.
[0087] The segmentation results were evaluated using the three metrics in GlaS Challenge:
[0088] (1) Object F1, which measures the accuracy of detecting individual glands.
[0089] (2) Object Dice, which evaluates the volume-based accuracy of gland segmentation.
[0090] (3) Object Hausdorff, which evaluates the shape similarity between the segmentation results and the ground truth.
[0091] According to these metrics, each method was assigned three ranking numbers, and their sum was calculated to determine the final ranking of the overall performance of each method.
[0092] The GlaS dataset has a total of 165 images, of which 85 images are used for training and 80 images are used for testing. The test set is further divided into Test A (60 images) and Test B (20 images). Each image in the GlaS dataset has a size of 775x522 pixels and is accompanied by an instance segmentation Ground Truth that accurately highlights the gland boundaries and a precise lumen annotation. The dataset comes from 16 H&E stained whole slide images (WSIS) scanned using a Mirax MIDI slide scanner at a pixel resolution of 20x. The CRAG dataset has a total of 213 images, of which 173 are used for training and 40 are used for testing. Each image in the CRAG dataset has a dimension of 1512x1512 pixels and includes instance-level ground truth annotations. The dataset is obtained by scanning 38 H&E stained WSIS using a VL120 scanner at a pixel resolution of 20x.
[0093] All experiments were performed on a PC equipped with an i5 quad-core 2.59GHz CPU, 8GB RAM, and a GTX 1070 GPU.
[0094] The experimental results on the GlaS dataset are shown in Table 1, and the experimental results on the CRAG dataset are shown in Table 2, and are compared with the results of other state-of-the-art methods.
[0095] Table 1 Results on the GlaS dataset
[0096]
[0097]
[0098] Table 2 Results on CRAG dataset
[0099]
[0100] Specifically, the embodiment introduces a diffusion model into gland segmentation, and proposes a segmentation network containing an instance-aware filter, a Mask FCN Head and conditional encoding. First, by introducing a diffusion model and an instance-aware filter, a new method for gland segmentation of colonic histological images is realized. In the ResNet network and the FPN network, the original features and multi-scale features are obtained, and the original features and the noise frame generated by the Gaussian noise are fused; then, the features enter the diffusion model for gland instance prediction, realizing accurate gland segmentation of colonic histological images.
[0101] Embodiment two
[0102] The embodiment discloses a gland segmentation system for colonic histological images based on an instance-aware diffusion model, comprising:
[0103] The acquisition module is configured to acquire a colonic histological image to be segmented.
[0104] The colonic histological image gland segmentation module is configured to input the colonic histological image to be segmented into a preset instance-aware diffusion model for processing to obtain a gland segmentation result; specifically comprising:
[0105] The colonic histological image to be segmented is input into the trained image encoder and FPN network respectively to obtain original image features and multi-scale features.
[0106] The original image features and the Gaussian noise are fused to generate noise image features, and the noise image features and the original image features are input into the segmentation network to obtain predicted gland instance features and filtering.
[0107] According to the filtered gland instance features and the multi-scale features, a gland instance segmentation result is obtained.
[0108] It should be noted that the above acquisition module and colonic histological image gland segmentation module correspond to the steps in embodiment one, and the above modules have the same examples and application scenarios as the corresponding steps, but are not limited to the content disclosed in the above embodiment one. It should be noted that the above modules as part of the system can be executed in a computer system such as a set of computer executable instructions.
[0109] Embodiment three
[0110] The embodiment three of the present application provides an electronic device, comprising a memory and a processor, and computer instructions stored in the memory and running on the processor, when the computer instructions are run by the processor, the steps of the colon histology image gland segmentation method based on the instance perception diffusion model are completed.
[0111] Embodiment four
[0112] The embodiment four of the present application provides a computer readable storage medium, used for storing computer instructions, when the computer instructions are executed by a processor, the steps of the colon histology image gland segmentation method based on the instance perception diffusion model are completed.
[0113] The present application is described with reference to flowcharts and / or block diagrams of the method, device (system) and computer program product according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the computer or other programmable data processing device produce a device implemented in the flowcharts and / or block diagrams. Figure 1 The function specified in one flow or multiple flows and / or blocks Figure 1 The device for the function specified in one block or multiple blocks.
[0114] These computer program instructions can also be stored in a computer readable storage medium, which can guide the computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer readable storage medium produce a product including instruction devices, which implement the flowcharts and / or block diagrams. Figure 1 The function specified in one flow or multiple flows and / or blocks Figure 1 The device for the function specified in one block or multiple blocks.
[0115] These computer program instructions can also be loaded into a computer or other programmable data processing device, and a series of operation steps are performed on the computer or other programmable device to produce a computer implemented process, so that the instructions executed on the computer or other programmable device provide a process for implementing the flowcharts and / or block diagrams. Figure 1 The function specified in one flow or multiple flows and / or blocks Figure 1 The device for the function specified in one block or multiple blocks.
[0116] The description of each embodiment in the above embodiments is focused on each embodiment, and the part not described in detail in a certain embodiment can refer to the related description of other embodiments.
[0117] The above merely provides the preferred embodiments of the present application, and is not used to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modifications, equivalent replacements, improvements, etc. made within the principles and technical scope of the present application shall fall into the scope of the present application.
Claims
1. A method for gland segmentation in colonic histological images based on an instance-aware diffusion model, characterized in that, include: Obtain histological images of the colon to be segmented; The colon histological image to be segmented is input into a trained instance-aware diffusion model for processing to obtain gland segmentation results. Specifically, it includes: The colon histology images to be segmented are input into the trained ResNet network and FPN network respectively to obtain the original image features and multi-scale features; The original image features and Gaussian noise are fused to generate noisy image features. The noisy image features and the original image features are then input into a segmentation network to obtain predicted gland instance features and filter them. Based on multi-scale features and filtered gland instance features, the gland instance segmentation results are obtained; The step of inputting noisy image features and original image features into a segmentation network to obtain predicted gland instance features and filter them includes: performing correlation operations on noisy image features and original image features to obtain fused extracted features; inputting the fused extracted features into an image decoder based on a diffusion model for processing to obtain predicted gland instance features; and inputting the predicted gland instance features into an instance-aware filter for filtering to obtain the final predicted gland instance features. The process of obtaining gland instance segmentation results based on multi-scale features and filtered gland instance features specifically involves: inputting the multi-scale features and filtered gland instance features into the instance segmentation model to obtain the bounding box and mask of each gland instance; the instance segmentation model is a masking network.
2. The colonic histological image gland segmentation method based on the instance-aware diffusion model as described in claim 1, characterized in that, The specific steps for fusing the original image features and Gaussian noise to generate noisy image features are as follows: Randomly generate Gaussian noise and generate multiple noise boxes based on the original image features.
3. The colonic histological image gland segmentation method based on the instance-aware diffusion model as described in claim 1, characterized in that, The step of inputting the fused extracted features into the image decoder for processing includes: The fusion-extracted features are back-diffused using a Markov chain to transform them into gland instance features.
4. A colonic histological image gland segmentation system based on an instance-aware diffusion model, characterized in that, include: The acquisition module is configured to acquire histological images of the colon to be segmented; The colon histology image gland segmentation module is configured to: input the colon histology image to be segmented into a preset instance-aware diffusion model for processing to obtain gland segmentation results; specifically including: The colon histology images to be segmented are input into the trained image decoder and FPN network respectively to obtain the original image features and multi-scale features; The original image features and Gaussian noise are fused to generate noisy image features. The noisy image features and the original image features are then input into a segmentation network to obtain predicted gland instance features and filter them. Based on the filtered gland instance features and multi-scale features, the gland instance segmentation results are obtained; The step of inputting noisy image features and original image features into a segmentation network to obtain predicted gland instance features and filter them includes: performing correlation operations on noisy image features and original image features to obtain fused extracted features; inputting the fused extracted features into an image decoder based on a diffusion model for processing to obtain predicted gland instance features; and inputting the predicted gland instance features into an instance-aware filter for filtering to obtain the final predicted gland instance features. The process of obtaining gland instance segmentation results based on filtered gland instance features and multi-scale features specifically involves: inputting the multi-scale features and filtered gland instance features into the instance segmentation model to obtain the bounding box and mask of each gland instance; the instance segmentation model is a masking network.
5. An electronic device, characterized in that, The method includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor, which, when executed by the processor, perform the gland segmentation method for colon histology images based on an instance-aware diffusion model as described in any one of claims 1-3.
6. A computer-readable storage medium, characterized in that, Used to store computer instructions, which, when executed by a processor, perform the gland segmentation method for colon histology images based on an instance-aware diffusion model as described in any one of claims 1-3.