A semi-supervised segmentation method based on consistency between annotations and predictions

Abstract

This paper discloses a semi-supervised segmentation method based on annotation and prediction consistency. First, a dataset of image samples with fuzzy segmentation targets is obtained and preprocessed. Then, an annotation and prediction consistency model is constructed and trained. Finally, the trained annotation and prediction consistency model is input into the test set image information, and a segmentation mask corresponding to the input image information is output. This method introduces multiple annotations into semi-supervised segmentation, leveraging the diversity and complementarity of multiple annotations to extract key information about object boundaries for accurate segmentation. Compared to other semi-supervised methods that rely solely on a single annotation, the annotation and prediction consistency network performs well in segmenting medical images with fuzzy boundaries.

Description

Technical Field

[0001] The present invention relates to the technical field of computer-aided pathological diagnosis, and in particular to a semi-supervised segmentation method based on annotation and prediction consistency. Background Art

[0002] Medical image segmentation plays a key role in computer-aided diagnosis systems, but obtaining large-scale labeled datasets is challenging. To overcome this limitation, semi-supervised learning has attracted attention in the field of medical image segmentation by incorporating unlabeled data. Consistency regularization is a commonly used method in semi-supervised semantic segmentation, which promotes consistent predictions by introducing various perturbations. However, noise and bias are common in medical image segmentation, especially when dealing with fuzzy edges. The development and research of semi-supervised segmentation techniques for fuzzy edges remains relatively limited. The presence of fuzzy edges prevents labeled data from providing accurate prior information, while consistency regularization struggles to address the uncertainty and subjective interpretation of fuzzy edges. Multi-annotation is a strategy to address this issue, combining pseudo-annotations generated by multiple labelers or different algorithms. However, such merging can introduce new biases, necessitating the development of methods that effectively utilize information from multiple annotations. Existing methods can be categorized into two categories: fusion strategies, which aim to efficiently and reasonably combine multiple annotations to obtain more reliable and accurate segmentation results; and direct learning, which leverages the diverse and complementary information in the annotations to improve segmentation performance. Direct learning methods consider the consistency or inconsistency between annotations and can effectively learn from multiple sources of information. Therefore, developing a multi-annotation network is an effective approach in semi-supervised segmentation tasks with fuzzy edges. Summary of the Invention

[0003] To address the shortcomings of existing methods, this paper proposes a semi-supervised segmentation method based on the consistency of annotations and predictions. We introduce multiple annotations into semi-supervised segmentation. By leveraging the diversity and complementarity of multiple annotations, we extract key information about the edges of the segmented object for accurate segmentation.

[0004] The semi-supervised segmentation method based on annotation and prediction consistency of the present invention is achieved through the following technical features:

[0005] A semi-supervised segmentation method based on consistency between annotations and predictions, comprising the following steps:

[0006] Step 1, obtain an image sample dataset with fuzzy segmentation targets;

[0007] Step 2: preprocess the data set;

[0008] Step 3: Build and train the annotation and prediction consistency model;

[0009] Step 4: Input the test set image information into the trained annotation and prediction consistency model, and output the segmentation mask corresponding to the input image information.

[0010] Furthermore, the specific method of step 1 is as follows:

[0011] The dataset of image samples with fuzzy segmentation targets uses an existing dataset and is divided into a training set and a test set. In the training set, 10% of the data contains ground truth masks annotated by multiple experts, while the remaining 90% of the data does not contain ground truth masks. In the test set, all data contains only one corresponding ground truth mask.

[0012] Furthermore, the specific method of step 3 is as follows:

[0013] The annotation and prediction consistency model consists of K Unet architecture networks with different initialization values, one for each annotation.

[0014] The training prediction and annotation consistency model is achieved by continuously iterating during training so that all K networks learn from each other and reach a consistent prediction result for the training set. Specifically, this is achieved by all networks performing a consistency operation at each iteration.

[0015] The consistency operation described above involves randomly selecting another network as a comparison network for each iteration of training. Two different types of consistency are used between networks: inter-annotation consistency and inter-prediction consistency, to constrain the networks to learn from each other and maintain consistency.

[0016] The inter-annotation consistency is achieved by performing pixel-level consistency operations on multiple annotations of the same image, separating consistent pixel sets from inconsistent pixel sets, and only using information from reliable consistent pixel sets to train the network.

[0017] The inter-prediction consistency includes multi-annotation prediction consistency and unannotated prediction consistency: multi-annotation prediction consistency is used to identify and correct annotations and refine fuzzy edge annotations, specifically by replacing annotations in inconsistent pixel sets with pseudo-labeled pixels that are predicted to be consistent between networks; unannotated prediction consistency constrains the prediction consistency between networks on unannotated data by encouraging networks with different initial values ​​to produce consistent outputs for the same sample.

[0018] Furthermore, the image samples with fuzzy segmentation targets adopt the ISIC dataset and the RIGA dataset.

[0019] Furthermore, the data preprocessing includes adjusting the image size of the input network to 256×256 pixels and normalizing it using the mean and standard deviation of each channel.

[0020] Furthermore, the annotation and prediction consistency model is constructed by constructing a Unet architecture network with K different initialization values. The consistency between annotations and predictions in the consistency operation is achieved by using different loss functions during training. The specific loss function includes three parts: cross entropy loss for the set of pixels with consistent annotations Cross entropy loss for consistent predictions between annotations Pseudo-supervised cross entropy loss on unannotated data

[0021] Furthermore, the inter-annotation consistency loss of the k-th network is:

[0022]

[0023] in is the pixel point with the same annotation as the i-th annotation of the k-th annotation, is the kth annotation pixels, is the number of pixels contained in the consistent pixel set of the kth annotation, is with The corresponding predicted pixel point, is the cross entropy loss.

[0024] The multi-annotation prediction consistency loss of the k-th network is:

[0025]

[0026] in is the pixel whose i-th annotation of the k-th annotation is inconsistent but whose prediction is consistent, is the kth network with inconsistent annotations but consistent predictions pixels, is the corresponding total number of pixels, is with The corresponding predicted pixel point.

[0027] The unannotated consistency loss for the k-th network is:

[0028]

[0029] in is the unannotated prediction mask of the k-th network’s contrast network j-th network, is the predicted probability map of the unannotated data of the kth network, and n is the total number of pixels. The total loss function is the sum of the loss functions of all K networks:

[0030]

[0031] Among them, α, β, and λ are the hyperparameters of the loss function.

[0032] Furthermore, the test set input network obtains the output segmentation mask by averaging the predicted probability maps of all K networks;

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] The present invention proposes a semi-supervised medical image segmentation method based on an annotation and prediction consistency network, which is particularly suitable for images with fuzzy boundaries. Multiple annotations are introduced into semi-supervised segmentation, and through the diversity and complementarity of multiple annotations, key information of object boundaries is extracted to achieve accurate segmentation. Accurate prior knowledge is provided to the network through pixel-level multi-annotation consistency. For pixels with inconsistent annotations, these annotations are corrected through the network's multi-annotation prediction consistency. For unannotated data, unannotated prediction consistency is used to encourage networks with different initializations to produce consistent outputs for the same sample. In the inference stage, the predicted probability maps of all networks are summed and averaged to obtain the final prediction mask. This average fusion method combines the knowledge of multiple networks to produce robust and reliable predictions. Compared with other semi-supervised methods that rely only on a single annotation, the annotation and prediction consistency network performs well in processing medical image segmentation with fuzzy boundaries. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0036] Figure 1 A structural diagram of the annotation and prediction consistency model of the present invention;

[0037] Figure 2 This is a network structure diagram of the backbone network Unet of the present invention. DETAILED DESCRIPTION

[0038] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0039] This paper is implemented in the Pytorch deep learning framework, and the computer configuration uses: RTX3090 graphics card, 24G video memory, and Linux operating system. The present invention provides a semi-supervised segmentation method based on annotation and prediction consistency, which specifically includes the following steps:

[0040] Step 1, obtain an image sample dataset with fuzzy segmentation targets;

[0041] The ISIC and RIGA datasets are used as examples of images with fuzzy segmentation targets. The ISIC dataset contains two experts' different annotations of skin disease areas. The RIGA dataset contains six experts' different annotations of the optic disc and cup. In the training set, 10% of the data contains ground truth masks annotated by multiple experts, while the remaining 90% of the data does not contain a ground truth mask. In the test set, all data contain only a single ground truth mask.

[0042] Step 2: preprocess the data set;

[0043] Data preprocessing involves resizing the input images to 256 × 256 pixels and normalizing them using per-channel mean and standard deviation.

[0044] Step 3: Build and train the annotation and prediction consistency model;

[0045] As attached Figure 1-2 As shown in Figure 2, the annotation and prediction consistency model is specifically designed to solve semi-supervised medical image segmentation under fuzzy boundaries. The model contains K (assuming there are K annotations) Unet architecture networks with different initialization values, one for each annotation.

[0046] When building the annotation and prediction consistency model, the basic backbone network uses Unet, which specifically includes K (assuming there are K annotations) networks with Unet architectures with different initialization values, each corresponding to one annotation.

[0047] The training of the prediction and annotation consistency model is achieved by continuous iteration during training so that all K networks learn from each other and the prediction results of the training set tend to be consistent. When training the annotation and prediction consistency model, it is specifically achieved through consistency operations. First, the consistency operation is that for each iteration during training, each network randomly selects another network as a comparison network. Subsequently, two different consistency methods are used between the two networks, inter-annotation consistency and inter-prediction consistency, to constrain the networks to learn from each other and remain consistent. Inter-annotation consistency performs pixel-level consistency operations on multiple annotations of the same image and separates consistent pixel sets from inconsistent pixel sets. Only information from reliable consistent pixel sets is used to train the network. The inter-annotation consistency loss of the kth network is:

[0048]

[0049] in is the pixel point with the same annotation as the i-th annotation of the k-th annotation, is the kth annotation pixels, is the number of pixels contained in the consistent pixel set of the kth annotation, is with The corresponding predicted pixel point, is the cross entropy loss. Inter-prediction consistency includes multi-annotation prediction consistency and unannotated prediction consistency: multi-annotation prediction consistency is used to identify and correct annotations and refine fuzzy edge annotations by replacing the annotations in the inconsistent pixel set with pseudo-labeled pixels that are predicted to be consistent across networks. The multi-annotation prediction consistency loss of the k-th network is:

[0050]

[0051] in is the pixel whose i-th annotation of the k-th annotation is inconsistent but whose prediction is consistent, is the kth network with inconsistent annotations but consistent predictions pixels, is the corresponding total number of pixels, is with The corresponding predicted pixel point. Unannotated prediction consistency is to further constrain the consistency between networks by encouraging networks with different initializations to produce consistent outputs for the same sample. The unannotated consistency loss of the k-th network is:

[0052]

[0053] in is the unannotated prediction mask of the k-th network’s contrast network j-th network, is the predicted probability map of the unannotated data of the kth network, and n is the total number of pixels. The total loss function is the sum of the loss functions of all K networks:

[0054]

[0055] Among them, α, β, and λ are the hyperparameters of the loss function.

[0056] Step 4: Input the test set image information into the trained annotation and prediction consistency model, and output a segmentation mask corresponding to the input image information;

[0057] Input the predicted image and get the predicted probability map of all K networks {P 1, P 2,..., P k}, average all the predicted probability maps to get the final predicted mask P.

[0058] Although the embodiments of the present invention have been disclosed above, they are not limited to the applications listed in the description and implementation methods. They can be fully applied to various fields suitable for the present invention. For those familiar with the art, additional modifications can be easily implemented. Therefore, without departing from the general concept defined by the claims and the scope of equivalents, the present invention is not limited to specific details.

Claims

1. A semi-supervised segmentation method based on consistency between annotations and predictions, characterized in that The following steps are involved: Step 1, obtain an image sample dataset with fuzzy segmentation targets; Step 2: preprocess the data set; Step 3: Build and train the annotation and prediction consistency model; Step 4: Input the test set image information into the trained annotation and prediction consistency model, and output a segmentation mask corresponding to the input image information; Step 3: The annotation and prediction consistency model consists of K Unet architecture networks with different initialization values, one for each annotation; The training annotation and prediction consistency model is achieved by continuously iterating during training so that all K networks learn from each other and tend to have consistent prediction results for the training set; specifically, this is achieved by all networks performing consistency operations at each iteration; The consistency operation is that during each iteration of training, each network randomly selects another network as a comparison network; two different consistency methods are used between networks: inter-annotation consistency and inter-prediction consistency, to constrain the networks to learn from each other and maintain consistency; The inter-annotation consistency is achieved by performing pixel-level consistency operations on multiple annotations of the same image, separating consistent pixel sets from inconsistent pixel sets, and only using information from reliable consistent pixel sets to train the network; The inter-prediction consistency includes multi-annotation prediction consistency and unannotated prediction consistency: multi-annotation prediction consistency is used to identify and correct annotations and refine fuzzy edge annotations, specifically by replacing annotations in inconsistent pixel sets with pseudo-labeled pixels that are predicted to be consistent between networks; unannotated prediction consistency constrains the prediction consistency of unannotated data between networks by encouraging networks with different initial values ​​to produce consistent outputs for the same sample; The annotation and prediction consistency model contains K networks with different initialization values ​​of the Unet architecture; the consistency between annotations and predictions in the consistency operation is achieved by using different loss functions during training; the specific loss function consists of three parts: the cross entropy loss of the set of pixels that are consistent between annotations Cross entropy loss for consistent predictions between annotation inconsistencies Pseudo-supervised cross entropy loss on unannotated data 2. A semi-supervised segmentation method based on annotation and prediction consistency according to claim 1, characterized in that Step 1: The image sample dataset with fuzzy segmentation targets adopts an existing dataset and is divided into a training set and a test set; in the training set, 10% of the data contains true value masks annotated by multiple experts, and the remaining 90% of the data does not contain true value masks; In the test set, all data contain only one corresponding ground-truth mask.

3. A semi-supervised segmentation method based on annotation and prediction consistency according to claim 1, characterized in that The image samples with fuzzy segmentation targets adopt the ISIC dataset and the RIGA dataset.

4. A semi-supervised segmentation method based on annotation and prediction consistency according to claim 2, characterized in that The data preprocessing described includes resizing the input network image to 256×256 pixels and normalizing it using the mean and standard deviation of each channel.

5. The semi-supervised segmentation method based on annotation and prediction consistency according to claim 1, characterized in that The inter-annotation consistency loss of the k-th network is: in is the pixel point with the same annotation as the i-th annotation of the k-th annotation, is the kth annotation pixels, is the number of pixels contained in the consistent pixel set of the kth annotation, is with The corresponding predicted pixel point, l ce is the cross entropy loss; The multi-annotation prediction consistency loss of the k-th network is: in is the pixel whose i-th annotation of the k-th annotation is inconsistent but whose prediction is consistent, is the kth network with inconsistent annotations but consistent predictions pixels, is the corresponding total number of pixels, is with The corresponding predicted pixel point; The unannotated consistency loss for the k-th network is: in is the unannotated prediction mask of the k-th network’s contrast network j-th network, is the predicted probability map of the unannotated data of the kth network, n is the total number of pixels; the total loss function is the sum of the loss functions of all K networks: Among them, α, β, and λ are the hyperparameters of the loss function.

6. A semi-supervised segmentation method based on annotation and prediction consistency according to claim 1 or 5, characterized in that The test set input network obtains the output segmentation mask by averaging the predicted probability maps of all K networks.

Images