Semi-supervised skin image automatic segmentation method based on boundary perception and hybrid prototype consistency learning

Abstract

The invention discloses a semi-supervised skin image automatic segmentation method based on boundary perception and hybrid prototype consistency learning. According to the method, a shared encoder and a teacher-student decoder are used as a network structure, and the teacher decoder and the student decoder are respectively provided with an independent main task area segmentation decoder and an independent auxiliary task boundary segmentation decoder. And combining the image-level feature vector and the feature-level feature vector obtained by the encoder and the decoder to obtain a mixed prototype vector, and calculating prototype-based region segmentation. According to the method, the proposed model is evaluated on three common data sets, the feature information of an auxiliary task is fully utilized, a prototype vector can effectively segment instances with large intra-class gaps and small inter-class gaps, the model with high adaptability is obtained through training of a small amount of labeled data, and the method is high in robustness and high in robustness. Compared with an existing semi-supervised method, the method obtains a better segmentation result, can be used for improving the skin image segmentation performance, and has important application prospects and practical significance.

Description

Technical Field

[0001] The present invention designs a semi-supervised medical image automatic segmentation method, which is a semi-supervised skin image automatic segmentation method based on boundary perception and hybrid prototype consistency learning. Background Art

[0002] In the past, a lot of research has been done on medical image segmentation and skin segmentation tasks. Early medical image segmentation methods mainly relied on contour-based and traditional machine learning algorithms. With the development of deep convolutional neural networks, UNet was introduced to medical image segmentation and many variants were developed. Although a large amount of data is available for the skin lesion segmentation task, the interference of hair, veins and markers, as well as significant intra-class variations and minimal inter-class differences, often lead to subpar performance of the trained network. Therefore, our goal is to use a semi-supervised approach to train high-performance and adaptable multi-task models, because in clinical practice, it is feasible to obtain a small amount of labeled data from expert examinations as well as a large amount of unlabeled images. And by training a multi-task model with boundary awareness and embedding hybrid prototype consistency learning, the influence of hair, veins and markers is eliminated. The following methods have emerged in recent years:

[0003] (1) Boundary-aware network: embed boundary segmentation-related tasks into the model. For example, MTI-Net uses different task heads, BA-Transformer and XBound-Former embed boundary heat map prediction into the feedforward network, and DCoonNet uses the deepest potential features to predict the connection graph of auxiliary tasks.

[0004] (2) Prototype consistency learning: Prototype refers to the centralized representation of feature vectors in the same category, and prototype-based prediction complements the prediction of the segmentation head. UPCoL uses prototypes to represent the uncertainty of the segmentation results, thereby improving the quality of pseudo labels. EPL uses pixel-level double uncertainty masks to combine labeled and unlabeled prototypes. PMCL uses prototypes derived from the deepest features extracted by the encoder and adopts prototype-based consistency loss for supervised training. SSPENet combines prototypes with federated learning and utilizes image intelligence and customer intelligence prototypes. CPCL uses prototypes to transfer precise information from the labeled part to the unlabeled part.

[0005] The above boundary-aware networks are usually unable to clearly distinguish the boundaries between tasks, resulting in limited performance of the model on both tasks. There are also a lot of disturbances in the prototype vector during the extraction process, which lacks representativeness. The present invention aims to design an appropriate network structure that can clearly distinguish the differences between tasks and utilize shared representations of different tasks, while avoiding model disturbances and extracting high-quality and representative prototype vectors. And through semi-supervised learning, a small amount of labeled training data can be used to obtain a high-performance and adaptable model. Summary of the invention

[0006] The purpose of the present invention is to provide a semi-supervised skin image automatic segmentation method based on boundary perception and hybrid prototype consistency learning. The technical solution to achieve the purpose of the present invention is: teacher-student decoder construction based on task-independent decoder UNet and hybrid prototype consistency learning.

[0007] Step 1: Based on the existing medical image dataset, preprocess the data and generate a dilation boundary.

[0008] Step 2: Divide the processed data into training set, test set and validation set.

[0009] Step 3: Design a semi-supervised training framework based on the training set:

[0010] Step 3.1: Use UNet as the backbone network to build a shared encoder, and each task uses a multi-task network architecture with a corresponding independent decoder. Based on the current multi-task network, build a shared encoder, but with independent decoders for the teacher and student decoders as a semi-supervised training framework.

[0011] Step 3.2: Use the feature map after the encoder input convolution to extract the image-level prototype vector, use the feature map of the decoder output before the convolution to extract the feature-level prototype vector, use average pooling to fuse the image-level and feature-level prototype vectors to obtain a mixed prototype vector.

[0012] Step 3.3, calculate the prototype segmentation result based on the cosine similarity of the mixed prototype vector and the feature map of the region segmentation task in the student decoder.

[0013] Step 3.4: The student decoder prediction result is used as the segmentation result of the model, and the teacher decoder prediction result is used as the pseudo label.

[0014] Step 3.5: Based on the loss functions of the skin lesion segmentation task and the dilated boundary segmentation task and the consistency loss between tasks, the loss functions of the student model segmentation results, prototype segmentation results and labels are calculated for labeled data, and the loss functions of the student model segmentation results, prototype segmentation results and pseudo labels are calculated for unlabeled data, and the gradient descent convergence model is used.

[0015] Step 4: Repeat step 3 to train the semi-supervised segmentation network model.

[0016] Step 5: Use the trained segmentation network to test the test data.

[0017] Compared with the prior art, the present invention has the following significant advantages:

[0018] (1) The teacher encoder and the student encoder adopt a multi-task structure with independent decoders and share an encoder. The network can clearly distinguish the differences between tasks, capture the different features of different tasks in the branch network, and mine the common representation of different tasks in the shared part, while reducing the disturbance in the encoder part when the teacher-student model is completely separated.

[0019] (2) The hybrid prototype vector is calculated using the image-level prototype vector and the feature-level prototype vector. The prototype vector is more representative, retains the graphic features, avoids interference such as artifacts in the skin image, and enables the model to perform well in the case of small intra-class differences and large inter-class differences.

[0020] (3) Using the dilated boundary as an auxiliary task, the single-point information of the ordinary boundary is expanded into plane information, which greatly enriches the amount of information on the boundary. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 This is the overall process and internal details of the semi-supervised skin image automatic segmentation method based on boundary perception and hybrid prototype consistency learning of the present invention.

[0022] Figure 2 This is the comparative experimental result of the method of the present invention and several existing semi-supervised methods on a certain skin medical image dataset.

[0023] Figure 3 The results of the method of the present invention are compared with those of several semi-supervised methods on actual test data.

[0024] Figure 4 It is a summary of the specification of the present invention, which describes the overall process logic of the present invention. DETAILED DESCRIPTION

[0025] Combination Figure 1 The semi-supervised skin image automatic segmentation method based on boundary perception and hybrid prototype consistency learning of the present invention comprises the following steps:

[0026] Step 1: Based on the existing medical image dataset, preprocess the data and generate a dilation boundary.

[0027] Step 2: Divide the processed data into training set, test set and validation set.

[0028] Step 3: Design a semi-supervised training framework based on the training set:

[0029] Step 3.1: Use UNet as the backbone network to build a shared encoder, and each task uses a multi-task network architecture with a corresponding independent decoder. Based on the current multi-task network, build a shared encoder, but with independent decoders for the teacher and student decoders as a semi-supervised training framework.

[0030] Step 3.2: Use the feature map after the encoder input convolution to extract the image-level prototype vector, use the feature map of the decoder output before the convolution to extract the feature-level prototype vector, use average pooling to fuse the image-level and feature-level prototype vectors to obtain a mixed prototype vector.

[0031] Step 3.3, calculate the prototype segmentation result based on the cosine similarity of the mixed prototype vector and the feature map of the region segmentation task in the student decoder.

[0032] Step 3.4: The student decoder prediction result is used as the segmentation result of the model, and the teacher decoder prediction result is used as the pseudo label.

[0033] Step 3.5: Based on the loss functions of the skin lesion segmentation task and the dilated boundary segmentation task and the consistency loss between tasks, the loss functions of the student model segmentation results, prototype segmentation results and labels are calculated for labeled data, and the loss functions of the student model segmentation results, prototype segmentation results and pseudo labels are calculated for unlabeled data, and the gradient descent convergence model is used.

[0034] Step 4: Repeat step 3 to train the semi-supervised segmentation network model.

[0035] Step 5: Use the trained segmentation network to test the test data.

[0036] The present invention is further described in detail below with reference to examples:

[0037] The present invention uses medical dermoscopic images as input, and automatically generates lesion area segmentation and dilation boundary segmentation through data preprocessing and feature extraction network.

[0038] The process of the present invention is as follows Figure 1 shown.

[0039] (1) The dataset used for training and testing of the present invention is the International Skin Imaging Collaboration Challenge dataset ISIC2018, which consists of 2694 skin images with segmentation mask labels. According to the generally recognized processing method in the work of others, the dataset is divided into a training set and a test set in a ratio of 7:3 for experiments. The training set contains 1886 images and the test set contains 808 images. Each case data is processed into a picture of 256*256 pixels. Since the amount of data included in the training is small, in order to obtain a better model, we augmented the source data, including randomly selected image flipping, image rotation, adding Gaussian noise, blurring and other operations. We performed semi-supervised learning at a labeled data ratio of 5% and 10%, respectively.

[0040] (2) Figure 1 A model architecture diagram of the present invention is given in FIG.

[0041] The network architecture is a teacher-student semi-supervised network with a shared encoder and task-independent decoders. The student decoder contains an independent boundary segmentation decoder and region segmentation decoder, and the teacher decoder also contains an independent boundary segmentation decoder. We use the loss function to update the model parameters of the encoder and student decoder, and use the exponential moving average of the student decoder to update the model parameters of the teacher decoder.

[0042] The labeled data and unlabeled data are input into a shared encoder, and the feature map after the input convolution is used as the image-level feature map. It is filtered by labels or pseudo-labels to obtain all feature vectors of the foreground and background. All feature vectors of each class are averaged and pooled to calculate the image-level prototype vector. Figure 1 As shown in (b), the same operation extracts the feature-level prototype vector from the student region segmentation decoder features with labeled data and the teacher region segmentation decoder features without labeled data, and finally averages the image-level and feature-level prototype vectors to obtain a mixed prototype vector. The cosine similarity is calculated for each pixel feature of the mixed prototype vector of the foreground and background and the feature map before the convolution of the student segmentation decoder output. The pixel with the higher similarity is taken as the category of the pixel to obtain the prototype prediction.

[0043] In order to improve the auxiliary task of damage boundary segmentation, the dilation operation is used to expand the boundary composed of point information into surface information, which effectively improves the help of boundary segmentation for skin damage area segmentation.

[0044] Loss function: The present invention includes two tasks: skin lesion region segmentation and dilated boundary segmentation. The loss function includes three parts: skin lesion region segmentation loss, dilated boundary segmentation loss, prototype segmentation loss, and inter-task consistency loss. The regional segmentation loss uses BCE and Dice loss functions; the dilated boundary segmentation loss uses weighted CE, MSE and Dice loss functions; the prototype segmentation loss and the regional segmentation loss use BCE and Dice loss functions in a consistent manner; the inter-task consistency loss uses BCE and Dice loss functions when calculating regional consistency, and uses weighted CE, MSE and Dice loss functions when calculating dilated boundary consistency loss.

[0045] (3) Repeat step 3 to train the segmentation network model. The method of the present invention uses mIoU, Dice, Accuracy, and Sensitivity as indicators for judging the quality of the model.

[0046] (4) Use the trained segmentation network to test the test data. The method of the present invention and several existing methods (mean teacher, UAMT, DTC, CPS, cross-teach, UPCoL, UCMT, PMCL, Noisy Student) are compared on the test set. Based on the experimental results, the mIoU, Dice, Accuracy, and Sensitivity of this method have achieved the best experimental results, which are better than the pure Transformer-based cross-teach (swin), and also ahead of the prototype-based UPCoL, PMCL, etc., and higher than the current optimal method Noisy Student.

Claims

1. A semi-supervised skin image automatic segmentation method based on boundary perception and hybrid prototype consistency learning, which features the following steps: Step 1: Based on the existing medical image dataset, preprocess the data and generate a dilation boundary. Step 2: Divide the processed data into training set, test set and validation set. Step 3: Design a semi-supervised training framework based on the training set: Step 3.1: Use UNet as the backbone network to build a shared encoder, and each task uses a multi-task network architecture with a corresponding independent decoder. Based on the current multi-task network, build a shared encoder, but with independent decoders for the teacher and student decoders as a semi-supervised training framework. Step 3.2: Use the feature map after the encoder input convolution to extract the image-level prototype vector, use the feature map of the decoder output before the convolution to extract the feature-level prototype vector, use average pooling to fuse the image-level and feature-level prototype vectors to obtain a mixed prototype vector. Step 3.3, calculate the prototype segmentation result based on the cosine similarity of the mixed prototype vector and the feature map of the region segmentation task in the student decoder. Step 3.4: The student decoder prediction result is used as the segmentation result of the model, and the teacher decoder prediction result is used as the pseudo label. Step 3.5: Based on the loss functions of the skin lesion segmentation task and the dilated boundary segmentation task and the consistency loss between tasks, the loss functions of the student model segmentation results, prototype segmentation results and labels are calculated for labeled data, and the loss functions of the student model segmentation results, prototype segmentation results and pseudo labels are calculated for unlabeled data, and the gradient descent convergence model is used. Step 4: Repeat step 3 to train the semi-supervised segmentation network model. Step 5: Use the trained segmentation network to test the test data.

2. The semi-supervised skin image automatic segmentation method based on hybrid prototype consistency learning according to claim 1 is characterized by: The expansion boundary described in step 1. The specific method is: For the boundary composed of point information, we expand the boundary through the dilation operation and expand the point information into surface information.

3. The semi-supervised skin image automatic segmentation method based on hybrid prototype consistency learning according to claim 1, characterized in that: The semi-supervised boundary perception network in step 3.1 is specifically: Using UNet as the network model, the boundary-aware network shares an encoder; the decoder of the semi-supervised network is divided into two parts: a teacher decoder and a student decoder, both of which contain two parallel independent decoders for region segmentation and dilated boundary segmentation. In general, the network consists of a shared encoder and four independent decoders, namely the teacher region segmentation decoder, the teacher dilated boundary segmentation decoder, the student region segmentation decoder, and the student dilated boundary segmentation decoder.

4. The semi-supervised skin image automatic segmentation method based on hybrid prototype consistency learning according to claim 1, characterized in that: The hybrid prototype vector extraction method described in step 3.2: The input convolution feature map is filtered and averaged through labels or pseudo labels to obtain image-level prototype vectors of the foreground and background; the output convolution feature map is filtered and averaged through labels or pseudo labels to obtain feature-level prototype vectors of the foreground and background; finally, the mean of the prototype vectors of the corresponding categories at the image level and feature level is calculated to obtain the mixed prototype vector.

5. The semi-supervised skin image automatic segmentation method based on hybrid prototype consistency learning according to claim 1, characterized in that: The prototype segmentation result calculation described in step 3.3 is: The foreground and background mixed prototype vectors calculated in step 3.2 are respectively used to calculate the cosine similarity with the feature vector of each pixel in the feature map before the output layer. The category with higher similarity is taken as the category of the pixel, thereby obtaining the prototype-based segmentation result.

Images