Semi-supervised image semantic segmentation method based on similarity weighted multi-teacher network model

Abstract

The application discloses a kind of semi-supervised image semantic segmentation methods based on similarity weighting multi-teacher network model, comprising the following steps: preparing training set and verification set data for semantic segmentation model training;Construct similarity weighting based multi-teacher network model SW-MMTNet;The unlabeled data of training set is respectively subjected to strong data enhancement and weak data enhancement, to realize the data disturbance of unlabeled data;The training of SW-MMTNet model is carried out using the labeled data and unlabeled data of training set;The effect of segmentation model is evaluated using verification set, and the final image semantic segmentation model is obtained.The method can integrate the prediction information of multiple teacher networks, generate high-quality and stable pseudo-labels for unlabeled data, thereby effectively utilizing unlabeled data to assist the training of semantic segmentation model and reducing the demand for large amounts of data for semantic segmentation.In addition, the method also enables the student network of SW-MMTNet to learn more information through data disturbance.

Description

Technical Field

[0001] This invention relates to the field of image semantic segmentation, and in particular to a semi-supervised image semantic segmentation method based on a similarity-weighted multi-teacher network model. Background Technology

[0002] Semantic segmentation is a fundamental pixel-level classification task in computer vision, aiming to understand images at the pixel level. It is crucial for computers to move from detecting images to understanding them. Because image semantic segmentation can analyze image content pixel by pixel, it has wide applications in autonomous driving, medical image processing, and industrial defect detection. Thanks to the development of deep convolutional neural networks, image semantic segmentation has achieved great success in recent years. However, fully supervised image semantic segmentation model training requires a large amount of pixel-level labeled data, which is very expensive and time-consuming. For high-resolution urban road datasets, drawing semantic segmentation annotations on an object takes 8 times longer than drawing bounding box annotations, and 78 times longer than simply labeling the object's category. Semi-supervised image semantic segmentation reduces the need for large amounts of pixel-level labeled data by using a small amount of labeled data and a large amount of unlabeled data to train the segmentation model.

[0003] Most existing semi-supervised image semantic segmentation methods are based on consistency regularization. Consistency regularization, based on smoothness and clustering assumptions, aims to ensure that applying actual perturbations to unlabeled data does not significantly change the network's prediction results. This forces the network to make consistent predictions for unlabeled data from the same source under various perturbations, thus enabling semi-supervised learning using unlabeled data. However, the effectiveness of these methods can be severely affected by unreliable predictions of unlabeled data.

[0004] In consistency regularization methods, the Mean Teacher network (MT) model includes a student network and a teacher network, where the student network is trained by learning the predicted outputs of the teacher network. However, pseudo-labels obtained from the predicted outputs of a single teacher network may be unreliable and noisy, potentially leading to incorrect learning by the student network. In the field of knowledge distillation, existing techniques, referencing real-world scenarios where a student learns different knowledge from multiple teachers, propose multi-teacher knowledge distillation, allowing the student network to extract knowledge from multiple teacher networks, thereby improving the learning quality. However, simply averaging and weighting the predicted outputs of multiple teacher networks as the student network's learning objective does not consider the varying accuracy of the predicted outputs from different teacher networks. Summary of the Invention

[0005] The object of the present invention is to achieve efficient image semantic segmentation using a small amount of labeled image data, and a semi-supervised image semantic segmentation method based on a similarity-weighted multi-teacher network model is proposed. The method expands the single teacher network of the MT model into multiple teacher networks, and on this basis, measures the accuracy of the prediction output of each teacher network by calculating the similarity between the prediction output of each teacher network for unlabeled data and the prediction output of other teacher networks for unlabeled data. Then, using the similarity as the weight, the prediction outputs of multiple teacher networks are integrated by similarity weighting, so as to generate better-quality and more stable pseudo-labels for unlabeled data, and then use the unlabeled data for semi-supervised learning of the student network. In addition, the method also performs data perturbation on unlabeled data in a weak data augmentation manner, uses multiple teacher networks to predict pseudo-labels on the data after weak data augmentation, and performs similarity-weighted calculation on the prediction results, so as to obtain the similarity-weighted pseudo-labels of unlabeled data; at the same time, performs data perturbation on the homologous unlabeled data after weak data augmentation in a strong data augmentation manner, and then uses the similarity-weighted pseudo-labels as the learning target when the student network uses the strongly augmented unlabeled data for network training, forcing the student network to learn more information from the perturbed unlabeled data, and further improving the learning quality of the student network.

[0006] To achieve the above object, a semi-supervised image semantic segmentation method based on a similarity-weighted multi-teacher network model SW-MMTNet (Similarity Weighting-Multiple Mean Teacher Network) provided by the present invention includes the following steps:

[0007] Step S1: Prepare the training set and validation set data for training the semantic segmentation model;

[0008] The training set data consists of containing a small amount of labeled images and

[0009] containing a large amount of unlabeled images

[0010] where n << m, and the validation set data consists of labeled images for evaluating the training effect of the model;

[0011] Step S3: Perform strong data augmentation and weak data augmentation on the unlabeled data in the training set respectively to achieve data perturbation of the unlabeled data;

[0012] Step S5: Evaluate the effectiveness of the segmentation model using the validation set to obtain the final segmentation model.

[0013] Furthermore, in step S2, the similarity-weighted multi-teacher network model SW-MMTNet consists of a student network. and n teacher networks , , … Composition, n≥3, where , , , … They are , , , … The network parameters are initialized differently during the training of SW-MMTNet to increase the differences between different teacher networks, thereby better distinguishing the quality of prediction results of different teacher networks through similarity. Both the student network and the teacher network use the supervised semantic segmentation model deeplabv3+. Using other supervised semantic segmentation models besides deeplabv3+ will not affect the implementation of the method, as long as the student network and the teacher network use the same network structure.

[0014] Furthermore, in step S3, data perturbation of the unlabeled data is intended to enable the student network to learn more information from the perturbed data, thereby further improving the learning quality of the student network. The weak data augmentation used only changes the size and positional relationships of the image without affecting the network's prediction results. This includes random flipping, adjusting the length and width of the image within a range of 0.5 to 2.0 times, and randomly cropping the image into fixed-resolution 321×321 image blocks. The strong data augmentation used, however, changes the color properties of the image, thus affecting the network's prediction results. This includes random grayscale, blurring, and randomly changing the image's brightness, contrast, and saturation.

[0015] Furthermore, in step S4, labeled and unlabeled data are used only for training the SW-MMTNet student network. The parameters of the n teacher networks are updated using the exponential moving average of the student network parameters at the end of each model iteration. Specifically, for labeled data... The student network is trained using traditional supervised learning methods; however, for unlabeled data... SW-MMTNet first uses a similarity-weighted algorithm to augment unlabeled data from weak data generated by n teacher networks. Obtain similarity-weighted pseudo-labels from the predicted output. Then, similarity-weighted pseudo-labels are used. As student networks use unlabeled data that has undergone strong data augmentation The learning objectives during network training enable students to utilize unlabeled data. The specific process of network training is as follows:

[0016] Step S4.1: Transfer unlabeled data After weak data augmentation, the data is input into n teacher networks to obtain the n teacher networks' feedback on the unlabeled data. The pseudo-label prediction output, where the j-th teacher network predicts the unlabeled data. Pseudo-label prediction output for:

[0017] (1)

[0018] In the formula This indicates that weak data augmentation is being performed. This indicates that the j-th teacher network has parameters In this case, for unlabeled data that has undergone weak data augmentation The pseudo-label prediction output;

[0019] Step S4.2: Calculate the similarity between the pseudo-label prediction output of each teacher network and the pseudo-label prediction output of other teacher networks, and use the similarity calculation result as the confidence score of the pseudo-label prediction output of each teacher network, where the confidence score of the pseudo-label prediction output of the j-th teacher network is... for:

[0020] (2)

[0021] In the formula, mIoU() represents the ratio of the intersection to the union of two sets. and Let represent the pseudo-label prediction outputs of the j-th teacher network and the k-th teacher network, respectively;

[0022] Step S4.3: Using similarity as the weight, perform similarity-weighted integration of the predicted outputs of the n teacher networks, and obtain pseudo-labels from the similarity-weighted integration result using the argmax function. :

[0023] (3)

[0024] The argmax function in the formula returns the category with the highest probability value from the similarity-weighted result (category probability map), thus obtaining the pseudo-label for the unlabeled data. ;

[0025] Step S4.4: Transfer the weakly augmented, homologous unlabeled data After strong data augmentation, the data is input into the student network to obtain the pseudo-label prediction output of the student network. :

[0026] (4)

[0027] In the formula, This indicates the need for strong data augmentation. This indicates that weak data augmentation is being performed. The student network, under parameter θ, performs well on unlabeled data. The prediction results;

[0028] Step S4.5: Analyze the prediction results and pseudo-tags The consistency loss between them is minimized using the standard pixel cross-entropy loss function, thereby enabling the training of the student network using unlabeled data.

[0029] The present invention, by adopting the above technical solution, has the following beneficial effects:

[0030] (1) This invention expands the single teacher network of the MT model into multiple teacher networks and integrates the prediction outputs of multiple teacher networks using similarity weighting as the training target for the student network, thereby providing more stable and higher-quality pseudo-labels for unlabeled data. On the one hand, when a teacher network makes a wrong prediction, the divergence information from the correct prediction results of other teacher networks can, to some extent, correct the prediction error of the single teacher network, thereby reducing the impact of the single teacher network's prediction error. On the other hand, the higher the similarity between the prediction outputs of different teacher networks, the higher their accuracy to some extent. Therefore, the similarity weighting integration approach takes into account the quality of the prediction outputs of different teacher networks and assigns greater weight to the prediction outputs of teacher networks with higher confidence, thereby obtaining higher-quality pseudo-labels.

[0031] (2) The present invention achieves data perturbation by performing strong and weak data augmentation on the unlabeled data when training SW-MMTNet using unlabeled data, so that the student network can learn more information from the perturbed data, thereby further improving the learning quality of the student network.

[0032] (3) This invention proposes a semi-supervised image semantic segmentation method based on a similarity-weighted multi-teacher network model, which can effectively utilize unlabeled data to assist in the training of the semantic segmentation model, reduce the need for a large amount of pixel-level labeled data for semantic segmentation, and thus enable semantic segmentation to be applied in more real-world scenarios. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. The drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0034] Figure 1 This is a flowchart of a semi-supervised image semantic segmentation method based on a similarity-weighted multi-teacher network in an embodiment of the present invention;

[0035] Figure 2 This is a schematic diagram illustrating the use of unlabeled data for training of the SW-MMTNet model in an embodiment of the present invention;

[0036] Figure 3 This is a performance comparison diagram of SW-MMTNet and the fully supervised baseline model (deeplabv3+) in an embodiment of the present invention;

[0037] Figure 4 This is a comparison of the segmentation performance of SW-MMTNet and the fully supervised baseline model (deeplabv3+) on some unlabeled images in an embodiment of the present invention. Detailed Implementation

[0038] like Figure 1 As shown, a semi-supervised image semantic segmentation method based on a similarity-weighted multi-teacher network model is presented. Figure 1 The flowchart of the method includes the following steps:

[0039] Step 101: Prepare the training and validation sets for training the semantic segmentation model.

[0040] In this analysis embodiment, the dataset used for training and evaluating the SW-MMTNet model is the Pascal VOC2012 augmented dataset. The original Pascal VOC 2012 dataset consists of 1464 images as the training set and 1449 images as the validation set. By introducing the Semantic Boundary Dataset (SBD), the training set is expanded to 10582 images, resulting in the Pascal VOC 2012 augmented dataset. Then, images at 1 / 16, 1 / 8, and 1 / 4 of the training set are respectively set as labeled dataset D.l The remaining images are ignored and treated as an unlabeled dataset D. u use.

[0041] Step 102: Construct a similarity-weighted multi-teacher network model SW-MMTNet.

[0042] like Figure 2 As shown, in this embodiment, the constructed similarity-weighted multi-teacher network model SW-MMTNet consists of a student network. and three teacher networks , , The network consists of three networks. Both the student and teacher networks use the Deeplabv3+ network with ResNet-50 as the backbone. When SW-MMTNet is trained on unlabeled data, the three teacher networks provide similarity-weighted pseudo-labels to the student networks, which then use these similarity-weighted pseudo-labels as the learning objective for network training.

[0043] Step 103: Perform strong data augmentation and weak data augmentation on the unlabeled data respectively to achieve data perturbation of the unlabeled data.

[0044] In this embodiment, weak data enhancement for unlabeled data includes random flipping, adjusting the length and width of the image within a range of 0.5 to 2.0 times, and randomly cropping the image into image blocks with a fixed resolution of 321×321; strong data enhancement for unlabeled data includes random grayscale, blurring, and random changes to the brightness, contrast, and saturation of the image.

[0045] Step 104: Train the SW-MMTNet model using a combination of labeled and unlabeled data.

[0046] In this embodiment, before training the SW-MMTNet model, the student network and three teacher networks use a ResNet-50 model pre-trained on ImageNet to initialize the parameters of the backbone network, while the Kaiming initialization method is used to randomly initialize the parameters of the segmentation network. During SW-MMTNet model training, a labeled dataset D is used. l and unlabeled dataset D u This is intended solely for training student networks. Specifically, it includes labeled data. The student network is trained using traditional supervised learning methods; however, for unlabeled data... SW-MMTNet first uses a similarity-weighted algorithm to augment unlabeled data from weak data generated by three teacher networks. Similarity-weighted pseudo-labels obtained from the predicted output Then, similarity-weighted pseudo-labels are used. Unlabeled data with strong data augmentation for student network use The learning objectives during network training enable students to utilize unlabeled data. Network training was conducted. During training, a mini-batch stochastic gradient descent (SGD) optimizer with momentum was used to train the student network, and a poly learning strategy was employed to dynamically adjust the learning rate. The initial learning rate of the backbone network was set to 0.001, the initial learning rate of the partitioning network was set to 0.01, the momentum of SGD was fixed at 0.9, the weight decay rate was set to 0.001, and the batch size was set to 8. Following the experimental setup described above, the SW-MMTNet model was trained iteratively for 120 rounds using labeled and unlabeled data at data partitioning ratios of 1 / 16, 1 / 8, and 1 / 4, respectively. The process of obtaining similarity-weighted pseudo-labels from the predicted outputs of the three teacher networks using a similarity-weighted algorithm was as follows:

[0047] Step A1: Transfer unlabeled data After weak data augmentation, the data is input into three teacher networks to obtain unlabeled data from the three teacher networks regarding the weak data augmentation. Pseudo-label prediction output , , ;

[0048] Step A2: Calculation , , The similarity between them is used to obtain:

[0049] confidence level ;

[0050] confidence level ;

[0051] confidence level .

[0052] Step A3: Using confidence level as weight, perform similarity weighting on the pseudo-label prediction outputs of the three teacher networks, and then... The function extracts similarity-weighted pseudo-labels from the similarity-weighted results of unlabeled data. .

[0053] Step 105: Evaluate the effectiveness of the segmentation model using the validation set to obtain the final segmentation model.

[0054] In this embodiment, the mean Intersection over Union (mIoU) of the segmentation model on the Pascal VOC 2012 validation set is used as the evaluation metric for the model training performance. mIoU is the ratio of the intersection to the union of the true and predicted values, and its calculation is shown in Equation (5):

[0055] (5)

[0056] In the formula, K is the total number of object categories in the dataset, i is the i-th category out of K categories, and p ij This represents the number of pixels of category i predicted as category j.

[0057] Figure 3 This embodiment demonstrates the improvement of SW-MMTNet over the segmentation performance of a fully supervised baseline model on the PASCAL VOC 2012 augmented dataset. The baseline model in this embodiment is the supervised semantic segmentation model deeplabv3+, which is trained using only the partitioned labeled data. Through comparison... Figure 3 The results comparing the fully supervised baseline model with SW-MMTNet show that SW-MMTNet significantly improves the segmentation performance compared to the fully supervised baseline model. Specifically, under data partitioning criteria of 1 / 16, 1 / 8, and 1 / 4, SW-MMTNet achieves mIoU of 72.56%, 74.22%, and 75.75%, respectively, which are improvements of 7.44%, 5.23%, and 5.08% compared to the baseline model that only uses the partitioned labeled data for fully supervised learning. Therefore, this method can significantly improve the segmentation performance of the baseline model by utilizing additional unlabeled data to assist in the training of the segmentation model. It can reduce the need for large amounts of pixel-level labeled data in semantic segmentation, and can better apply semantic segmentation to fields with limited data, such as industrial defect detection and medical image processing.

[0058] Furthermore, Figure 4 This example demonstrates the segmentation performance of the fully supervised baseline model (deeplabv3+) and the SW-MMTNet model, trained under a 1 / 8 scale labeled data partitioning standard, on a portion of unlabeled images. From... Figure 4As can be seen, compared to the fully supervised baseline model, the SW-MMTNet model in this embodiment, by learning from perturbed unlabeled data, has the following advantages: 1) Better robustness. For example, for the input image with low contrast in the first row and the input image with lighting interference in the second row, the fully supervised baseline model incorrectly predicts some pixels belonging to the background category as the foreground category. However, SW-MMTNet, trained using perturbed unlabeled data, has better anti-interference ability than the fully supervised baseline model and therefore does not exhibit the aforementioned errors. 2) Stronger discrimination ability. For example, in the prediction results of the fully supervised baseline model in the third row, some pixels belonging to "train" are predicted as "car"; in the prediction results of the fully supervised baseline model in the fourth row, some pixels belonging to "dog" are predicted as "sheep". However, the SW-MMTNet model in this embodiment, by learning from unlabeled data, can obtain more information about the differences between similar objects, enabling it to correctly predict the category of similar objects and exhibiting stronger discrimination ability, thus avoiding the aforementioned errors. 3) Stronger recognition ability. In the prediction results of the fifth row, the fully supervised model barely recognized the "bottle" on the table; in the prediction results of the sixth row, the fully supervised model completely failed to recognize the "person" next to the airplane, while the SW-MMTNet model was able to learn more information that was helpful for object recognition from the unlabeled data, thus having a stronger object recognition ability and being able to recognize the "bottle" and the "person".

[0059] Among existing semi-supervised image semantic segmentation methods based on consistency regularization, PseudoSeg is a method proposed by Zou et al. at the ICLR conference in 2021; MT is the original method before the improvement of SW-MMTNet, proposed by Tarvainen et al. at the NIPS conference in 2017; CCT is a method proposed by Ouali et al. at the CVPR conference in 2020; GCT is a method proposed by Ke et al. at the ECCV conference in 2020; DCC is a method proposed by Lai et al. at the CVPR conference in 2021; and CPS is a method proposed by Chen et al. at the CVPR conference in 2021.

[0060] In this embodiment, the average intersection-over-union ratio (mIoU) of the segmentation model on the Pascal VOC 2012 validation set is used as the evaluation metric for model training performance. The SW-MMTNet model is compared with the aforementioned existing semi-supervised image semantic segmentation methods, and the comparison results are shown in Table 1. In Table 1, the scores (e.g., "1 / 16") represent the proportion of labeled data, and the numerical units in Table 1 are (mIoU, %). By comparing the segmentation performance of SW-MMTNet with that of existing semi-supervised image semantic segmentation methods in Table 1, it can be seen that the SW-MMTNet model in this embodiment achieves superior segmentation performance compared to other existing methods.

[0061] Table 1

[0062] Method / Tag Ratio 1 / 16 1 / 8 PseudoSeg 65.44 69.15 MT 66.77 70.78 CCT 65.22 70.87 GCT 64.05 70.47 DCC 70.1 72.4 CPS 71.98 73.67 SW-MMTNet (This invention) 72.56 74.22

Claims

1. A semi-supervised image semantic segmentation method based on a similarity-weighted multi-teacher network model, characterized in that, Includes the following steps: Step S1: Prepare the training and validation sets for training the semantic segmentation model; The training set data includes a small number of labeled images. and containing a large number of unlabeled images The validation set data consists of labeled images used to evaluate the model training performance; Step S2: Construct a similarity-weighted multi-teacher network model SW-MMTNet; The similarity-weighted multi-teacher network model SW-MMTNet described in step S2 consists of a student network. and n teacher networks , , … The network consists of n≥3, where both the student network and the teacher network use the supervised semantic segmentation model deeplabv3+. , , , … They represent , , , … The corresponding network parameters; using other supervised semantic segmentation models besides deeplabv3+ will not affect the implementation of the method, as long as the student network and the teacher network use the same network structure. Step S3: Perform strong data augmentation and weak data augmentation on the unlabeled data in the training set to perturb the unlabeled data; Step S4: Train the SW-MMTNet model using both labeled and unlabeled data; In step S4, when SW-MMTNet trains the student network using unlabeled data, the pseudo-labels of the unlabeled data... The unlabeled data is analyzed from n teacher networks using a similarity-weighted algorithm. The pseudo-labels are obtained from the prediction output, and the specific process is as follows: Step S4.1: Transfer unlabeled data After weak data augmentation, the data is input into n teacher networks to obtain the n teacher networks' feedback on the unlabeled data. The pseudo-label prediction output, where the j-th teacher network predicts the unlabeled data. Pseudo-label prediction output for: （1） In the formula This indicates unlabeled data Perform weak data augmentation. This indicates that the j-th teacher network has parameters In this case, for unlabeled data that has undergone weak data augmentation The network output results; Step S4.2: Calculate the similarity between the pseudo-label prediction output of each teacher network and the pseudo-label prediction output of other teacher networks to obtain the confidence score of the pseudo-label prediction output of each teacher network, where the confidence score of the pseudo-label prediction output of the j-th teacher network is... for: （2） In the formula, mIoU() represents the ratio of the intersection to the union of two sets. and Let represent the pseudo-label prediction outputs of the j-th teacher network and the k-th teacher network, respectively; Step S4.3: Using similarity as the weight, perform similarity-weighted integration of the predicted outputs of the n teacher networks, and obtain pseudo-labels from the similarity-weighted integration result using the argmax function. : （3） The argmax function in the formula returns the category with the highest probability value from the similarity-weighted results, thus obtaining the pseudo-label for the unlabeled data. ; Step S5: Evaluate the effectiveness of the segmentation model using the validation set to obtain the final segmentation model.

2. The semi-supervised image semantic segmentation method based on a similarity-weighted multi-teacher network model according to claim 1, characterized in that, In step S3, data perturbation is applied to the unlabeled data. The weak data augmentation used only changes the size and positional relationships of the image without affecting the network's prediction results. This includes random flipping, adjusting the length and width of the image within a range of 0.5 to 2.0 times, and randomly cropping the image into fixed-resolution 321×321 image blocks. The strong data augmentation used changes the color properties of the image, thereby affecting the network's prediction results. This includes random grayscale, blurring, and random changes to the brightness, contrast, and saturation of the image. The purpose of perturbing the unlabeled data is to enable the student network to learn more information from the perturbed data, thereby further improving the learning quality of the student network.

Images