Semi-supervised semantic segmentation method based on teacher-student network and foreground prototype

Abstract

The invention provides a semi-supervised semantic segmentation method based on a teacher-student network and a foreground prototype, and belongs to the field of artificial intelligence. The architecture comprises teachers, students and prototype auxiliary models. The teacher and student model adopts a SegFormer structure in which an exhibition attention mechanism is introduced, and comprises an image feature extraction layer, a multi-scale feature fusion layer, a decoder and a linear transformation layer; the prototype auxiliary model comprises an image feature extraction layer, a prototype feature extraction layer, a multi-scale feature fusion layer, a decoder and a segmentation generation layer; the teacher and the prototype model generate pseudo labels for the unlabeled data to guide student model training; training of the student model is combined with supervised loss, unsupervised loss and prototype comparison loss; performing index moving average updating on the teacher model by using the student model; a semantic segmentation map is output from the test data through a student network; according to the method, the classification precision and foreground and background distinguishing capability of semantic segmentation are effectively improved, background noise interference is reduced, and the method is superior to the prior art in multiple evaluation indexes.

Description

Technical Field

[0001] This invention proposes a semi-supervised semantic segmentation method based on teacher-student networks and foreground prototypes, belonging to the field of artificial intelligence. Background Technology

[0002] Deep learning, by learning the inherent patterns and representational levels of sample data, achieves accurate recognition of text, images, and sound data, and is one of the core technologies driving the rapid development of artificial intelligence. In recent years, deep learning networks have been widely applied in computer vision research, with semantic segmentation being a typical computer vision problem. Unlike image classification, which only requires identifying the content present in an image, semantic segmentation is a pixel-level classification. Based on features such as grayscale, color, and texture, it assigns a unique category label to each pixel in the image, dividing the image into several visually independent regions, with pixels belonging to the same category grouped together. Semantic segmentation has wide applications in various fields, such as autonomous driving, smart healthcare, and industrial inspection.

[0003] Training semantic segmentation models typically relies on a sufficient amount of pixel-level labeled data, which requires significant human annotation resources. Therefore, addressing the expansion of labeled datasets and the utilization of unlabeled datasets has become a research hotspot, with semi-supervised semantic segmentation being a representative technique. Semi-supervised learning is a learning method that falls between supervised and unsupervised learning. It utilizes a small amount of labeled data and a large amount of unlabeled data for training, using the assistance of unlabeled data to improve the model's learning performance. Currently, there are five main semi-supervised semantic segmentation methods: adversarial learning, consistency regularization, pseudo-labeling, contrastive learning, and hybrid methods. Adversarial learning involves adversarial training between networks; consistency regularization minimizes the differences between different predictions of the same image; pseudo-labeling involves generating pseudo-labels for unlabeled data based on labeled data and using them in training; contrastive learning groups similar elements based on a loss function; and hybrid methods, combinations of the first four methods, are gaining increasing attention due to their innovative integration. Many current typical technologies are based on teacher-student networks, combining consistency regularization and pseudo-labeling methods. However, using separate training streams, with real labels training labeled data and pseudo-labels training unlabeled data, can easily lead to low-quality pseudo-labels. Furthermore, when iterating over incorrect supervision, the errors of pseudo-labels will accumulate, greatly reducing the prediction and generalization performance of student models. Therefore, making full use of unlabeled data and improving the quality of pseudo-labels have become the main research objectives.

[0004] Therefore, this invention proposes a novel semi-supervised semantic segmentation method. Its first feature is the use of prospective attention for multi-scale feature fusion, which preserves the detailed information of the original features while further capturing global contextual information to obtain richer and more comprehensive features. Its second feature is the use of labeled data to enhance the image level of unlabeled data, enabling pre-interaction between labeled and unlabeled data, decoupling the training stream and obtaining higher-quality pseudo-labels. Its third feature is the use of a prototype segmentation network to provide foreground pseudo-labels, allowing these pseudo-labels to participate secondary in the prediction of unlabeled data, fully utilizing the information in the unlabeled data, reducing the impact of background noise, and improving the performance of both model classification and segmentation. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this paper proposes a prospective attention-based and prototype segmentation network. Attention enhancement captures image contextual information, which is then effectively fused with original features to obtain more fine-grained features. The prototype segmentation network provides effective foreground pseudo-label information. Furthermore, during data processing, labeled data is used to enhance unlabeled data, providing effective data perturbation. This invention offers stronger semi-supervised semantic segmentation capabilities, achieving more accurate and efficient predictions even with limited datasets.

[0006] To achieve the above technical objectives, the present invention adopts the following technical solution:

[0007] 1. A semi-supervised semantic segmentation method based on teacher-student networks and foreground prototypes, comprising the following steps:

[0008] Step 1: Construct a semi-supervised semantic segmentation network, which includes a teacher network, a student network, and a prototype-assisted network. The teacher and student models are the same, using the SegFormer network model, which includes an image feature extraction layer, a multi-scale feature fusion layer, a decoder, and a linear transformation layer. The multi-scale feature fusion layer includes a prospective attention mechanism. The prototype-assisted model includes an image feature extraction layer, a prototype feature extraction layer, a multi-scale feature fusion layer, a decoder, and a segmentation generation layer.

[0009] Step 2: Construct a semi-supervised semantic segmentation dataset. After preprocessing the data with and without labels and with strong and weak enhancements, divide it into training set, validation set and test set; assign pre-training parameters to mit_b2 of the image feature extraction layer.

[0010] Step 3: Input the unlabeled images from the training set into the feature extraction layer of the teacher model to obtain the original features containing high-resolution coarse image features and low-resolution image features. Input the original image features into the multi-scale feature fusion module, and after obtaining the detailed local features of the image using the prospect attention mechanism, add them to the original features. Then, pass through the feedforward network layer to obtain the encoded image features. Input the obtained image features into the decoder to obtain the decoded features, and then pass through the linear transformation layer to obtain the prediction result, which serves as the pseudo-label for the unlabeled image.

[0011] Step 4: Calculate the confidence of the pseudo-labels obtained in Step 3; adopt the random hybrid enhancement method to randomly add the labeled image information to the unlabeled data in the region below the threshold, so as to obtain the strongly enhanced unlabeled image and its pseudo-labels.

[0012] Step 5: Input the labeled images, unlabeled images, and the strongly enhanced unlabeled images and their pseudo-labels obtained in Step 4 into the image feature extraction layer of the prototype model to obtain the original features containing high-resolution coarse image features and low-resolution image features. Input the original image features into the prototype feature extraction layer, and use dot product and global pooling to obtain the foreground prototype of the image. Input the foreground prototype and the original features into the decoder to obtain the decoded features. Finally, fuse them with the original features and send them into the output generation layer to obtain the final prediction result. The prediction result includes labeled image prediction and unlabeled image prediction. The loss of labeled image prediction is calculated with the real label. The loss of unlabeled image prediction is calculated with the pseudo-label of the teacher model obtained in Step 3, and the gradient is backpropagated to correct the weights of the prototype model.

[0013] Step 6: Input labeled and unlabeled images from the training set into the prototype model, repeat step 5, and obtain the prediction results of the prototype model for the images, which will be used as the foreground pseudo-labels for the unlabeled images.

[0014] Step 7: Input labeled and unlabeled images from the training set into the student model, repeat step 3, and obtain the student model's prediction results for the images. The prediction results include the prediction of labeled images and the prediction of unlabeled images. Calculate the loss between the labeled image prediction and the real label, and calculate the comparison loss between the unlabeled image and the foreground pseudo-label of the prototype model obtained in step 6. Use this to filter reliable and unreliable unlabeled data, assign weights, and then calculate the loss with the pseudo-label of the teacher model obtained in step 3. Use gradient backpropagation to correct the weights of the student model, and finally update the teacher model using exponential moving average to complete the training of stage 2.

[0015] Step 8: Perform data augmentation on the unreliable data obtained in Step 7, and repeat the training in Step 7, without comparing it with the prototype model, until the specific training is completed, thus completing the training of Phase 2.

[0016] Step 9: After the semi-supervised semantic segmentation model is trained, it enters the inference stage. At this time, the semi-supervised semantic segmentation model no longer changes the weights. The image to be predicted is fed into the trained teacher model, and Step 3 is repeated to obtain the final semantic segmentation result.

[0017] Furthermore, when constructing the dataset, the weak augmentation methods used were flipping and cropping, which are common techniques in this field, while the strong augmentation methods were color dithering and contrast adjustment. The random data augmentation used during the training phase utilized CutMix to augment unlabeled image data with labeled image data. The specific implementation process is as follows:

[0018] Calculate the confidence level C for pseudo-labels without labeled data within a batch. f The formula is as follows:

[0019] C f =[1-Lpseudo*logist(Lpseudo)]

[0020] Where Lpseudo is the probability of the predicted class;

[0021] Phase 1 training uses random numbers generated by the numpy module's random function and C++. f Compared to, higher than C f Then, image augmentation is performed, copying pixels from random regions of the labeled image to corresponding regions of the unlabeled image; Phase 2 training involves data augmentation on all unreliable unlabeled data; the data augmentation formula is as follows:

[0022] U = (1-U rand )*U+U rand *L

[0023] Where U refers to unlabeled data and L refers to labeled data. rand This refers to random regions in unlabeled data. Furthermore, the specific implementation process of the image feature layers for the student and teacher models is as follows:

[0024] F = Transformer_blcok(x)

[0025] The hierarchical encoder Transformer_block includes an Efficient Self-Attention (ESA) layer and a Mixed Feed-Forward Network (Mix-FFN) layer. The ESA employs multi-head attention while reducing sequence length through a sequence decay strategy. For a given sequence, it first uses the mapping parameter matrix W... Q W K W VConverted to Q, K, V, the ESA expression is as follows:

[0026]

[0027]

[0028] in, Here, n is the scaling factor, and n represents the number of long positions. Let d be the mapping parameter matrix. model d is the input dimension. k For the output dimension, here, each header's Q, K, and V have the same dimension N*C, where N = h*w represents the sequence length, h and w represent the sequence height and width, and C represents the number of channels; M ij This represents the mask at the corresponding position (i, j); taking K as an example, Reshape K to its dimension. The complexity of self-attention is reduced; the sequence decay expression is as follows:

[0029]

[0030] K = Linear(C·R,C)(K')

[0031] Where R is the reduction ratio, Reshape is the reshaping layer that changes size, and Linear is the linear layer that changes dimension.

[0032] Furthermore, the specific implementation process of the multi-scale feature fusion layer in the teacher-student model is as follows:

[0033] The low-level feature f1 and high-level feature f4 from the feature F = {f1, f2, f3, f4} obtained from the image feature extraction layer are enhanced with Outlook Attention (OA) and then fused with the original feature to obtain a new F.

[0034] feat=OA(Linear(f))+f

[0035] F = Cat(f, feat)

[0036] The OA (Optimization and Analysis) method uses different local windows to calculate the attention weights of features adjacent to the selected position. The different weights from the same position in different local windows are added together to obtain the output. The specific implementation process is as follows:

[0037] Y i,j =∑Y Δi,j

[0038] Y Δi,j =MatMul(softmax(Ai,j ),V Δi,j )

[0039] A i,j =X*W A V i,j =X*W V

[0040] Where Y Δi,j Represents a local window, mapping parameter matrix For learnable parameters, V represents the features of input feature X after being mapped by the mapping parameter matrix. Δi,j This represents the data within a window centered at each position (i, j);

[0041] MatMul represents multi-head attention, with n set to 4, and the expression is as follows:

[0042]

[0043] Z = F MatMul (X,z′).

[0044] Furthermore, the specific implementation process of the decoder is as follows:

[0045] Features from the image feature extraction layer are processed through a multilayer perceptron (MLP) layer to unify the channel dimensions and obtain a fused feature Fuse. This includes linear transformation of channel dimensions, upsampling layer to unsample feature dimensions, and cat layer to fuse features.

[0046] F i =Unsample(Linear(f i )){i=1,2,3,4}

[0047] Fuse = Linear(Cat(F) i )).

[0048] Furthermore, the linear prediction layer in the teacher-student model consists of convolutional layers. The input feature dimension of the linear layer is [bs, dim, h, w], and the output dimension is [bs, n_cls, h, w], where bs represents the number of images in the batch, dim represents the number of input feature channels, and n_cls is the number of predicted categories. The formula is expressed as follows:

[0049] P = Linear(Fuse)

[0050] Where h and w are the height and width of the input feature map, respectively, and cls is the number of predicted categories.

[0051] Furthermore, the specific implementation process of the image feature extractor in the prototype-assisted model is as follows:

[0052] Given input image data x, the enhanced version x_s, and the label Pseudo, x and x_s are passed to the feature extraction layer to obtain the unencoded raw features Query_feat = {f0, f1, f2, f3} and support features Support_feat = {f′0, f′1, f′2, f′3}. ′ 3}; For labeled data, the labels are the real labels; for unlabeled data, the labels are the pseudo labels predicted by the teacher network.

[0053] Query_feat=Resnet_50(x), Support_feat=Resnet_50(x_s).

[0054] Furthermore, the multi-scale feature fusion layer process in the prototype-assisted model is as follows:

[0055] The low-level features obtained by the image feature extractor and the high-level features are linearly interpolated to be of the same size but different dimensions. Then, they are concatenated at the channel level and downsampled to obtain the high- and low-dimensional fused features.

[0056] Fuse_feat=Down(Cat(f0+UP(f3)))

[0057] Feat = Down(Cat(f1+UP(f2)))

[0058] S_feat=Down(Cat(f′1+UP(f′2)))

[0059] Where Down represents downsampling, UP represents upsampling, and Fuse_feat, Feat, and S_feat all represent the fused features.

[0060] Furthermore, the specific implementation process of the prototype feature extractor in the prototype-assisted model is as follows:

[0061] The multi-class semantic segmentation pseudo-label Pseudo is converted into a binary mask M, where 1 represents the foreground predicted target and 0 represents the background. S_feat is abstracted into a foreground prototype vector through mask average pooling, as shown in the following expression:

[0062] pro = Avg_pool(X⊙M)

[0063] X=S_feat*M,M=Mask(Pseudo)

[0064] Where S_feat represents the fused query features, X represents the feature extraction of the target region using M pairs, retaining the features of the region containing the predicted target, and setting the features of the remaining regions to 0, and Avg_pool represents masked average pooling, expressed as follows:

[0065] F Avg_pool =Avg_pool2d(S_feat,S_feat) / Avg_pool1d(S_feat,M)

[0066] Where Avg_pool1d performs weighted global average pooling on M, with the pooling window being the height and width of S_feat. This is used to calculate the average value of the effective region in the mask, which is the proportion of the area of ​​the region with a value of 1 in the mask to the entire feature map. Then, the result of the average pooling is multiplied by the height h of the feature map. ′ With width w ′ This yields the area of ​​the effective region in the mask; where Avg_pool2d performs global average pooling on S_feat, with the pooling window being the height and width of S_feat, thus obtaining a single feature vector. The result of the average pooling is then multiplied by the h of the feature map. ′ With w ′ And divide by the area of ​​the effective region of the mask to obtain the feature vector that represents the features of the effective region in the mask, i.e., the foreground prototype Pro.

[0067] Furthermore, the specific implementation process of the decoder in the prototype-assisted model is as follows:

[0068] The prototype features Pro obtained from the prototype feature extractor are passed through a residual network (RN) layer, then undergo self-attention (SA) calculation, followed by a normalization layer (AddNorm, AN). Finally, cross-attention (CA) is calculated between Pro and the features Feat from the unlabeled image x obtained from the image feature extraction layer, and then passed through another AN layer. The final decoded features Proto are then obtained through a feedforward layer.

[0069] Proto=Transfomer_decoder(Pro,Feat)

[0070] Transformer_decoder=AN(CA(AddNorm(SA(RN(Pro))),Feat))

[0071] Where CA utilizes the mapping parameter matrix W Q W K Convert Pro to Q, Feat to K, and V respectively.

[0072]

[0073] Q = W Q Pro,K=W K Feat, V = W K Feat.

[0074] Furthermore, the output generation layer process in the prototype-assisted model is as follows:

[0075] The fused features Fuse_feat and Proto are subjected to Einstein Summation (ES), i.e., dot product. Fuse_feat has dimensions [bs, c, h, w], and Proto has dimensions [bs, c, 1]. The output is [bs, 1, h, w], which serves as the predicted foreground semantic segmentation map. The ES expression is as follows:

[0076]

[0077] Where i, k, and p all represent the corresponding dimensions.

[0078] Furthermore, the specific implementation process for filtering and weighting unreliable data is as follows:

[0079] τ = Quantile(Lcon, ratio)

[0080] reliable_indice=Compare(Lcon,τ)

[0081] Weight[reliable_indice]=high_Weight,Weight[unreliable_indice]=low_weight

[0082] Where ratio is the ratio value, τ is the threshold that meets the ratio value calculated using torch.quantile, and compare is to compare the sample loss value with the threshold. Samples with a loss value less than the threshold are identified as reliable samples, and samples with a loss value greater than the threshold are identified as unreliable samples. Their index is stored in reliable_index.

[0083] Furthermore, during the training process of the gradient backpropagation-corrected semi-supervised semantic segmentation generation model, three loss functions are included, with the supervised loss L... l Unsupervised loss L u Comparison of loss L con The expression is as follows:

[0084]

[0085] Y i=Mask(P i )

[0086] L represents a labeled dataset, U represents an unlabeled dataset, and PL... i The representation model's prediction of labeled data, PU i The representation model's prediction of unlabeled data, Y i Statement P i The mask obtained after mask transformation, Y represents the true label; the Mask transformation process is to take the category with the highest predicted probability among each category prediction. The difference is that the prototype model only needs foreground and background prediction. In the prediction result, 1 indicates the presence of an object and 0 indicates no object; S and P represent the predictions of the student model and the prototype model for unlabeled data, respectively.

[0087] The formulas for the loss function Loss1 for stage 1 and the loss function Loss2 for stage 2 are as follows:

[0088] Loss l =L l +L u +L con

[0089] Loss2=L l +L u . Attached Figure Description

[0090] Figure 1 This is a schematic diagram of the semi-supervised semantic segmentation method based on teacher-student networks and foreground prototypes of the present invention;

[0091] Figure 2 This is a schematic diagram illustrating feature extraction and multi-scale feature fusion based on the prospective attention mechanism in an example of the present invention;

[0092] Figure 3 This is a schematic diagram of a prototype-assisted model based on prototypes and attention in an example of the present invention. Detailed Implementation

[0093] The embodiments of the present invention will be further described below with reference to the accompanying drawings:

[0094] like Figure 1 As shown, a semi-supervised semantic segmentation method based on teacher-student networks and foreground prototypes includes three models: a teacher model, a student model, and a prototype-assisted model. The teacher and student models are identical, employing a Segformer network model with an added prospective attention mechanism, including an image feature extraction layer, a multi-scale feature fusion layer, a decoder, and a linear transformation layer. Figure 2As shown, the image feature extraction layer, composed of multiple Transformer blocks including ESA and Mix-FFN, extracts high-resolution coarse features and low-resolution fine-grained features. The multi-scale feature fusion layer, consisting of a prospective attention mechanism and a multi-scale feature fusion module, acquires richer and more detailed features. The decoder, composed of a multilayer perceptron, aggregates information from different levels. The linear transformation layer outputs the final prediction result. The prototype-aided model includes an image feature extraction layer, a prototype feature extraction layer, a multi-scale feature fusion layer, a decoder, and a segmentation generation layer. The prototype feature extraction layer, composed of pooling layers, generates foreground prototype features. The decoder, composed of Transformers, uses the foreground prototype as a query and the fused features as keys and values ​​to decode and obtain semantic features.

[0095] A semi-supervised semantic segmentation method based on teacher-student networks and foreground prototypes includes the following steps:

[0096] S1. Construct a semi-supervised semantic segmentation neural network, which includes a teacher model, a student model, and a prototype auxiliary model.

[0097] S2. Perform strong and weak enhancement and labeling preprocessing on the official semantic segmentation dataset Pascal VOCAugmented to obtain perturbed image data.

[0098] S3. Feature extraction is performed on the unlabeled dataset S2. The feature extraction model pre-trained by the teacher model is used to extract low-level, mid-level, and high-level features of the image. The low-level and high-level features are fed into the multi-scale feature fusion layer for prospective attention enhancement, and then fused and encoded with the mid-level features to obtain richer image semantic features. The features are then fed into the decoder to obtain the prediction results, thereby obtaining pseudo-labels for the unlabeled image data.

[0099] S4. For the unlabeled data and the pseudo-labels obtained in S3, data augmentation is performed using labeled data to obtain more robust unlabeled data and its pseudo-labels.

[0100] S5. For the dataset in S2 and the pseudo-labels obtained in S4, the pre-trained feature extraction model in the prototype-assisted model is used to extract low-level, mid-level, and high-level features of the image respectively. The low-level and high-level features are fed into the multi-scale fusion layer to obtain f', the mid-level features are fed into the prototype extractor to obtain the prototype vector Pro, and Pro is fed into the decoder to obtain the decoded features. The features are then processed with f' through the segmentation generation layer to obtain the final result. Consistency regularization is performed with the masked real labels and the pseudo-labels obtained in S4 to update the prototype-assisted model.

[0101] S6. For the unlabeled data in the dataset of S2, repeat the prediction process in S5 to obtain the foreground pseudo-labels.

[0102] S7. For the dataset in S2, the student model repeats S3 to obtain the prediction results of the data. It is compared with the prediction results of S6, and reliable unlabeled data is selected and weighted for loss calculation. Then, consistency regularization is performed with the true labels and the prediction results of S4 to update the student model. Finally, the exponential moving average of the student model parameters is used to update T to complete the training of stage 1.

[0103] S8. After data augmentation of the unreliable data obtained in S7, repeat the training steps of S3, S4, and S7 without the participation of the prototype auxiliary model, thus completing the training of stage 2.

[0104] S9. In the inference stage, select the student model and perform semantic segmentation prediction of image data.

[0105] Furthermore, S3 uses the layered encoder Transformer_block of the mit_b2 model in the pre-trained Segformer model, including ESA and Mix-FFN, to extract the features F = {f1, f2, f3, f4} of each image in the PascalAug dataset, where f1 is the low-level feature, f2 and f3 are the mid-level features, and f4 is the high-level feature.

[0106] F = Transformer_blcok(x)

[0107] Transformer_block=[ESA(x)+MixFNN(x)]

[0108]

[0109] K = Linear(C·R,C)(K')

[0110] The number of multi-head units in the four Transformer_blocks are [1, 2, 5, 8], and the decay ratio is [8, 4, 2, 1].

[0111] Subsequently, the multi-scale fusion module in S3 consists of an OA and channel connections, in order to obtain new F1 and F4 with rich information:

[0112] feat 1,4 =OA(Linear(f) 1,4 ))+f 1,4

[0113] F 1,4 =Cat(f 1,4 feat 1,4)

[0114] The OA (Optimization and Analysis) method uses different local windows to calculate the attention weights of features adjacent to the selected position. The different weights from the same position in different local windows are added together to obtain the output. The specific implementation process is as follows:

[0115]

[0116] Y Δi,j =MatMul(softmax(A i,j ),V Δi,j )

[0117]

[0118] A i,j =X*W A V i,j =X*W V

[0119]

[0120] Z = F MHCA (X,z′)

[0121] In MatMul multi-head attention, the number of multi-heads is set to 4. Represents the mapping parameter matrix, V represents the features of input feature X after being mapped by the mapping parameter matrix. Δi,j This represents the data within a window centered at each position (i, j).

[0122] Furthermore, the data augmentation in S2 employs weak augmentation techniques such as flipping and cropping, which are common in this field, and strong augmentation techniques such as color dithering and contrast adjustment. The data augmentation in S4 utilizes CutMix to augment unlabeled image data with labeled image data. The training in stage 1 uses random numbers generated by the NumPy module and the confidence level C of the pseudo-labels. f Compared to, higher than C f Then, image augmentation is performed. The training in Phase 2 involves data augmentation on all unreliable unlabeled data. The specific implementation process is as follows:

[0123] C f =[1-Lpseudo*logist(Lpseudo)]

[0124] U = (1-U rand )*U+U rand *L.

[0125] Furthermore, the prototype feature extractor in S5 employs global average pooling (Avg_pool) to obtain the prototype vector Pro representing the features of the effective region in the representation mask, as shown in the following formula:

[0126] pro=Avg_pool(X⊙M)*h*w / area

[0127] F Avg_pool =Avg_pool2d(S_feat,S_feat) / Avg_pool1d(S_feat,M)

[0128] X=S_feat*M, S_feat=Cat(f′1,f′2), M=Mask(Pseudo)={1, 1, 0, 0...}

[0129] Where M is the binary mask of the multi-class semantic segmentation pseudo-label Pseudo, 1 represents the foreground predicted target, 0 represents the background, h and w represent the height and width of the feature map, area is the area of ​​the effective region of the mask, S_feat is the query feature of the fused mid-level features, X is the feature of the region containing the predicted target, Avg_pool1d is the weighted global average pooling of M, and Avg_pool2d is the global average pooling of S_feat.

[0130] Subsequently, in S5, the decoder performs CA on the prototype feature Pro after self-attention calculation and the feature Feat of the unlabeled image x obtained by the image feature extraction layer. Then, it obtains the decoded feature Proto through the segmentation generation layer. The fused feature Fuse_feat and Proto are then subjected to ES dot product to obtain the final prediction result, expressed as:

[0131] Proto=Transfomer_decoder(Pro,Feat)

[0132] Y = ES(Fuse_feat, Proto)

[0133] Transformer_decoder=AN(CA(AddNorm(SA(RN(Pro))),Feat))

[0134]

[0135] Q = W Q Pro,K=W K Feat, V = W K Feat.

[0136] Furthermore, in S5 and S7, the training process of gradient backpropagation to correct the weights of the semi-supervised semantic segmentation generation model includes three loss functions, including the supervised loss L. l Unsupervised loss L u Comparison of loss L con The expression is as follows:

[0137]

[0138]

[0139] Y i =Mask(P i )

[0140] L represents a labeled dataset, U represents an unlabeled dataset, and PL i The representative model's predictions on labeled data, PU i The representative model's prediction for unlabeled data, Y i Y represents the pseudo-label obtained after masking transformation, and Y represents the real label. The Mask transformation process is to take the category with the highest predicted probability among all categories. The difference is that the prediction result of the prototype model is 1, which means that there is an object at the location, and 0 means that there is no object. S and P represent the predictions of the student model and the prototype model for the unlabeled data, respectively.

[0141] This invention provides a semi-supervised semantic segmentation method based on a teacher-student network and foreground prototypes, comprising three models: a teacher model, a student model, and a prototype-assisted model. The teacher and student models are identical, employing a Segformer network model with an incorporating prospective attention mechanism. This model includes an image feature extraction layer, a multi-scale feature fusion layer, a decoder, and a linear transformation layer. The image feature extraction layer, composed of multiple hierarchical encoder transformer blocks, includes efficient self-attention layers and hybrid connection layers, extracting high-resolution coarse features and low-resolution fine-grained features. The multi-scale feature fusion layer, consisting of a prospective attention mechanism and a multi-scale feature fusion module, acquires richer and more detailed features. The decoder, composed of a multilayer perceptron, aggregates information from different levels. The linear transformation layer outputs the final prediction result. The prototype-assisted model includes an image feature extraction layer, a prototype feature extraction layer, a multi-scale feature fusion layer, a decoder, and a segmentation generation layer. The prototype feature extraction layer, composed of pooling layers, generates foreground prototype features. The decoder, composed of Transformers, uses the foreground prototype as a query and the fused features as keys and values ​​to decode and obtain semantic features. The teacher model and prototype model generate pseudo-labels for the unlabeled data to semi-supervise the prediction results of the student model; the prototype model is trained using DiceLoss loss, which includes supervised loss and unsupervised loss; the student model is trained using cross-entropy loss, which includes supervised loss, unsupervised loss and prototype contrast loss; the teacher model is trained using the exponential moving average of the student model; after training, the test data is used to obtain the predicted semantic segmentation map through the student network.

[0142] Example:

[0143] A semi-supervised semantic segmentation method based on teacher networks and foreground prototypes, with training divided into two stages, includes the following steps:

[0144] Step 1: Perform strong and weak enhancement and labeling preprocessing on the official semantic segmentation dataset Pascal VOCAugmented to obtain perturbed image data;

[0145] Step 2: Input the unlabeled image data into the teacher model to obtain pseudo-labels;

[0146] Step 3: Input the data into the prototype-assisted model. After prototype feature extraction, input the data into the decoder to obtain the image prediction results. Then, use Dice loss to calculate the loss between the prediction results of labeled data and the real labels, and between the prediction results of unlabeled data and the pseudo labels, and update the model.

[0147] Step 4: Input the unlabeled data into the prototype auxiliary model to obtain foreground pseudo-labels;

[0148] Step 5: Input the data into the student model, obtain the image prediction results, calculate the contrast loss between the Dice loss and the foreground pseudo-labels from Step 4, filter out unreliable data, and assign weights to the data to calculate the unsupervised loss; calculate the loss between the prediction results of labeled data and the real labels, and between the prediction results of unlabeled data and the pseudo-labels using cross-entropy loss, update the student model in combination with the contrast loss, and finally update the teacher model using the exponential moving average of the parameters of the student model, thus completing the training of Stage 1.

[0149] Step 6: For the unreliable data obtained in Step 5, perform data augmentation using labeled data, and repeat Step 2 and Step 5, excluding the prototype prediction comparison process in Step 5, to complete the training of Stage 2.

[0150] Furthermore, the weak enhancement adopted in step 1 is the flipping and cropping techniques commonly used in this field, while the strong enhancement is color dithering and contrast adjustment.

[0151] Furthermore, in step 3, the training hyperparameters are set to 64 for the prototype feature extraction channels; for other training hyperparameters, the optimizer is Adam and the learning rate is set to 1e-3.

[0152] Furthermore, in the models of steps 2 and 5, the Transformer_block feature extraction channels are set to [64, 128, 320, 512], the number of multi-heads is [1, 2, 5, 8], and the decay ratio is [8, 4, 2, 1]. Other training hyperparameters are: batch size is set to 4, the optimizer is SGD, the learning rate is set to 0.0005, the proportion of reliable data is 0.7, the weight of reliable data when calculating the loss is set to 0.7, and the weight of unreliable data is set to 0.3.

[0153] Furthermore, the data augmentation operation in step 6 utilizes CutMix to augment unlabeled image data with labeled image data. The training in stage 1 employs random numbers generated by the numpy module and the confidence level C of the pseudo-labels. f Compared to, higher than C f Then, image augmentation is performed. Phase 2 training involves data augmentation on all unreliable unlabeled data, using the following formula:

[0154] C f =[1-Lpseudo*logist(Lpseudo)]

[0155] U = (1-U rand )*U+U rand *L

[0156] Where Lpseudo represents the probability of the predicted class, U refers to unlabeled data, and L refers to labeled data. rand This refers to random regions in unlabeled data.

[0157] Furthermore, the Stage 1 loss function Loss l The stage 2 loss function Loss2 is:

[0158] Loss l =L l +L u +L con

[0159] Loss2=L l +L u

[0160]

[0161]

[0162] Y i =Mask(P i )

[0163] Where L represents a labeled dataset, U represents an unlabeled dataset, and PL i PU represents the model's predictions on labeled data. i Y represents the model's prediction of labeled data, where Y represents the true label. i represents pseudo-labels, Mask represents category mask transformation, and S and P represent the predictions of the student model and the prototype model for unlabeled data, respectively.

[0164] To demonstrate the effectiveness of this invention, comparative experiments were conducted with current mainstream semi-supervised semantic segmentation models and methods. Ablation experiments were also performed on the proposed module. The evaluation metrics adopted were mainstream descriptive metrics: Intersection over Union (IoU) is the ratio of the area of ​​intersection between the predicted result and the ground truth label to the area of ​​their union, used to measure the model's pixel classification accuracy for each category and the spatial matching degree of the segmentation results; Mean Intersection over Union (MIoU) is the average IoU value for each category, comprehensively considering the segmentation accuracy of all categories, and is commonly used to measure the performance of semantic segmentation models, as shown in the expression:

[0165]

[0166] The experimental results are shown in Table 1:

[0167] Table 1 compares the experimental results of our method and the contrasting model on labeled data of different proportions in the Pascal VOC Augmented dataset.

[0168]

[0169] Among them U 2 PL represents a representative data setting in this field, and * indicates the adoption of U. 2 PL, the rest use CPS settings, B2 refers to the Mit_B2 model in the Segformer series, B5 refers to the Mit_B5 model in the Segformer series, the comparison model is the existing method, using the DeepLabV3+ model in CNN (the part before the slash indicates that ResNet50 is used as the encoder, the part after the slash indicates that ResNet101 is used as the encoder), no slash indicates the result of using ResNet101 encoder. Ours method uses the Segformer model, 1 / 16, 1 / 8, 1 / 4 represent the proportion of labeled data. In order to make a fair comparison, all models are trained with the same proportion of labeled data. According to the experimental results, our method and the comparison model both show a high level, and achieve the best results in most metrics.

[0170] Table 2 shows the ablation experimental results of this method on the Pascal VOC Aug dataset.

[0171]

[0172] Where O indicates the absence of the design, and W indicates its presence. Ablation experiments provide a clear analysis showing that data augmentation effectively utilizes limited labeled data to enhance unlabeled information, improving the model's generalization ability. Prospective attention for local feature enhancement effectively improves the semantic segmentation performance of the baseline model. Prospective attention employs a sliding window operation to further extract local details and avoid feature omission. A prototype-assisted model is used, utilizing the foreground prototype to obtain a foreground segmentation map. This helps filter unreliable data for secondary augmentation, further guiding the training of student networks.

Claims

1. A semi-supervised semantic segmentation method based on teacher-student networks and foreground prototypes, characterized in that, include: Step 1: Construct a semi-supervised semantic segmentation network, which includes a teacher network, a student network, and a prototype auxiliary network; The teacher and student models are the same, using the SegFormer network model, which includes an image feature extraction layer, a multi-scale feature fusion layer, a decoder, and a linear transformation layer; The multi-scale feature fusion layer incorporates a prospective attention mechanism; the prototype-assisted model includes an image feature extraction layer, a prototype feature extraction layer, a multi-scale feature fusion layer, a decoder, and a segmentation generation layer. Step 2: Construct a semi-supervised semantic segmentation dataset. After preprocessing the data with and without labels and with strong and weak enhancements, divide it into training set, validation set and test set; assign pre-training parameters to mit_b2 of the image feature extraction layer. Step 3: Input the unlabeled images of the training set into the feature extraction layer of the teacher model to obtain the original features containing high-resolution coarse image features and low-resolution image features. Input the original image features into the multi-scale feature fusion module, use the prospect attention mechanism to obtain the detailed local features of the image, fuse the original features, and then pass through the feedforward network layer to obtain the encoded image features. The obtained image features are fed into the decoder to obtain the decoded features, and then passed through the linear transformation layer to obtain the prediction result, which serves as the pseudo label for the unlabeled image. Step 4: Calculate the confidence of the pseudo-labels obtained in Step 3; adopt the random hybrid enhancement method to randomly add the labeled image information to the unlabeled data in the region below the threshold, so as to obtain the strongly enhanced unlabeled image and its pseudo-labels. Step 5: Input the labeled images, unlabeled images, and the strongly enhanced unlabeled images and their pseudo-labels obtained in Step 4 into the image feature extraction layer of the prototype model to obtain the original features containing high-resolution coarse image features and low-resolution image features. Input the original image features into the prototype feature extraction layer, and use dot product and global pooling to obtain the foreground prototype of the image. Input the foreground prototype and the original features into the decoder to obtain the decoded features. Finally, fuse them with the original features and send them into the output generation layer to obtain the final prediction result. The prediction result includes labeled image prediction and unlabeled image prediction. The loss of labeled image prediction is calculated with the real label. The loss of unlabeled image prediction is calculated with the pseudo-label of the teacher model obtained in Step 3, and the gradient is backpropagated to correct the weights of the prototype model. Step 6: Input labeled and unlabeled images from the training set into the prototype model, repeat step 5, and obtain the prediction results of the prototype model for the images, which will be used as the foreground pseudo-labels for the unlabeled images. Step 7: Input labeled and unlabeled images from the training set into the student model, repeat step 3, and obtain the student model's prediction results for the images. The prediction results include the prediction of labeled images and the prediction of unlabeled images. Calculate the loss between the labeled image prediction and the real label, and calculate the comparison loss between the unlabeled image and the foreground pseudo-label of the prototype model obtained in step 6. Use this to filter reliable and unreliable unlabeled data, assign weights, and then calculate the loss with the pseudo-label of the teacher model obtained in step 3. Use gradient backpropagation to correct the weights of the student model, and finally update the teacher model using exponential moving average to complete the training of stage 1. Step 8: Perform data augmentation on the unreliable data obtained in Step 7, and repeat the training in Step 7, without comparing it with the prototype model, until the specific training is completed, thus completing the training of Phase 2. Step 9: After the semi-supervised semantic segmentation model is trained, it enters the inference stage. At this time, the semi-supervised semantic segmentation model no longer changes the weights. The image to be predicted is fed into the trained teacher model, and Step 3 is repeated to obtain the final semantic segmentation result.

2. The semi-supervised semantic segmentation method based on teacher-student networks and pre-prototypes according to claim 1, characterized in that, The semantic segmentation dataset used as the training dataset is divided proportionally into labeled and unlabeled image data. Weak augmentation techniques, such as flipping and cropping, are common in this field, while strong augmentation techniques include color dithering and contrast adjustment. During the training phase, data augmentation operations utilize CutMix to enhance the information in unlabeled image data from labeled image data. The specific implementation process is as follows: Calculate the confidence level C for pseudo-labels without labeled data within a batch. f The formula is as follows: C f =[1-Lpseudo*logist(Lpseudo)] Where Lpseudo is the probability of the predicted class; Phase 1 training uses random numbers generated by the numpy module's random function and C++. f Compared to, higher than C f Then, image augmentation is performed, copying pixels from random regions of the labeled image to corresponding regions of the unlabeled image; Phase 2 training involves data augmentation on all unreliable unlabeled data; the data augmentation formula is as follows: U=(1-U rand )*U+U rand *L Where U refers to unlabeled data and L refers to labeled data. rand This refers to random regions in unlabeled data.

3. The semi-supervised semantic segmentation method based on teacher-student networks and foreground prototypes according to claim 1, characterized in that, The specific implementation process of the image feature layer is as follows: The hierarchical Transformer_block encoder of the mit_b2 model from the Segformer model series is used to extract the features F = {f1, f2, f3, f4} for each image in the Pascal dataset, where f1 is the low-level feature, f2 and f3 are the mid-level features, and f4 is the high-level feature. The Transformer_block expression is as follows: F = Transformer_block(x) The Transformer_block includes an Efficient Self-Attention (ESA) layer and a Mixed-Feed-Forward Network (Mix-FFN) fully connected layer. In ESA, multi-head attention is employed, and a sequence decay strategy is used to reduce the sequence length. For a given sequence Z, it is first transformed into Q, K, and V using a mapping parameter matrix. The ESA expression is as follows: in, Here, n is the scaling factor, and n represents the number of long positions. Let d be the mapping parameter matrix. model d is the input dimension. k For the output dimension, here, each header's Q, K, and V have the same dimension N*C, where N = h*w represents the sequence length, h and w represent the sequence height and width, and C represents the number of channels; M ij The mask representing the position (i, j); The sequence decay expression is as follows: K = Linear(C·R,C)(K') Where R is the reduction ratio, Reshape is the reshaping layer that transforms the size, and Linear is the linear layer that transforms the dimension; taking K as an example, reshape K into a dimension through reshape. This reduces the complexity of self-attention.

4. The semi-supervised semantic segmentation method based on teacher-student networks and foreground prototypes according to claim 1, characterized in that, The specific implementation process of the multi-scale feature fusion layer in the teacher-student model is as follows: The low-level feature f1 and high-level feature f4 in the feature F = {f1, f2, f3, f4} obtained from the image feature extraction layer are enhanced with Outlook Attention (OA) respectively, and then fused with the original f1 and f4 to obtain new F1 and F4: feat=OA(Linear(f))+f F = Cat(f, feat) The OA (Optimization and Analysis) method uses different local windows to calculate the attention weights of features adjacent to the selected position. The different weights from the same position in different local windows are added together to obtain the output. The specific implementation process is as follows: AND i,j =∑Y Δi,j AND Δi,j =MatMul(softmax(A i,j ),V Δi,j ) A i,j =X*W A ,V i,j =X*W V Where Y Δi,j Represents a local window, mapping parameter matrix For learnable parameters, V represents the features of input feature X after being mapped by the mapping parameter matrix. Δi,j This represents the data within a window centered at each position (i, j); MatMul represents multi-head attention, and its expression is as follows: Where n is set to 4, the final result is the sum of the results of 4 multiple heads; M ij This represents the mask at the corresponding position.

5. A semi-supervised semantic segmentation method based on teacher-student networks and foreground prototypes according to claim 1, characterized in that, The specific implementation process of the decoder is as follows: Features from the image feature extraction layer are processed through a multilayer perceptron (MLP) layer to unify the channel dimensions and obtain a fused feature Fuse. This includes a linear layer to change the channel dimensions, an upsampling layer to change the feature dimensions, and a cat layer to fuse the features. F i =Unsample(Linear(f i )){i=1,2,3,4} Fuse=Linear(Cat(F i ))。 6. The semi-supervised semantic segmentation method based on teacher-student networks and foreground prototypes according to claim 1, characterized in that, The linear prediction layer in the teacher and student model consists of convolutional layers. The input feature dimension of the linear layer is [bs, dim, h, w], and the output dimension is [bs, n_cls, h, w]. Here, bs represents the number of images in the batch, dim represents the number of input feature channels, n_cls is the number of predicted categories, and h and w represent the height and width of the feature map, respectively. P = Linear(Fuse) Where P represents the prediction result.

7. A semi-supervised semantic segmentation method based on teacher-student networks and foreground prototypes according to claim 1, characterized in that, The specific implementation process of the image feature extractor in the prototype-assisted model is as follows: Given input image data x, the enhanced version x_s, and label Pseudo, x and x_s are passed to the feature extraction layer to obtain uncoded original features Query_feat = {f0, f1, f2, f3} and support features Support_feat = {f′0, f′1, f′2, f′3}. For labeled data, the labels are the real labels; for unlabeled data, the labels are the pseudo labels predicted by the teacher network. Query_feat=Resnet_50(x),Support_feat=Resnet_50(x_s) ResNet-50 is a commonly used model for semantic segmentation.

8. A semi-supervised semantic segmentation method based on teacher-student networks and foreground prototypes according to claim 1, characterized in that, The multi-scale feature fusion layer in the prototype-assisted model uses linear interpolation to adjust low-level and high-level features into features of the same size but different dimensions. These features are then concatenated at the channel level and downsampled to obtain the high- and low-dimensional fused features, as shown in the following expression: Fuse_feat=Down(Cat(f0+UP(f3))) Feat = Down(Cat(f1+UP(f2))) S_feat=Down(Cat(f′1+UP(f′2))) Where Down represents downsampling, UP represents upsampling, and Fuse_feat, Feat, and S_feat all represent the fused features.

9. A semi-supervised semantic segmentation method based on teacher-student networks and foreground prototypes according to claim 1, characterized in that, The specific implementation process of the prototype feature extractor in the prototype-assisted model is as follows: The multi-class semantic segmentation pseudo-label Pseudo is transformed into a binary mask M, where 1 represents the foreground predicted target and 0 represents the background; S_feat is abstracted into a foreground prototype vector Pro through mask average pooling, as shown in the following expression: Pro = Avg_pool(X⊙M) X=S_feat*M,M=Mask(Pseudo) Where X represents the target region feature extraction using M on S_feat, retaining the features of the region containing the predicted target and setting the features of the remaining regions to 0, and Avg_pool is masked average pooling, expressed as follows: F Avg_pool =Avg_pool2d(S_feat,S_feat) / Avg_pool1d(S_feat,M) Avg_pool1d performs weighted global average pooling on M, with the pooling window being the height and width of S_feat. This is used to calculate the average value of the effective region in the mask, which is the proportion of the area of ​​the region with a value of 1 in the mask to the entire feature map. Then, the result of the average pooling is multiplied by the height h′ and width w′ of the feature map to obtain the area of ​​the effective region in the mask. Avg_pool2d performs global average pooling on S_feat, with the pooling window being the height and width of S_feat. This is used to obtain a single feature vector. Then, the result of the average pooling is multiplied by the h′ and w′ of the feature map and divided by the area of ​​the effective region in the mask to obtain the feature vector representing the features of the effective region in the mask, i.e., the foreground prototype Pro.

10. A semi-supervised semantic segmentation method based on teacher-student networks and foreground prototypes according to claim 1, characterized in that, The specific implementation process of the decoder in the prototype-assisted model is as follows: The Pro obtained from the prototype feature extractor is passed through a residual network (RN) layer, then self-attention (SA) is calculated, then through a layer normalization layer (AddNorm (AN), and then cross-attention (CA) is calculated with the features of the unlabeled image x obtained from the image feature extraction layer. It is passed through AN again, and finally through a feedforward layer to obtain the final decoded feature Proto. Proto=Transfomer_decoder(Pro,Feat) Transformer_decoder=AN(CA(AddNorm(SA(RN(Pro))),Feat)) The specific process of CA involves using the mapping parameter matrix W. Q W K Convert Pro to Q, and Feature to K and V respectively: Q=W Q Pro,K=W K Feat,V=W V Feat。 11. A semi-supervised semantic segmentation method based on teacher-student networks and foreground prototypes according to claim 1, characterized in that, In the prototype-assisted model, the output generation layer takes the fused features Fuse_feat as input and performs Einstein Summatio (ES) with Proto. The dimensions of Fuse_feat are [bs, c, h, w], and the dimensions of Proto are [bs, c, 1]. The output is [bs, 1, h, w], which serves as the predicted foreground semantic segmentation map. Y = ES(Fuse_feat, Proto) Where i, k, and p all represent the corresponding dimensions.

12. The semi-supervised semantic segmentation method based on teacher-student networks and foreground prototypes according to claim 1, characterized in that, The specific implementation process for filtering and weighting unreliable data is as follows: τ = Quantile(Lcon, ratio) reliable_indice=Compare(Lcon,τ) Weight[reliable_indice]=high_weight,Weight[unreliable_indice]=low_weight Where ratio is the ratio value, τ is the threshold that meets the ratio value calculated using torch.quantile, compare is to compare the sample loss value with the threshold, samples with a loss value less than the threshold are considered reliable samples, and samples with a loss value greater than the threshold are considered unreliable samples, and their index is stored in reliable_indice, high_weight is the weight assigned when calculating the loss of reliable data, and low_weight is the weight assigned when calculating the loss of unreliable data.

13. A semi-supervised semantic segmentation method based on teacher-student networks and foreground prototypes according to claim 1, characterized in that, The specific implementation process of the loss function Loss1 for Phase 1 and the loss function Loss2 for Phase 2 is as follows: Loss l =L l +L u +L ocn Loss2=L l +L u Y i =Mask(P i ) Among them, L l To monitor the loss, L u For unsupervised loss, L con For comparison of losses, L represents the labeled dataset, U represents the unlabeled dataset, and PL represents the loss. i For the model to predict labeled data, PU i For the model's prediction of unlabeled data, Y i For P i The mask obtained after mask transformation, Y is the true label; the Mask transformation process is to take the category corresponding to the highest predicted probability among each category prediction. The difference is that the prototype model only needs foreground and background prediction, and only needs to convert the pixel positions that exist in the prediction to 1 and the non-existent ones to 0; S and P represent the predictions of the student model and the prototype model for unlabeled data, respectively.

Images