Differential double branch strongly enhanced perturbation semi-supervised image semantic segmentation training method

Abstract

This invention provides a semi-supervised image semantic segmentation training method with differential dual-branch strong enhancement perturbation, comprising: performing weak enhancement perturbation processing on labeled images and weak enhancement perturbation processing and two-way random strong enhancement perturbation processing on unlabeled images in the same network; training on the perturbation-processed images to obtain perturbation predictions for labeled images, weak enhancement perturbation predictions for unlabeled images, strong enhancement perturbations for unlabeled images, and feature perturbation predictions for weak enhancement perturbation images of unlabeled images; establishing a loss function to supervise the perturbation predictions obtained from labeled images, and using the predictions of weak enhancement perturbation images to simultaneously supervise the predictions of feature perturbation images and the predictions of the two-way strong enhancement perturbation images.

Description

Technical Field

[0001] This invention relates to a semantic segmentation method, and more particularly to a differential bi-branch strongly enhanced perturbation semi-supervised image semantic segmentation training method. Background Technology

[0002] Semantic segmentation is a key research area in computer vision. Traditional methods require a large number of manually annotated pixel-level labels. To address this challenge, semi-supervised semantic segmentation enhances model performance by leveraging both labeled and unlabeled data while reducing reliance on manual annotation.

[0003] Existing semi-supervised semantic segmentation methods are mainly divided into pseudo-labeling and consistency regularization methods. Pseudo-labeling methods are further subdivided into self-training and mutual training methods. In contrast, consistency regularization methods have attracted considerable attention in recent years, but they are limited by insufficient perturbation techniques, resulting in poor model training performance. Furthermore, most existing semi-supervised semantic segmentation frameworks employ a teacher-student network architecture, which injects network perturbations during training. This positively impacts consistency regularization, but excessive computation of network weights increases computation time. Summary of the Invention

[0004] This invention provides a differential bi-branch strongly enhanced perturbation semi-supervised image semantic segmentation training method, comprising:

[0005] Step S100: In the same network, weak enhancement perturbation processing is performed on the labeled image, and weak enhancement perturbation processing and two-way random strong enhancement perturbation processing are performed on the unlabeled image;

[0006] Step S200: Train the perturbation-processed image to obtain perturbation prediction for labeled image, weak enhancement perturbation prediction for unlabeled image, strong enhancement perturbation for unlabeled image, and feature perturbation prediction for weak enhancement perturbation image of unlabeled image;

[0007] Step S300: Establish a loss function to supervise the perturbation prediction obtained from the labeled image, and use the prediction of the weakly enhanced perturbation image to simultaneously supervise the prediction of the feature perturbation image and the prediction of the two strongly enhanced perturbation images.

[0008] Further, in step S100, the images of the labeled dataset samples are subjected to weak enhancement perturbation to obtain a first weakly enhanced image; the images of the unlabeled dataset samples are subjected to weak enhancement perturbation to obtain a second weakly enhanced image, and a copy of the second weakly enhanced image is subjected to two random strong enhancement perturbation branches to obtain a first strongly enhanced image and a first strongly enhanced image.

[0009] Furthermore, in step S100, each random strong enhancement perturbation process randomly selects three processing methods from {standard, automatic contrast, equalization, inversion, Gaussian blur, contrast, sharpness, color, brightness, hue, tone separation, overexposure}.

[0010] Furthermore, step S200 specifically includes the following processes:

[0011] Step S201: Divide the image obtained in step S100 into multiple image blocks and extract features, encode each image block into a vector, and concatenate the vectors of all image blocks.

[0012] Step S202: The concatenated image is processed through multiple layers of encoding modules with different resolutions to obtain a multi-scale feature image;

[0013] Step S203: Perform feature perturbation processing on the unlabeled weakly enhanced perturbation image to obtain a feature perturbation image;

[0014] Step S204: Capture global contextual information of each image through a self-attention mechanism;

[0015] Step S205: Multi-scale features are obtained through a multi-layer decoding module and then aggregated to obtain the final feature image;

[0016] Step S206: Convolve and upsample the final feature image to the original image size to obtain the perturbation prediction of the labeled image, the weak enhancement perturbation prediction of the unlabeled image, the strong enhancement perturbation of the unlabeled image, and the feature perturbation prediction of the weak enhancement perturbation image of the unlabeled image.

[0017] Furthermore, in step S202, each layer of the coding module has the same structure, with the resolution decreasing sequentially; each layer of the coding module includes an efficient attention submodule, a Mix-FFN submodule, and an Overlap Patch Merging submodule.

[0018] Furthermore, in step S205, each layer decoding module obtains the feature image of the corresponding layer encoding module.

[0019] Furthermore, the loss function L in step S300 UniMatch for

[0020]

[0021] Wherein, the supervised loss function L sup It is the cross-entropy loss between labeled data and real data, and the unsupervised loss L. u It is used to evaluate the prediction consistency between weakly enhanced image predictions and strongly enhanced image predictions.

[0022] Furthermore, the supervised loss function Lsup for

[0023]

[0024] in, Indicates the batch size of the labeled data. This represents a prediction of the labeled data. It is cross-entropy loss, y i This is real label data.

[0025] Furthermore, the unsupervised loss function L u for

[0026]

[0027] Where, p W It is a prediction of weak enhancement perturbations in unlabeled images, p s1 and p s2 It is a strong enhancement perturbation of the unlabeled image, p fp It is a feature perturbation prediction of an unlabeled image with weak enhancement perturbation. This is the batch size of the unlabeled data. It is the confidence threshold for filtering noise labels, where H represents minimizing two probability distributions. and This is a hyperparameter.

[0028] Compared with the prior art, the present invention has the following advantages: (1) It designs a random strong enhancement perturbation technique to make the predictions of the two enhanced data streams different, better conforming to the consistency regularization assumption and enhancing the model's perception ability; (2) It performs the operation in the same network, which, compared with perturbation processing through the teacher-student network, does not require updating the network weights and enhances the computational efficiency.

[0029] The present invention will now be further described with reference to the accompanying drawings. Attached Figure Description

[0030] Figure 1 This is a schematic diagram of the method flow of the present invention.

[0031] Figure 2 A visual illustration of a random, strongly enhanced perturbation applied to the input image.

[0032] Figure 3 A diagram illustrating the comparison in grouping cases on the original Pascal dataset.

[0033] Figure 4 This is a schematic diagram comparing the random strong augmentation method and the fixed strong augmentation method. Detailed Implementation

[0034] This embodiment is implemented using the UniFormer structure. The UniFormer structure includes a data perturbation processing network and a SegFormer network.

[0035] The data perturbation processing network includes a weakly enhanced perturbation processing sub-sub ... W ) and the strongly enhanced perturbation processing submodule (A SR Weak enhancement perturbation involves minor transformations of the data, such as scaling or translation, without altering the data's fundamental characteristics. Strong enhancement perturbation involves significant transformations of the data, such as rotation, shearing, adding noise, or adjusting color attributes, which typically change the data's fundamental characteristics. For N... l A dataset sample with 1 label The image is subjected to weak enhancement perturbation to obtain a weakly enhanced image X. i For N u Unlabeled dataset samples The image is subjected to weak enhancement perturbation to obtain a weakly enhanced image X. W Weakly enhanced image X W The copy is then subjected to two random, strongly enhanced perturbation branches to obtain the enhanced image X. s1 and X s2 The methods for strong enhancement perturbations are shown in Table 1, from which k methods are randomly selected to enhance the input image. The processing methods for two random strong enhancement perturbations are not exactly the same, ensuring the diversity of the images after strong enhancement perturbation and ensuring that two different predictions are presented. However, excessively distorted enhancements can harm the data distribution and reduce the performance of the semi-supervised semantic segmentation model. Therefore, it is necessary to select the appropriate number of perturbation methods. Experiments show that the best results are achieved when k=3.

[0036] Table 1

[0037]

[0038] Figure 2 The visualizations show 12 strongly enhanced perturbations and two example visualizations X after randomly applying 3 of these perturbations. s1 and X s2 Two different output results X s1 and X s2 The effectiveness of random augmentation perturbations was demonstrated, showing that they effectively utilize data diversity, adapt to the nuances of semi-supervised semantic segmentation tasks, and achieve improved performance metrics. This approach not only enhances the model's ability to handle variations in input data but also adheres to the principle of consistent regularization, thus achieving better generalization and performance in semi-supervised semantic segmentation tasks.

[0039] The SegFormer network consists of an embedding module, an encoding module, a self-attention module, a decoding module, and a self-supervised module.

[0040] The embedding module uses the Overlap Patch Merging image encoding method to enhance the image X. i X W X s1 X s2 The image is divided into multiple overlapping patches, each encoded as a vector representation and concatenated. Feature extraction is performed on each patch to obtain a high-dimensional vector representation. There is typically some overlap between the divided patches to ensure that important image information is not lost. Each part of the image can be represented and processed independently, providing more local information and contextual relationships. By using overlapping patches, Overlap Patch Embeddings can capture more detailed information in the image. When using Overlap Patch Embeddings, the vector representations of each patch can be concatenated to form the encoded representation of the entire image.

[0041] The encoding module includes a four-layer Transformer structure, namely Transformer Block1, Transformer Block2, Transformer Block3, and Transformer Block4. Each layer of the Transformer structure has the same architecture but different sizes, and the multi-scale features with decreasing resolution are obtained from Block1 to Block4. Each layer of the Transformer structure includes an Efficient Self-Attention submodule, a Mix-FFN submodule, and an Overlap Patch Merging submodule. Unlike traditional Transformer encoders, the encoder in this embodiment uses a Mix-FFN feedforward neural network to introduce positional information; Mix-FFN combines a global feedforward network (GlobalFFN) and a local feedforward network (Local FFN); Global FFN is a global feedforward network with a large receptive field, used to capture global context; Local FFN is a local feedforward network with a small receptive field, used to capture local details; the operation of Mix-FFN is shown in Equation (1).

[0042] (1)

[0043] Where, x inThis represents the features output by Efficient Self-Attention, where MLP stands for Multilayer Perception Module, GELU is the activation function, and Conv... 3×3 This is a 3×3 convolution calculation. The Overlap Patch Merging submodule merges overlapping image patches. It uses the method of dividing the image into overlapping patches to maximize the preservation of image information. Overlap Patch Merging processes the overlapping areas of the image patches using methods such as averaging and weighted averaging to merge them into a complete image. When the next Transformer Block receives the image from the previous Transformer Block, it first segments it into image patches. The outputs of Transformer Block1, Transformer Block2, and Transformer Block3 are then skipped to the subsequent decoding modules.

[0044] In Transformer Block 4, the enhanced image X is obtained after merging. i X W X s1 X s2 Corresponding image X i '、X W '、X s1 '、X s2 ', where image X W Image X is obtained after feature perturbation. fp '.

[0045] Self-attention modules capture long-range dependencies across locations, enabling models to capture global contextual information and improve segmentation accuracy. The core idea of ​​the self-attention mechanism is to dynamically adjust feature representations by modeling the relationships between each element and other elements in the sequence. Self-attention can examine the entire input image to identify prominent regions. The input to the self-attention module is a feature matrix, which undergoes three different linear transformations to generate query (Q), key (K), and value (V) matrices:

[0046] Q=XW q

[0047] K=XW k

[0048] V=XW v

[0049] Among them W q W k and W vHere, X is the learned weight matrix, and X is the input embedding. Subsequent steps involve calculating attention weights, achieved by determining the dot product of the query matrix and the key matrix, thus generating a similarity matrix. This similarity matrix is ​​then transformed into a probability distribution by applying the Softmax function.

[0050]

[0051] Where d k This represents the dimension of the keys, which is scaled to avoid excessively high values. Finally, the calculated attention weights are weighted and summed to obtain the output feature matrix:

[0052]

[0053] The weighted feature matrix output by the attention module can be passed to the decoder for decoding. Combining the self-attention module between the last layer of the encoder and the first layer of the decoder can significantly enhance model performance. Semantic segmentation benefits from rich contextual information, and the self-attention mechanism enables the model to capture long-range dependencies, effectively integrating global context into the deep network structure. By applying the self-attention module before passing features to the decoder, the model benefits from rich contextual representations. The flexibility and versatility of the self-attention mechanism allow the model to adapt more effectively to various application scenarios, thereby improving performance and generalization ability.

[0054] The decoding module comprises a four-layer MLP structure: MLP Layer 1, MLP Layer 2, MLP Layer 3, and MLPLayer4. MLP Layer 1 receives the Self-Attention output as input; MLP Layer 2 receives the concatenation of the outputs of MLPLayer1 and Transformer Block 1; MLP Layer 3 receives the concatenation of the outputs of MLP Layer 2 and Transformer Block 2; and MLP Layer 4 receives the concatenation of the outputs of MLP Layer 3 and Transformer Block 3. Through this multi-layered MLP structure, each MLP layer fuses the output of its corresponding Transformer Block, effectively aggregating local feature information from different layers. Simultaneously, each subsequent MLP layer receives global feature information from the previous MLP layer, combining local and global attention to generate a powerful feature representation. In each MLP layer, multi-level features from the Transformer Block and / or the previous MLP layer are upsampled and concatenated after channel dimension unification. A multi-layer perceptual module (MLP) is used to fuse the concatenated features, and the fused feature map is then passed through a fully connected layer to predict the semantic segmentation mask M.

[0055]

[0056] in This represents the upsampled feature map, where C is the number of channels output by the decoder. M is the number of categories, and M is the predicted segmentation mask. The output of MLP Layer 4 is processed by convolution and interpolation to obtain the perturbed predicted images p. i p W p fp p s1 and p s2 .

[0057] In the self-supervised module, the loss function L is used... UniMatch To supervise the perturbation prediction pairs. Loss function L UniMatch Including monitoring loss L x And unsupervised loss L u composition

[0058]

[0059] Supervision loss function L sup It is the cross-entropy loss between labeled data and real data.

[0060]

[0061] in, Indicates the batch size of the labeled data. This represents a prediction of the labeled data. It is cross-entropy loss, y i Data that is manually labeled.

[0062] Unsupervised loss function L u It is used to evaluate the prediction consistency between weakly enhanced image predictions and strongly enhanced image predictions, i.e., the prediction p using a weakly enhanced perturbation image. W To simultaneously supervise the prediction p of the feature-perturbed image fp And the predicted p of two strongly enhanced perturbation images s1 and p s2

[0063]

[0064] in This is the batch size of the unlabeled data. It is the confidence threshold for filtering noise labels, where H represents minimizing two probability distributions. and These are hyperparameters, and all have a value of 0.5.

[0065] Combination Figure 1 A differential bi-branch strongly enhanced perturbation semi-supervised image semantic segmentation training method includes:

[0066] Step S100: In the same network, weak enhancement perturbation processing is performed on the labeled image, and weak enhancement perturbation processing and two-way strong enhancement perturbation processing are performed on the unlabeled image.

[0067] Step S200: The SegFormer network is used to train the perturbation prediction p of the labeled image. i Unlabeled image weak enhancement perturbation prediction p W Unlabeled image strong enhancement perturbation p s1 and p s2 And feature perturbation prediction p of unlabeled weakly enhanced perturbation images fp ;

[0068] Step S300: Monitor the consistency of each disturbance prediction.

[0069] In step S100, for N l A dataset sample with 1 label The image is subjected to weak enhancement perturbation to obtain a weakly enhanced image X. i ; For N u Unlabeled dataset samples The image is subjected to weak enhancement perturbation to obtain a weakly enhanced image X. W Weakly enhanced image X W The copy is then subjected to two random strong enhancement perturbation branches to obtain the strongly enhanced image X. s1 and X s2 .

[0070] In step S100, the computation is performed within the same network. Compared to perturbation processing via a teacher-student network, this eliminates the need to update network weights, thus reducing the computational process.

[0071] Combination Figure 1 The specific process of step S200 includes:

[0072] Step S201, enhance image X i X W X s1 X s2 The image is divided into multiple image blocks and its features are extracted. Each image block is encoded into a vector and the vectors of all image blocks are concatenated to obtain image X.

[0073] Step S202: Obtain multi-scale features of image X through multiple layers of coding modules with different resolutions and sizes;

[0074] Step S203: Perform feature perturbation processing on the unlabeled weakly enhanced perturbation image to obtain the feature perturbation image X. fp ';

[0075] Step S204: Capture global contextual information of each image through a self-attention mechanism;

[0076] Step S205: Obtain multi-scale features of image X through a multi-layer decoder and aggregate them to obtain the final features;

[0077] Step S206: Convolve the final features and upsample them to the original image size to obtain the perturbation prediction p of the labeled image. i Unlabeled image weak enhancement perturbation prediction p W Unlabeled image strong enhancement perturbation p s1 and p s2 And feature perturbation prediction p of unlabeled weakly enhanced perturbation images fp .

[0078] In step S206, weak enhancement prediction p W and two strongly enhanced prediction p s1 and p s2 They are respectively represented as

[0079]

[0080] Among them, AW Indicating a weakly enhanced perturbation, A S This indicates a strong enhancement perturbation. The prediction p for a weak enhancement perturbation image... W And two strongly enhanced perturbation predictions p s1 and p s2 It was obtained within the same network.

[0081] Feature perturbations are introduced between encoder g and decoder h.

[0082]

[0083] Among them, e W The encoder extracts unlabeled data x from weakly amplified perturbations. u The features extracted are represented by P, which is the method for adding feature perturbations. During training, the prediction p of the weakly enhanced perturbation image is used. W To simultaneously supervise the prediction p of the feature-perturbed image fp And the predicted p of two strongly enhanced perturbation images s1 and p s2 .

[0084] In step S300, the loss function L is set. UniMatch Supervise the perturbation prediction pairs. For the perturbation prediction p obtained from the labeled image. i Through the cross-entropy loss L between the data and the real label data sup Supervise

[0085]

[0086] in, Indicates the batch size of the labeled data. This represents a prediction of the labeled data. It is cross-entropy loss, y i This is real label data.

[0087] For supervision of prediction of unlabeled image perturbations, the prediction p of the weakly enhanced perturbation image is used. W To simultaneously supervise the prediction p of the feature-perturbed image fp And the predicted p of two strongly enhanced perturbation images s1 and p s2

[0088]

[0089] in This is the batch size of the unlabeled data; It is the confidence threshold for filtering noise labels; H represents minimizing two probability distributions; and As hyperparameters, image perturbation and feature perturbation each have their advantages. and The value is 0.5.

[0090] Example

[0091] 1. Dataset

[0092] This embodiment evaluates the UniFormer method on three datasets commonly used in the field of semi-supervised semantic segmentation: PASCAL VOC 2012, Cityscapes, and COCO datasets. The PASCAL VOC 2012 dataset has approximately 4000 samples, divided into three subsets: training set, validation set, and test set; the training set contains 1464 images, the validation set contains 1449 images, and the test set contains 1456 images; this dataset has 21 categories, including one background class and 20 different foreground classes; each category has detailed pixel-level annotations; 9,118 coarsely labeled images were added from the SBD dataset as supplementary training data, called the augmentation set. The Cityscapes dataset is a large dataset for semantic understanding of urban street scenes. In the field of semantic segmentation, the finely annotated set in Cityscapes is essential. It contains 5000 high-quality, pixel-level finely annotated images, covering different scenes in 50 cities. The images are divided into three sets: training, validation, and test. The training set contains 2975 images, the validation set contains 500 images, and the test set contains 1524 images. The Cityscapes dataset has 19 semantic categories, divided into 6 superclasses. The COCO dataset is a large-scale dataset for visual fields such as semantic segmentation. The COCO dataset contains 118,000 training images and 5,000 validation images, covering 91 object categories, such as people, animals, vehicles, furniture, and other everyday objects. The images come from different scenes, including indoor, outdoor, and urban street scenes, exhibiting high diversity and complexity.

[0093] 2. Implementation details

[0094] The experimental model is based on the SegFormer architecture. For the three datasets mentioned above, the model was trained using a single GPU and stochastic gradient descent (SGD) as the optimizer. The initial learning rates for the Pascal, Cityscapes, and COCO datasets were set to 0.0005, 0.002, and 0.0005, respectively. During training, [the model was trained using...]. To adjust the learning rate, among which Let represent the initial learning rate, i represent the current iteration, and I represent the maximum number of iterations. The Pascal and COCO datasets used the cross-entropy loss (CELoss) function, while the Cityscapes dataset used Online Hard Sample Mining (OHEM) and a sliding window evaluation. A channel loss with a probability of 0.5 (nn.Dropout2d(0.5) in PyTorch) was used as a feature perturbation and inserted between the encoder and decoder. The self-attention module was located between the last module (Block4) in the encoder and the first layer (Layer1) in the decoder. The image resolution of the Pascal and COCO datasets was set to 513×513, and they were trained for 80 and 10 epochs respectively. The image resolution of the Cityscapes dataset was set to 801×801, and it was trained for 240 epochs. The results are shown in Tables 2-5 after training.

[0095] Table 2 compares the results with other methods on the original Pascal dataset.

[0096]

[0097] Table 3 Comparison with other methods on the Pascal augmented dataset.

[0098]

[0099] † indicates the use of U 2 PL method same split

[0100] Table 4 Comparison of methods on the Cityscapes dataset (using a lightweight model)

[0101]

[0102] Table 5 compares the results with other methods on different partitions of the COCO dataset.

[0103]

[0104] 3. Comparison of Results

[0105] The original PASCAL VOC 2012 dataset is used. Table 2 shows the performance metrics of various semi-supervised semantic segmentation methods using different proportions of labeled data from the original Pascal dataset. As shown in the table, the UniFormer method outperforms previous state-of-the-art methods, especially when using the fully labeled dataset (1464), where the score improvement is most significant. These results effectively demonstrate the effectiveness of the UniFormer method.

[0106] The PASCAL VOC 2012 augmentation dataset. Table 3 reports the comparison results of different methods using the augmented dataset. Again, the UniFormer method still outperforms previous state-of-the-art methods on the augmented dataset. Furthermore, comparisons were made using the same settings as U2PL, where all labeled data is of high quality. In this scenario, the improvements of the proposed UniFormer are very significant, fully demonstrating the importance of high-quality labels.

[0107] The Cityscapes dataset. Table 4 lists the comparison results of the UniFormer method with previous methods on the Cityscapes dataset. Because the images in the Cityscapes dataset have richer urban landscape details, the resolution of the images in the dataset was set to 801×801. The smaller model SegFormerB3 was used as the backbone of the UniFormer method. Compared with other methods using smaller models, the UniFormer method achieved certain advantages.

[0108] The COCO dataset. Table 5 shows the comparison results of the UniFormer method with most previous methods. The images in the COCO dataset are from diverse sources and contain rich semantic information, which makes training difficult. However, the UniFormer method still outperforms previous state-of-the-art methods on some proportions of labeled data. This result fully demonstrates the effectiveness of the UniFormer method.

[0109] 4. Ablation test

[0110] Using the SegFormer-B5 model as the backbone, a series of ablation experiments were conducted on 1 / 4 (2646) of the labeled samples on the Pascal augmented dataset to verify the effectiveness of the proposed UniFormer method.

[0111] Table 6 shows the effectiveness of the proposed components on the Pascal augmented dataset.

[0112]

[0113] RSAP is the random strong enhancement perturbation method proposed in this invention. The results are compared on a 1 / 4 scale of labeled data. Replacing the backbone of the UniMatch method with SegFormer-B5 shows a significant improvement over using ResNet-101 as the backbone. The addition of the self-attention module and the random strong enhancement perturbation (RSAP) module also improves the segmentation performance.

[0114] Table 7. Evaluation of the performance of the self-attention module by adding it to different layers of the encoder.

[0115]

[0116] Table 7 evaluates the performance of the self-attention module by adding it after different layers of the encoder, with the best results observed when it is added to the last layer.

[0117] Table 8. Performance of strong augmentation methods using different numbers of k in the stochastic strong augmentation perturbation method on different datasets.

[0118]

[0119] Table 8 shows the performance of the random strong augmentation perturbation (RSAP) method using different numbers (k) of strong augmentations on different datasets, with the best performance when k=3.

[0120] The effectiveness of the UniFormer method is shown in Table 6, which illustrates the effectiveness of each structure by sequential addition. First, the standard semi-supervised semantic segmentation model SegFormer-B5 is used as the baseline. Next, the UniMatch framework is added to the SegFormer-B5 baseline, followed by self-attention and random strong augmentation perturbation (RSAP) modules. Results show that each component improves the model's performance. Adding UniMatch to SegFormer-B5 improves performance by 3.51% compared to the baseline. Introducing the RSAP module further improves performance by 0.43%, for a total improvement of 3.94%. Replacing RSAP with the self-attention module yields a 4.07% improvement, with the self-attention module contributing 0.56%. Finally, combining RSAP and the self-attention module achieves the best performance, improving performance by 4.28% compared to the baseline.

[0121] Figure 3 The UniFormer method is visualized in groupings of 1464 labeled data points on the original Pascal dataset. The UniFormer method is also compared with UniMatch and AllSpark methods. From top to bottom, the images are the original image, the visualization of UniMatch, the visualization of AllSpark, the visualization of UniFormer, and the ground truth labels.

[0122] like Figure 3As shown, to improve the reliability of the experimental results, the results of the UniFormer method are visualized and compared with the visualization results of the UniMatch and AllSpark methods. The visualization results are all based on the training results of all label ratios in the initial Pascal dataset. The results show that the UniFormer visualization results perform better in many details. The UniMatch and AllSpark methods revealed some prediction errors through visualization, while the UniFormer method had significantly fewer mispredictions, higher accuracy, and the visualization results were closer to the real situation.

[0123] The SegFormer model consists of four hierarchical structures connecting the encoder and decoder. Determining which hierarchical structure to add the self-attention module to achieve optimal performance is crucial. Table 7 uses SegFormer-B5 as a baseline to evaluate the model performance after adding the self-attention module to different encoder blocks. The dataset used is the 1 / 4 (2646) Pascal augmentation dataset. Optimal performance is achieved when the self-attention module is added between the fourth encoder layer (Block4) and the first decoder layer (Layer1) because this is where the semantic information is richest, allowing the self-attention module to perform optimally.

[0124] The performance of randomly selecting k strong enhancement types was evaluated through experiments and compared with the fixed strong enhancement method. Figure 4 The experimental results are presented, and Table 8 shows the performance of different numbers of strong augmentation methods on the two datasets (Pascal VOC and Cityscapes). For the standard Pascal dataset, the labeled data was split into 1 / 2 (732) proportions. The results clearly demonstrate the effectiveness of the randomized strong augmentation method, achieving optimal performance when k=3.

Claims

1. A differential bi-branch strongly enhanced perturbation semi-supervised image semantic segmentation training method, characterized in that, include: Step S100: In the same network, weak enhancement perturbation processing is performed on the labeled image, and weak enhancement perturbation processing and two-way random strong enhancement perturbation processing are performed on the unlabeled image; Step S200: Train the perturbation-processed image to obtain perturbation prediction for labeled image, weak enhancement perturbation prediction for unlabeled image, strong enhancement perturbation for unlabeled image, and feature perturbation prediction for weak enhancement perturbation image of unlabeled image; Step S200 specifically includes the following process: Step S201: Divide the image obtained in step S100 into multiple image blocks and extract features, encode each image block into a vector, and concatenate the vectors of all image blocks. Step S202: The concatenated image is processed through multiple layers of encoding modules with different resolutions to obtain a multi-scale feature image; Step S203: Perform feature perturbation processing on the unlabeled weakly enhanced perturbation image to obtain a feature perturbation image; Step S204: Capture global contextual information of each image through a self-attention mechanism; Step S205: Multi-scale features are obtained through a multi-layer decoding module and then aggregated to obtain the final feature image; Step S206: Convolve and upsample the final feature image to the original image size to obtain the perturbation prediction of the labeled image, the weak enhancement perturbation prediction of the unlabeled image, the strong enhancement perturbation of the unlabeled image, and the feature perturbation prediction of the weak enhancement perturbation image of the unlabeled image. Step S300: Establish a loss function to supervise the perturbation prediction obtained from the labeled image, and use the prediction of the weakly enhanced perturbation image to simultaneously supervise the prediction of the feature perturbation image and the prediction of the two strongly enhanced perturbation images. The loss function L in step S300 UniMatch for Wherein, the supervised loss function L sup It is the cross-entropy loss between labeled data and real data, and the unsupervised loss L. u It is used to evaluate the prediction consistency between weakly enhanced image predictions and strongly enhanced image predictions; Supervision loss function L sup for in, Indicates the batch size of the labeled data. This represents a prediction of the labeled data. It is cross-entropy loss, y i This is real label data; Unsupervised loss function L u for Where, p W It is a prediction of weak enhancement perturbations in unlabeled images, p s1 and p s2 It is a strong enhancement perturbation of the unlabeled image, p fp It is a feature perturbation prediction of an unlabeled image with weak enhancement perturbation. This is the batch size of the unlabeled data. It is the confidence threshold for filtering noise labels, where H represents minimizing two probability distributions. and This is a hyperparameter.

2. The method according to claim 1, characterized in that, In step S100, the images of labeled dataset samples are subjected to weak enhancement perturbation to obtain a first weakly enhanced image; the images of unlabeled dataset samples are subjected to weak enhancement perturbation to obtain a second weakly enhanced image, and a copy of the second weakly enhanced image is subjected to two random strong enhancement perturbation branches to obtain a first strongly enhanced image and a first strongly enhanced image.

3. The method according to claim 2, characterized in that, In step S100, each random strong enhancement perturbation process randomly selects three processing methods from {standard, automatic contrast, equalization, inversion, Gaussian blur, contrast, sharpness, color, brightness, hue, tone separation, overexposure}.

4. The method according to claim 3, characterized in that, In step S202, each layer of the coding module has the same structure, and the resolution decreases sequentially; each layer of the coding module includes an efficient attention submodule, a Mix-FFN submodule, and an Overlap PatchMerging submodule.

5. The method according to claim 4, characterized in that, In step S205, each layer decoding module obtains the feature image of the corresponding layer encoding module.

Images