Multi-behavior pattern collaborative training method for semi-supervised semantic segmentation training

Abstract

This invention discloses a multi-behavior pattern collaborative training method for semi-supervised semantic segmentation training, comprising: learning models R1 and R2 using labeled images and generating pseudo-labels for each subnet from unlabeled data; adding a behavior pattern vector extraction module to each model; introducing a new data stream for each subnet and applying different feature perturbations to the data stream; and integrating the data streams to obtain the total loss function for each model during training. This invention improves segmentation performance by enhancing the diversity of subnets in the collaborative training framework.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a multi-behavior pattern collaborative training method for semi-supervised semantic segmentation training. Background Technology

[0002] Semantic segmentation is a typical computer vision problem that involves taking some raw data (e.g., planar images) as input and transforming it into masks with highlighted regions of interest. Many people use the term full-pixel semantic segmentation, where each pixel in an image is assigned a category ID based on the object of interest it belongs to.

[0003] Early computer vision problems only detected elements such as edges (lines and curves) or gradients, but they never provided pixel-level image understanding in a way that fully resembled human perception. Semantic segmentation addresses this problem by grouping image parts belonging to the same object together, thus expanding its application scope.

[0004] The significant success of current semantic segmentation relies primarily on supervised training. However, supervised training requires a mask for each image, and detailed mask annotation is extremely time-consuming and labor-intensive. Compared to the difficulty of obtaining masks, images are much easier to acquire. Therefore, semi-supervised semantic segmentation has attracted considerable attention, aiming to train segmentation models using a small amount of labeled data and a large amount of unlabeled data.

[0005] The current mainstream approach combines pseudo-labeling and consistency regularization techniques. Based on frameworks such as the average teacher framework and FixMatch, it demonstrates impressive results. The main steps are: 1. The model itself (within the FixMatch framework) or its average teacher (within the average teacher framework) predicts weakly augmented unlabeled data. The predictions are then filtered by prediction confidence; the predictions that pass the confidence screening become pseudo-labels as supervisory signals for the unlabeled data. 2. The model predicts strongly augmented unlabeled data, and the pseudo-labels obtained in the previous step at the corresponding pixel positions provide supervision.

[0006] Based on the methods described above, the current obstacle to further performance improvement in semi-supervised semantic segmentation training paradigms is the inevitable noise inherent in the pseudo-labels generated by pseudo-labeling techniques. This is because: 1. In semi-supervised semantic segmentation training, the cross-entropy loss function is typically used, which can lead to the model exhibiting overconfidence in its predictions. 2. In semi-supervised semantic segmentation training, the limited labeled data cannot cover the diversity of the entire category, resulting in cognitive bias in the model. 3. Cognitive bias leads to incorrect predictions, and these incorrect predictions, due to their overconfidence, are filtered out by prediction confidence and become pseudo-labels. The noise generated by its own cognitive bias becomes a supervisory signal, making it more difficult for the model to overcome its cognitive bias and limiting further performance improvement.

[0007] For this problem, the co-training framework outperforms the two common frameworks mentioned above. The main difference between the co-training framework and the previous two is that the co-training framework includes two dissimilar subnets. The predictions from one subnet are processed using pseudo-labeling techniques and used as a supervisory signal to supervise the training of the other subnet. Because the supervisory signal comes from a subnet that is dissimilar to itself, the co-training framework outperforms the FixMatch framework and the average teacher framework when facing this problem.

[0008] However, due to the cross-supervision within the collaborative framework (where the supervision signal comes from another subnet), this mutual learning makes the subnets in the collaborative framework prone to rapid convergence. Converging subnets mean that different subnets will make the same predictions, thus losing the aforementioned advantages. Therefore, creating and maintaining the dissimilarity of subnets within the collaborative framework is extremely crucial.

[0009] Current methods include CPS (Semi-Supervised Semantic Segmentation with Cross-Pseudo Supervision), which obtains dissimilar subnetworks by initializing them differently; and CCVC (Conflict-Based Cross-View Consistency for Semi-Supervised Semantic Segmentation), which creates and maintains differences between subnetworks by constraining the dissimilarity of features extracted by the two subnetworks from the same image. However, current methods, such as CPS, tend to converge quickly to consistency and lose their distinctiveness as cross-supervision progresses. CCVC, on the other hand, directly optimizes the dissimilarity of features extracted by the two subnetworks from the same image. However, current deep learning model training primarily involves fine-tuning with pre-trained weights, and directly expecting feature dissimilarity for subsequent downstream tasks faces a trade-off between distinctiveness and performance. Summary of the Invention

[0010] To address the problems existing in the prior art, the purpose of this invention is to provide a multi-behavior pattern collaborative training method for semi-supervised semantic segmentation training. This invention improves segmentation performance by enhancing the diversity of subnets in the collaborative training framework.

[0011] To achieve the above objectives, the technical solution adopted by this invention is: a multi-behavior pattern collaborative training method for semi-supervised semantic segmentation training, comprising the following steps:

[0012] Step 1: Learn from labeled images for model R1 and model R2, and generate pseudo-labels for each subnet using unlabeled data;

[0013] Step 2: Add a behavior pattern vector extraction module to each model;

[0014] Step 3: Introduce a new data stream for each subnet and apply different feature perturbations to the data stream;

[0015] Step 4: Integrate steps 1-3 to obtain the total loss function for each model during training.

[0016] As a further improvement of the present invention, in step 1, the loss function of the labeled image is as follows:

[0017]

[0018] Among them, M x For labeled data fields, H and W represent the height and width of the image, and L... ce p represents the cross-entropy loss function. ij This represents the predicted value of the pixel at row i and column j, where GT represents its corresponding label value, and A w The weak perturbation operation representing the image, x i This represents the i-th labeled image.

[0019] As a further improvement of the present invention, in step 1, each subnet generates a pseudo-label y for the unlabeled data. 1 y 2 The generation process is as follows:

[0020] The pseudo-labels of each subnet, after being filtered by a confidence threshold, serve as the supervision signal for the training of another subnet on unlabeled images. The loss function is as follows:

[0021]

[0022] Among them, M u A represents an unlabeled data field. s The strong perturbation operation representing the image, u i This represents the i-th unlabeled image.

[0023] As a further improvement of the present invention, step 2 is specifically as follows:

[0024] The behavioral pattern vectors extracted by model R1 and model R2 are called v. 1 and v 2 And optimize their dissimilarity during training to promote different behavioral patterns in different encoders:

[0025]

[0026] in, This represents the behavior pattern vector extracted by R1 from the i-th image. represents the behavior pattern vector extracted by R2 for the i-th image, and sim represents the cosine similarity calculation.

[0027] As a further improvement to the present invention, step 3 is specifically as follows:

[0028] In the training of model R1, a data stream is introduced: the image is weakly enhanced at the input level and then fed into the encoder to extract features. Subsequently, weak perturbations are applied to the shallow features and strong perturbations to the deep features. In the training of model R2, a data stream is introduced: the image is weakly enhanced at the input level and then fed into the encoder to extract features. Subsequently, strong perturbations are applied to the shallow features and weak perturbations to the deep features. The loss function is as follows:

[0029]

[0030] Where A dfp This represents the aforementioned perturbation of the differential characteristics.

[0031] As a further improvement of the present invention, in step 4, the total loss function is as follows:

[0032] L total =λ x L x +λ u L u +λ sim L sim +λ dfp L dfp

[0033] Where, λ x , λ u , λ sim , λ dfp All of these are hyperparameters.

[0034] The beneficial effects of this invention are:

[0035] This invention does not directly involve the features extracted by the encoder or the prediction results obtained by the decoder. Instead, it improves the segmentation performance of images by enhancing the diversity of subnetworks in the collaborative training framework. Attached Figure Description

[0036] Figure 1 This is a schematic diagram of the behavior pattern vector extraction module in an embodiment of the present invention;

[0037] Figure 2 This is a comparison chart for prediction visualization in an embodiment of the present invention. Detailed Implementation

[0038] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0039] Example

[0040] A multi-behavior pattern collaborative training method for semi-supervised semantic segmentation training includes the following steps:

[0041] Step 1: In a common co-training framework used for semi-supervised training, there are two models, R1 and R2, each learning from labeled images. The loss function for labeled images can be described as:

[0042]

[0043] Among them, M x For labeled data fields, H and W represent the height and width of the image, and L... ce p represents the cross-entropy loss function. ij This represents the predicted value of the pixel at row i and column j, where GT represents its corresponding label value, and A w Represents weak perturbation operations on the image (e.g., flipping, resizing, etc.), x i This represents the i-th unlabeled image.

[0044] In addition, each subnet generates a pseudo-label y for each unlabeled image. 1 y 2 The generation process is as follows:

[0045]

[0046] The pseudo-labels of each subnet, after being filtered by a confidence threshold, will serve as supervision signals for the training of another subnet on unlabeled images. Taking R1 as an example, the loss function can be described as:

[0047]

[0048] Among them, M u For unlabeled data fields, A sRepresents strong perturbation operations on the image (e.g., color dithering, random grayscale conversion, etc.), u i This represents the i-th unlabeled image.

[0049] Step 2: Add a Behavior Pattern Vector Extraction (BPVEM) module to each model. The specific structure and forward propagation method of this module are as follows: Figure 1 As shown, Figure 1 In the middle, F 1 F 2 The encoders representing Model 1 and Model 2, C 0 C 1 The features extracted by the encoder are shallow to deep features, while BPV is the behavioral pattern vector extracted by this module for each subnet.

[0050] The behavioral pattern vectors extracted from R1 and R2 are called v. 1 and v 2 Furthermore, during training, their dissimilarity is optimized to promote different behavioral patterns in different encoders.

[0051]

[0052] in, This represents the behavior pattern vector extracted by R1 from the i-th image. represents the behavior pattern vector extracted by R2 for the i-th image, and sim represents the cosine similarity calculation.

[0053] Step 3: Introduce a new data stream for each subnet and apply different feature perturbations to it. Specifically, in the training of R1, a data stream is introduced: the image is weakly enhanced at the input level and then fed into the encoder to extract features. Weak perturbations are then applied to the shallow features (e.g., 20% random channel loss) and strong perturbations are applied to the deep features (e.g., 80% random channel loss). In the training of R2, the opposite is true: strong perturbations are applied to shallow features and weak perturbations to deep features. These differential feature perturbations cultivate different behavioral patterns in different decoders. Taking R1 as an example, the loss function for this step can be described as:

[0054]

[0055] Where A dfp This represents the aforementioned perturbation of the differential characteristics.

[0056] Step 4: By integrating the above steps, the total loss function for each model during training can be described as:

[0057] L total =λ x L x +λu L u +λ sim L sim +λ dfp L dfp

[0058] The values ​​of the four hyperparameters from left to right are 1, 0.5, 1, and 0.5, respectively.

[0059] On the classic Pascal VOC 2012 dataset, the mIoU values ​​of the implemented method during validation were compared with those of DeepLabv3plus+ResNet101 / ResNet-50 models trained using recent mainstream methods. Specific values ​​are shown in Tables 1 and 2, which indicate the specific backbone model used and the amount of labeled data used.

[0060]

[0061] Table 1

[0062]

[0063] Table 2

[0064] In addition, this embodiment was compared with other methods on the more challenging Cityscapes dataset, and the results are shown in Table 3:

[0065]

[0066] Table 3

[0067] In addition, such as Figure 2 As shown, the visualization of this embodiment's method allows for a more intuitive comparison with other methods' predictions during validation. Figure 2 In the image, from left to right, are the input image, the prediction result of the nearest neighbor method (UniMatch), the prediction result of the method proposed in this embodiment, and the ground truth.

[0068] The embodiments described above are merely illustrative of specific implementations of the present invention, and while the descriptions are detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.

Claims

1. A multi-behavior pattern collaborative training method for semi-supervised semantic segmentation training, characterized in that, Includes the following steps: Step 1: Learn from labeled images for model R1 and model R2, and generate pseudo-labels for each subnet using unlabeled data; Step 2: Add a behavior pattern vector extraction module to each model; Step 3: Introduce a new data stream for each subnet and apply different feature perturbations to the data stream; Step 4: Integrate steps 1-3 to obtain the total loss function for each model during training.

2. The multi-behavior pattern collaborative training method for semi-supervised semantic segmentation training according to claim 1, characterized in that, In step 1, the loss function for the labeled image is as follows: Among them, M x For labeled data fields, H and W represent the height and width of the image, and L... ce p represents the cross-entropy loss function. ij This represents the predicted value of the pixel at the i-th row and j-th column, where GT represents its corresponding label value, and A... w The weak perturbation operation representing the image, x i This represents the i-th labeled image.

3. The multi-behavior pattern collaborative training method for semi-supervised semantic segmentation training according to claim 2, characterized in that, In step 1, each subnet generates a pseudo-label y for the unlabeled data. 1 y 2 The generation process is as follows: The pseudo-labels of each subnet, after being filtered by a confidence threshold, serve as the supervision signal for the training of another subnet on unlabeled images. The loss function is as follows: Among them, M u For unlabeled data fields, A s The strong perturbation operation representing the image, u i This represents the i-th unlabeled image.

4. The multi-behavior pattern collaborative training method for semi-supervised semantic segmentation training according to claim 3, characterized in that, Step 2 is described in detail below: The behavioral pattern vectors extracted by model R1 and model R2 are called v. 1 and v 2 And optimize their dissimilarity during training to promote different behavioral patterns in different encoders: in, This represents the behavior pattern vector extracted by R1 from the i-th image. represents the behavior pattern vector extracted by R2 for the i-th image, and sim represents the cosine similarity calculation.

5. The multi-behavior pattern collaborative training method for semi-supervised semantic segmentation training according to claim 4, characterized in that, Step 3 is described in detail below: In the training of model R1, a data stream is introduced: the image is weakly enhanced at the input level and then fed into the encoder to extract features. Subsequently, weak perturbations are applied to the shallow features and strong perturbations to the deep features. In the training of model R2, a data stream is introduced: the image is weakly enhanced at the input level and then fed into the encoder to extract features. Subsequently, strong perturbations are applied to the shallow features and weak perturbations to the deep features. The loss function is as follows: Where A dfp This represents a perturbation of differential characteristics.

6. The multi-behavior pattern collaborative training method for semi-supervised semantic segmentation training according to claim 5, characterized in that, In step 4, the total loss function is as follows: L total =λ x L x +λ u L u +λ sim L sim +λ dfp L dfp Where, λ x , λ u , λ sim , λ dfp All of these are hyperparameters.

Images