An image semi-supervised semantic segmentation method based on conservative aggressive co-learning

Abstract

The present invention discloses a method for semi-supervised semantic segmentation of images based on conservative and radical collaborative learning, comprising: training an image semantic segmentation model with a traditional supervised training method for labeled data; and performing semi-supervised training of the model with a conservative and radical collaborative learning method for unlabeled data; wherein, the conservative and radical collaborative learning method trains two branches simultaneously. In the present invention, effective and efficient use of a large amount of unlabeled data is achieved, and based on the idea of ​​"seeking common ground while reserving differences", synergy between "conservative evolution" and "radical exploration" is achieved, which helps to reduce costs and increase efficiency in training environmental perception models of intelligent systems, has good semantic segmentation effect, robustness and versatility, and can achieve the same semantic segmentation effect as a supervised method with higher annotation costs using only a small amount of labeled data. It can be deployed and applied to various intelligent systems and has high application value.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and particularly relates to an image semi-supervised semantic segmentation method based on conservative aggressive co-learning. BACKGROUND

[0002] Image semantic segmentation is an important computer vision problem and a key basic technology widely applied in intelligent system environment perception. Thanks to the effective application of deep supervised learning, advanced methods can achieve good image semantic segmentation results and meet use requirements when sufficient pixel-level labeled data is available for supervised training. However, when the amount of labeled data is insufficient, the semantic segmentation results predicted by the model are poor, that is, the effect of the model is largely restricted by the amount of labeled data. At the same time, the labeling process of pixel-level labeled data is quite time-consuming and costly. For example, for a high-resolution urban road scene picture, the time for manually labeling a pixel-level sample usually exceeds 1.5 hours, which is 15 times and 60 times of the region-level and image-level labeling processes. Therefore, in recent years, a large number of research works focus on achieving better model training results with less labeling cost by relying only on a small amount of labeled data and using a large amount of unlabeled data. Since this type of research has significant cost reduction and efficiency improvement advantages, it has attracted much attention from the academic and industrial circles and is an important computer vision research direction and engineering problem in recent years.

[0003] To achieve the goal of utilizing large-scale unlabeled data, many semi-supervised learning methods for image semantic segmentation have been proposed in recent years. Methods based on the idea of minimizing entropy expect the model to have a small entropy on the prediction of unlabeled data, that is, the model can generate a clear and definite prediction result. Self-training is a typical representative of this kind of method, which first uses labeled data for supervised training, then generates pseudo-labels for unlabeled data by the trained model, and then uses the unlabeled data and the corresponding pseudo-labels for pseudo-supervised training of the model to achieve further training optimization. Methods based on the idea of consistency constraint aim to make the model maintain the consistency of the prediction result when facing various perturbations. Common perturbations include input perturbation (introducing noise to the input picture or image transformation such as cropping), feature perturbation and network model perturbation (such as using multi-branch model or using Dropout technology). The effect of this kind of method depends largely on the perturbation method selected. It is worth noting that the above two kinds of methods are based on pseudo-supervision in implementation, and the pseudo-supervision label is generated from the model, so it is inevitable that there are errors in the pseudo-supervision label. To solve this problem, some recent methods use the prediction confidence of the model as a sign to threshold the pseudo-label, but at the same time, this also means that a large proportion of unlabeled data will be discarded, which is contrary to the goal of utilizing large-scale unlabeled data. These methods are always plagued by the trade-off between quality and quantity. In addition to the prediction confidence, the consensus degree between multiple prediction results is another sign. Some mutual learning methods are based on this to achieve the generation or optimization of pseudo-supervision. Teacher-Student (T-S) and Student-Student (S-S) are two typical mutual learning modes. The inequality of the two models in T-S may lead to the wrong optimization direction of the student model and the limitation of the development of the student model, while the two models trained in the S-S mode have the potential problem of model coupling.

[0004] In view of the above limitations and deficiencies, the present application proposes a conservative and aggressive collaborative learning based image semi-supervised semantic segmentation method to solve the key problem that high-quality and large-quantity pseudo labels cannot be achieved simultaneously in the existing method and the actual use requirements. The method simultaneously trains two branches, one branch is conservative and uses high-quality pseudo labels for training based on intersection pseudo supervision, and the other branch is aggressive and uses a large number of pseudo labels for exploration under union pseudo supervision. The two branches use heterogeneous knowledge for pseudo supervision training, which can improve the model coupling problem. The method realizes the effective and efficient use of a large number of unlabeled data, achieves the synergy between conservative evolution and aggressive exploration, and helps to reduce the cost and increase the efficiency of intelligent system environment perception model training. The semantic segmentation effect is good, the robustness is good, and the same semantic segmentation effect as the supervised method with high labeling cost can be achieved by using only a small amount of labeled data, which has high application value. SUMMARY

[0005] In order to solve the technical problems mentioned in the background art, a conservative and aggressive collaborative learning based image semi-supervised semantic segmentation method is proposed.

[0006] In order to achieve the above purpose, the present application proposes a conservative and aggressive collaborative learning based image semi-supervised semantic segmentation method, which comprises the following steps:

[0007] Step S1, for the labeled data, a traditional supervised training method is used to train the image semantic segmentation model;

[0008] Step S2, for the unlabeled data, the conservative and aggressive collaborative learning method proposed by the present application is used for semi-supervised training of the model.

[0009] The conservative and aggressive collaborative learning method simultaneously trains two parallel image semantic segmentation branches, and the pseudo labels are generated based on the consensus and divergence of the prediction results of the two branches. Among them, one branch is used for conservative evolution, and the other branch is used for aggressive exploration. The conservative branch is trained using the intersection pseudo supervision proposed by the present application, and the high-quality pseudo labels are used to achieve reliable pseudo supervision training. The aggressive branch is trained using the union pseudo supervision proposed by the present application, and the pseudo labels generated by the union of the prediction results of the two branches are used to achieve the purpose of exploring the divergent part.

[0010] The two parallel image semantic segmentation branches use the same network structure, but use different initialization parameters. During training, two unlabeled image data X1 and X2 are input simultaneously, and a synthetic image X is generated based on the strong data enhancement method using the above two imagess Specifically, taking the conservative branch as an example, the above three images are input into the conservative branch semantic segmentation network Output the corresponding three semantic segmentation prediction results and Y cs .

[0011]

[0012]

[0013]

[0014] For the output and The same data augmentation method as the input is used to obtain the corresponding synthetic prediction results Y cw Similarly, for the aggressive branch, the corresponding prediction results and Y ps , and the corresponding synthetic prediction results Y pw can be obtained. Unlike other inventions that directly cross-pseudo-supervise based on the above results (such as using the synthetic prediction results of the conservative branch to supervise the prediction results of the synthetic images input into the aggressive branch and the synthetic prediction results of the aggressive branch to supervise the prediction results of the synthetic images input into the conservative branch ), the present invention proposes a new pixel-level pseudo-label generation method, which further operates on Y cw and Y pw to obtain pseudo-labels.

[0015] The pixel-level pseudo-label generation method is based on the consensus and divergence of the prediction results of the two branches. For pixel p i , if the corresponding prediction result in Y cw and the corresponding result in Y pw are the same, then the two branches reach a consensus at pixel p i , otherwise the two branches diverge at p i .

[0016]

[0017] wherein is the prediction result of Y cw at pixel p i , and is the corresponding prediction result in Y pw . The pixel-level pseudo-label generation method includes two parts: intersection pseudo-label L inter and union pseudo-label L unionThe intersection pseudo-label is generated only based on the consensus part, and for pixel p i If the two branches reach a consensus, the intersection pseudo-label corresponding to the pixel is the prediction result at the pixel.

[0018]

[0019] wherein is the consensus part pseudo-label L a The value of the intersection pseudo-label L i at pixel p inter is directly generated from the consensus part pseudo-label.

[0020] The union pseudo-label is determined by the consensus part pseudo-label L a and the disagreement part pseudo-label L d Since the prediction results of the two branches of the disagreement part are different, the pseudo-label determination method is relatively complex, and the present application does not use the common determination method based on the prediction confidence threshold, but proposes a pseudo-label determination method based on the class-wise disagreement indicator. The determination method based on the prediction confidence threshold only focuses on the prediction value at the current pixel, which has a certain degree of limitation, and the present application makes a decision based on the entire prediction result at the class level from a relatively macro perspective. Specifically, the pseudo-label determination method based on the class-wise disagreement indicator comprises the following steps:

[0021] Step A1, constructing a consensus matrix based on the prediction results of the two branches wherein C is the number of semantic categories. The specific element value m j,k in the consensus matrix M is the number of pixels that are predicted to be category c cw in Y j and are predicted to be category c pw in Y k .

[0022] Step A2, calculating the class-wise disagreement indicator I based on the consensus matrix M. Wherein, for category j, the corresponding class-wise disagreement indicator is I j ,

[0023]

[0024] wherein j∈[1,C] is the semantic category index.

[0025] Step A3, determining the pseudo-label i at pixel p

[0026]

[0027] The pseudo label determination method based on the category divergence index can be used to determine the pseudo label L of the divergent part. d Determination of the union pseudo label L union The pseudo-label L is obtained from the consensus a and the divergent pseudo labels L d Together constitute.

[0028] L union =L a ∪L d

[0029] In the conservative-aggressive collaborative learning, the conservative branch is trained under the pseudo-supervision of the intersection pseudo-labels, and the aggressive branch is trained under the pseudo-supervision of the union pseudo-labels.

[0030]

[0031]

[0032] The two parallel image semantic segmentation branches in the conservative and radical collaborative learning method are based on the idea of ​​"seeking common ground while reserving differences" during training, achieving the synergy of conservative evolution and radical exploration.

[0033] Considering that noisy pseudo-labels are unavoidable, that is, there will always be wrong labels in the automatically generated pseudo-labels, the present invention further proposes an adaptive dynamic loss function based on prediction confidence to deal with pseudo-labels with low credibility. The loss function reweights the loss function based on prediction confidence. Compared with the common threshold distinction based on prediction confidence, this method does not require a predefined threshold and can make more full use of unlabeled data. i The prediction confidence is defined as Similarly, pixel p in the radical branch i The prediction confidence of Then the dynamic weight based on the prediction confidence can be defined as

[0034]

[0035] Through this dynamic weight, the impact of pseudo labels with strong uncertainty is effectively reduced. Based on this dynamic reweighted loss function, it can be defined as

[0036]

[0037] in and is in pixel p i L inter and L unionThe corresponding pseudo labels, is the input X s The prediction result output by the image semantic segmentation network is at pixel p i The predicted value at , CE(·) is the cross entropy loss function.

[0038] The above-mentioned adaptive dynamic loss function based on prediction confidence is used in the present invention for pseudo-supervised training of unlabeled data. For labeled data, the semantic segmentation network adopts the traditional supervised training method and is trained using the cross-entropy loss function.

[0039]

[0040] where X l ∈D l Indicates labeled data, and G is the corresponding label. In the process of training semi-supervised semantic segmentation of images based on conservative-aggressive collaborative learning, the final training objective is composed of the above two loss functions.

[0041]

[0042] Where γ is the weight value.

[0043] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0044] In the present invention, by using a semi-supervised semantic segmentation method for images based on conservative-radical collaborative learning, the trained semantic segmentation model can effectively and efficiently utilize a large amount of unlabeled data, and achieve synergy between conservative evolution and radical exploration through a semi-supervised approach based on the idea of ​​"seeking common ground while reserving differences." When using the same amount of labeled data, the present invention can effectively improve the image semantic segmentation effect of the model by using a large amount of unlabeled data; in order to achieve the same semantic segmentation effect, the present invention greatly reduces the demand for labeled data, which helps to reduce the cost and increase the efficiency of training the intelligent system environment perception model, and has high practical engineering application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 A schematic diagram of a conservative and radical collaborative learning method framework of an image semi-supervised semantic segmentation method based on conservative and radical collaborative learning is shown according to an embodiment of the present invention;

[0046] Figure 2 A schematic diagram of a consensus matrix of an image semi-supervised semantic segmentation method based on conservative-aggressive collaborative learning provided according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0047] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0048] See also Figures 1-2 The present invention provides a technical solution: a method for semi-supervised semantic segmentation of an image based on conservative-radical collaborative learning, comprising the following steps:

[0049] Step S1: using traditional supervised training methods to train image semantic segmentation models for labeled data;

[0050] Step S2: For unlabeled data, the conservative-aggressive collaborative learning method proposed in the present invention is used to perform semi-supervised training of the model.

[0051] The conservative and aggressive collaborative learning method trains two parallel image semantic segmentation branches at the same time, and pseudo labels are generated based on the consensus and divergence of the prediction results of the two branches. Among them, one branch is used for conservative evolution and the other branch is used for radical exploration. The conservative branch is trained using the intersection pseudo supervision proposed in the present invention, and pseudo supervision training with high reliability is achieved by using high-quality pseudo labels. The radical branch is trained using the union pseudo supervision proposed in the present invention, and the purpose of exploring the divergent parts is achieved by using the pseudo labels generated by the union of the prediction results of the two branches. The framework diagram of the method is shown in FIG. Figure 1 shown.

[0052] The two parallel image semantic segmentation branches use the same network structure but different initialization parameters. In this embodiment, the network structure is Deeplabv3+ using ResNet-50. During training, two unlabeled image data X1 and X2 are input simultaneously, and a composite image X is generated using the two images based on a strong data enhancement method. s In this embodiment, the strong data enhancement method is the CutMix method. Specifically, taking the conservative branch as an example, the three images above are input into the conservative branch semantic segmentation network. Output the corresponding three semantic segmentation prediction results and Y cs .

[0053]

[0054]

[0055]

[0056] For the output and The corresponding synthetic prediction results Y cw are obtained by using the same data augmentation method as the input. ps Similarly, for the aggressive branch, the corresponding prediction results Y and Y pw can be obtained. Unlike other inventions that directly cross-pseudo-supervise based on the above results (such as using the synthetic prediction results of the conservative branch to supervise the prediction results of the synthetic images of the aggressive branch and using the synthetic prediction results of the aggressive branch to supervise the prediction results of the synthetic images of the conservative branch ), the present invention proposes a new pixel-level pseudo-label generation method to further operate Y cw and Y pw to obtain pseudo-labels.

[0057] The pixel-level pseudo-label generation method is based on the consensus and divergence of the prediction results of the two branches. For pixel p i , if the corresponding prediction result in Y cw and the corresponding result in Y pw are the same, then the two branches reach a consensus at pixel p i , otherwise the two branches diverge at p i .

[0058]

[0059] wherein is the prediction result of Y cw at pixel p i , is the corresponding prediction result in Y pw . The pixel-level pseudo-label generation method contains two parts: intersection pseudo-label L inter and union pseudo-label L union . Among them, the intersection pseudo-label is only generated based on the consensus part, for pixel p i , if the two branches reach a consensus, then the intersection pseudo-label corresponding to the pixel is the prediction result at the pixel.

[0060]

[0061] wherein is the value of consensus part pseudo-label L a at pixel p i , and the intersection pseudo-label L inter is directly generated from the consensus part pseudo-label.

[0062] The union pseudo label L a and the disagreement part pseudo label L d are determined jointly. Since the prediction results of the two branches of the disagreement part are different, the pseudo label determination method is relatively complex, and the present application does not use the common determination method based on the prediction confidence threshold, but proposes a pseudo label determination method based on the category divergence index. The determination method based on the prediction confidence threshold only focuses on the prediction value at the current pixel, which has a certain degree of limitation, and the present application makes a decision based on the entire prediction result at the category level from a relatively macro perspective. Specifically, the pseudo label determination method based on the category divergence index comprises the following steps:

[0063] Step A1, constructing a consensus degree matrix based on the prediction results of the two branches where C is the number of semantic categories. The specific element value m j,k in the two-dimensional matrix M is the number of pixels of the same pixel predicted as category c cw in Y j and predicted as category c pw in Y k , and its schematic diagram is shown in Figure 2 .

[0064] Step A2, calculating the category divergence index I based on the consensus degree matrix M. Wherein, for category j, the corresponding category divergence index is I j ,

[0065]

[0066] where j∈[1,C] is the semantic category index.

[0067] Step A3, determining the pseudo label i at pixel p

[0068]

[0069] The above-mentioned pseudo label determination method based on the category divergence index can realize the determination of the disagreement part pseudo label L d , and the union pseudo label L union is composed of the consensus part pseudo label L a and the disagreement part pseudo label L d .

[0070] L union = L a ∪L d

[0071] In the conservative-aggressive collaborative learning, the conservative branch is trained under the pseudo-supervision of the intersection pseudo-labels, and the aggressive branch is trained under the pseudo-supervision of the union pseudo-labels.

[0072]

[0073]

[0074] The two parallel image semantic segmentation branches in the conservative and radical collaborative learning method are based on the idea of ​​"seeking common ground while reserving differences" during training, achieving the synergy of conservative evolution and radical exploration.

[0075] Considering that noisy pseudo-labels are unavoidable, that is, there will always be wrong labels in the automatically generated pseudo-labels, the present invention further proposes an adaptive dynamic loss function based on prediction confidence to deal with pseudo-labels with low credibility. The loss function reweights the loss function based on prediction confidence. Compared with the common threshold distinction based on prediction confidence, this method does not require a predefined threshold and can make more full use of unlabeled data. i The prediction confidence of Similarly, pixel p in the radical branch i The prediction confidence of Then the dynamic weight based on the prediction confidence can be defined as

[0076]

[0077] Through this dynamic weight, the impact of pseudo labels with strong uncertainty is effectively reduced. Based on this dynamic reweighted loss function, it can be defined as

[0078]

[0079] in and is in pixel p i L inter and L union The corresponding pseudo labels, is the input X s The prediction result output by the image semantic segmentation network is at pixel p i The predicted value at , CE(·) is the cross entropy loss function.

[0080] The above-mentioned adaptive dynamic loss function based on prediction confidence is used in the present invention for pseudo-supervised training of unlabeled data. For labeled data, the semantic segmentation network adopts the traditional supervised training method and is trained using the cross-entropy loss function.

[0081]

[0082] wherein X l ∈D l represents the labeled data, and G is the corresponding label. In the process of image semi-supervised semantic segmentation training based on conservative aggressive co-learning, the final training target is composed of the above two loss functions,

[0083]

[0084] wherein gamma is a weight value.

[0085] In a specific implementation, the SGD optimizer with momentum is used to iteratively optimize the learning network parameters, wherein the momentum is set to 0.9. In the present embodiment, the initial learning rate is set to 10 -4 , and is gradually reduced according to with the increase of the iteration number.

[0086] The term "comprising" or any other similar word is intended to encompass non-exclusive inclusion, so that a process, method, article or equipment / device including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or also includes elements inherent to the process, method, article or equipment / device.

[0087] The above description is only the preferred specific implementation of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art within the technical range disclosed by the present application, according to the technical solution and the inventive concept of the present application, makes equivalent replacement or change, should be covered within the protection scope of the present application.

Claims

1. A method for semi-supervised semantic segmentation of images based on conservative-radical collaborative learning, characterized by: The following steps are involved: For labeled data, the traditional supervised training method is used to train the image semantic segmentation model; for unlabeled data, the conservative and radical collaborative learning method is used to perform semi-supervised training of the model; The conservative-aggressive collaborative learning method adopts traditional supervised training when using labeled data, and adopts an adaptive dynamic loss function based on prediction confidence to deal with low-confidence pseudo labels when using unlabeled data for pseudo-supervised training. For pixels The dynamic weight based on the prediction confidence can be defined as : ； in is the pixel in the conservative branch The prediction confidence of is the corresponding prediction confidence in the radical branch, Refers to the pseudo label l of the divergent part d The pseudo label value at pixel i is the predicted value of the conservative branch at the corresponding pixel position Sure, Refers to the pseudo label l of the divergent part d The pseudo label value at pixel i is the predicted value of the radical branch at the corresponding pixel position Determine, based on this dynamic reweighted loss function can be defined as ; ； Among them, W and H are the width and height of the image to be segmented respectively. and yes and In pixels The corresponding pseudo-labels are is input The prediction results output by the image semantic segmentation network are in pixels The predicted value at It is the cross entropy loss function. The input value of the cross entropy loss function consists of two parts. The first part is the label value and the second part is the predicted value. For intersection pseudo supervision and union pseudo supervision, the first part corresponds to the intersection pseudo label. and union pseudo labels , and the second part corresponds to the conservative branch prediction value and aggressive branch prediction values .

2. The image semi-supervised semantic segmentation method based on conservative-radical collaborative learning according to claim 1 is characterized in that: The conservative-aggressive collaborative learning method simultaneously trains two parallel branches for image semantic segmentation, and the pixel-level pseudo-label generation method is based on the consensus and disagreement of the prediction results of the two branches; one branch is used for conservative evolution and the other for aggressive exploration; The conservative branch is trained using intersection pseudo-supervision and pseudo-labels, and the radical branch is trained using union pseudo-supervision and pseudo-labels generated by the union of the prediction results of the two branches to achieve the purpose of exploring the divergent parts.

3. The image semi-supervised semantic segmentation method based on conservative-radical collaborative learning according to claim 1 is characterized in that: It is based on the consensus and disagreement of the prediction results of the two branches, and consists of two parts: intersection pseudo-label and union pseudo labels , where the intersection pseudo-label is generated only based on the consensus part, and the union pseudo-label Pseudo-labeling by consensus and the divergent pseudo labels Together, for pixels ,if The corresponding prediction results and The corresponding results in , then the two branches have the same value in pixels Otherwise, the two branches will reach a consensus at Disagreements arise; ； in yes In pixels The prediction results at yes If the two branches reach a consensus, the intersection pseudo label corresponding to the pixel is the prediction result at the pixel. represents the synthetic prediction result of the conservative branch, represents the synthetic prediction result of the radical branch; ； in It is the consensus part pseudo label In pixels The value at intersection pseudo label Directly generated from the consensus pseudo-labels; If the two branches diverge, the pseudo-label determination method based on the category divergence index is used to make a decision on the pseudo-label of the divergent part based on the entire prediction result at the category level.

4. The image semi-supervised semantic segmentation method based on conservative-radical collaborative learning according to claim 3 is characterized in that: The pseudo-label determination method based on the category divergence index comprises the following steps: Step A1: Construct a consensus matrix based on the prediction results of the two branches ,in is the number of semantic categories, the consensus matrix The specific element value in For the same pixel Predicted as category And in Predicted as category The number of pixels; Step A2, based on the consensus matrix Calculate the category divergence index , where for the category , and its corresponding category divergence index is , ； in is the semantic category subscript; Step A3, based on category divergence index Determine the pixel Pseudo labels at , ； The pseudo label determination method based on the category divergence index can be used to determine the pseudo labels of the divergent parts. Determination of the pseudo label Pseudo-labeling by consensus and the divergent pseudo labels together constitute; ； Representation category The corresponding category divergence index.

5. The method for semi-supervised semantic segmentation of images based on conservative-radical collaborative learning according to claim 4, characterized in that: It uses pixel-level pseudo-label generation method to generate intersection pseudo-labels and performs pseudo-supervision based on them; ； Operator Refers to the supervision operation, which means that the conservative branch is pseudo-supervised by the generated intersection pseudo-label, that is, the loss function of the pseudo-supervisory training of this branch is composed of the difference between the intersection pseudo-label and its predicted value; Represents the result obtained by image semantic segmentation prediction in the conservative branch model.

6. The method for semi-supervised semantic segmentation of images based on conservative-radical collaborative learning according to claim 5, characterized in that: It uses pixel-level pseudo-label generation method to generate union pseudo-labels and performs pseudo-supervision based on them; ； Operator Refers to the supervision operation, which means that the radical branch is pseudo-supervised by the generated union pseudo-label. That is, the loss function of the pseudo-supervisory training of this branch is composed of the difference between the union pseudo-label and its predicted value; Represents the result obtained by image semantic segmentation prediction in the radical branch model.

Images