Knowledge fusion assistance-based semi-supervised incremental semantic segmentation method

Abstract

The invention discloses a semi-supervised class increment semantic segmentation method based on knowledge fusion assistance, belongs to the technical field of image segmentation methods, and adopts the technical scheme that the method comprises the following steps: S1, defining a semi-supervised class increment semantic segmentation method; a double-teacher structure is adopted to generate pseudo labels for the unmarked data; and S3, simultaneously training models on the marked data and the unmarked data, and carrying out semantic segmentation by utilizing the trained models. According to the method, the dependence on the annotated data is reduced by using the information in the unannotated data, so that the generalization ability and the new category adaptation ability of the model are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of image segmentation methods, and provides a semi-supervised incremental semantic segmentation method based on knowledge fusion assistance. Background Art

[0002] In recent years, although deep learning has made significant progress in the field of image recognition, most models are still trained and tested based on static or stable data distributions. However, in the real world, data changes dynamically, which leads to the problem that old knowledge is easily forgotten when learning new information, the so-called "catastrophic forgetting" phenomenon. In order to deal with this problem, incremental learning strategies have been proposed, aiming to maintain the memory of previous information while adapting to new knowledge. In particular, in classification tasks, incremental learning is particularly important.

[0003] This challenge becomes more complex in the class incremental semantic segmentation (CISS) task. CISS not only requires the model to accurately classify at the pixel level, but also has to deal with the problem that the background may be redefined when new categories are introduced. Due to the change of background, the model is more likely to forget old categories when learning new categories. Therefore, developing strategies that can retain knowledge of old categories when learning new categories has become a key research direction. To address these challenges, researchers have proposed a variety of strategies. For example, MiB handles the CISS task by reconstructing virtual label backgrounds and distillation-based methods. PLOP extends MiB, defines pseudo labels and adopts local distillation technology. SSUL first introduced a replay strategy, maintained a memory library of historical samples, and froze some model parameters to prevent forgetting. RCIL adopts a dual-branch architecture with channel feature distillation loss. MicroSeg solves the background shift problem through a mask proposal mechanism. CoinSeg improves the stability and plasticity of the model through flexible tuning strategies. These methods have made significant progress in alleviating catastrophic forgetting and improving segmentation performance. However, the current incremental learning paradigm still cannot do without a certain scale of labeled data. In many practical tasks, it is very challenging to obtain large-scale, high-quality labeled data to achieve a strong generalization model. Therefore, using large-scale unlabeled data to alleviate the dependence on a large amount of labeled data in CISS has become an important research direction. Summary of the invention

[0004] The present invention solves the problems existing in the prior art and provides a semi-supervised incremental semantic segmentation method based on knowledge fusion assistance. The present invention utilizes information in unlabeled data to reduce the dependence on labeled data, thereby improving the generalization ability of the model and the ability to adapt to new categories.

[0005] The present invention is implemented by the following technical solution: a semi-supervised incremental semantic segmentation method based on knowledge fusion assistance, comprising the following steps:

[0006] Step S1: Define a semi-supervised class-incremental semantic segmentation method:

[0007] make and denote the sub-dataset of the t-th step of incremental learning and its corresponding category set, where t=1,…,T denotes a series of incremental learning steps, (x t ,y t ) represents the input image of the labeled data and its corresponding true label, u t Input image representing unlabeled data;

[0008] make Represents all categories of the current stage, where represents all visible categories that need to be learned in the tth step, represents the category learned before step t, c u Indicates that the category is not seen;

[0009] Step S2: construct a semi-supervised class-incremental semantic segmentation framework and adopt a dual-teacher structure to generate pseudo labels for unlabeled data;

[0010] Step S3: Train the model on the labeled data and the unlabeled data simultaneously, and use the trained model to perform semantic segmentation.

[0011] Furthermore, the specific generation process of generating pseudo labels from unlabeled data in step S2 is as follows:

[0012] Taking the t-th step learning as an example, the first teacher model Perform parameter updates to generate pseudo labels for unlabeled data and its corresponding confidence map ρ t , this process is expressed as:

[0013]

[0014] Where σ(·) is the Sigmoid function, and the teacher model Through the student model The exponential moving average of the weights is gradually updated to keep pace with the student model. This process can be expressed as:

[0015] θ t ←αθ t +(1-α)θ s (2)

[0016] Where α is a commonly used momentum parameter, the default value is 0.999;

[0017] The second teacher model Freeze the parameters and set them as the parameters of the first teacher model at step t-1 to generate pseudo labels for unlabeled data and its corresponding confidence map ρ t-1 , this process is expressed as:

[0018]

[0019] The second teacher model It also generates pseudo labels for the labeled data and consolidates the information of old categories. Finally, the pseudo labels generated by the two teacher networks are fused through the knowledge fusion module KFM to form comprehensive pseudo labels.

[0020] Furthermore, the specific operations of the knowledge fusion module KFM are:

[0021] When the teacher model generates credible predictions Pseudo labels for unlabeled data when it is not an all-zero label map The fusion result is expressed as:

[0022]

[0023] The threshold τ 2 =0.7, ^ represents the "AND" logic operation;

[0024] Otherwise, the pseudo labels for the unlabeled data The fusion result is still a full zero label map, which is expressed as follows:

[0025]

[0026] Furthermore, in step S3, when the model is trained on both labeled data and unlabeled data, the total training loss It is expressed as:

[0027]

[0028] in, represents the unsupervised loss calculated based on unlabeled data, represents the supervision loss calculated based on the labeled data, Follow and The same setting, with only slight differences in the input, λ u =1 is the weighted coefficient that balances the contributions of the two losses;

[0029] Unsupervised loss It is expressed as:

[0030]

[0031] in is the contrast loss, p and q are two prediction distributions from different models, with dimension N, It represents the segmentation result of the student model on the strongly enhanced unlabeled image. Indicates the strong enhancement method used, that is, randomly using no more than 3 methods from the commonly used strong enhancement methods for enhancement, express Multiply it with the feature map of stage t and then smooth it. express Multiply it with the feature map of stage t-1 and then smooth it. represents the total number of categories, Denotes the mask proposal B∈{0,1} N×H×W Multiply it with the feature map of stage t and then smooth it. represents the multiplication of the mask proposal and the feature map of the t-1 stage and then smoothed, λ c =10 -2 .

[0032] Compared with the prior art, the present invention has the following advantages:

[0033] 1. This paper applies semi-supervised learning to class-incremental semantic segmentation for the first time, and improves the model's ability to recognize new categories by effectively utilizing unlabeled data.

[0034] 2. The present invention proposes an innovative knowledge fusion module KFM, which successfully solves the challenges faced by unlabeled data in the pseudo-label fusion process.

[0035] 3. Experimental results show that on the Pascal-VOC 2012 dataset, the method of the present invention uses only one-fourth of the labeled data, and its performance is still comparable to the current most advanced methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 Schematic diagram of the semi-supervised incremental semantic segmentation framework assisted by knowledge fusion.

[0037] Figure 2 It is a comparative data diagram of the present invention and different solutions.

[0038] Figure 3 It is a comparative data diagram under different data ratios of the present invention.

[0039] Figure 4 This is a diagram of the ablation experiment data of the KFM module of the present invention.

[0040] Figure 5 This is a diagram of ablation experiment data of the parameters of the present invention.

[0041] Figure 6A comparative visualization diagram of the present invention and the CoinSeg method. DETAILED DESCRIPTION

[0042] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0043] like Figure 1 The semi-supervised incremental semantic segmentation method based on knowledge fusion assistance shown in the figure includes the following steps:

[0044] Step S1, defining a semi-supervised class-incremental semantic segmentation method;

[0045] make and denote the sub-dataset of the t-th step of incremental learning and its corresponding category set, where t=1,…,T denotes a series of incremental learning steps, (x t ,y t ) represents the input image of the labeled data and its corresponding true label, u t denotes the input image of the unlabeled data. Similarly, let Represents all categories of the current stage, where represents all visible categories that need to be learned in the tth step, represents the category learned before step t, c u Indicates that the category was not seen.

[0046] Step S2: construct a semi-supervised class-incremental semantic segmentation framework;

[0047] like Figure 1 As shown in Figure 2, the constructed framework uses a dual teacher structure to generate pseudo labels for unlabeled data. Taking the t-th step learning as an example, the first teacher model Parameter updates are needed to generate pseudo labels for unlabeled data and its corresponding confidence map ρ t , this process is expressed as:

[0048]

[0049] Where σ(·) is the Sigmoid function. Teacher model Through the student model The exponential moving average of the weights is gradually updated to keep pace with the student model. This process can be expressed as:

[0050] θ t ←αθ t +(1-α)θ s (2)

[0051] Where α is a commonly used momentum parameter, which defaults to 0.999. The second teacher model Parameter freezing is required and is set to the parameters of the first teacher model at step t-1 to generate pseudo labels for unlabeled data. and its corresponding confidence map ρ t-1 , this process is expressed as:

[0052]

[0053] In addition, the second teacher model It also generates pseudo labels for the labeled data and consolidates the information of old categories. Finally, the pseudo labels generated by the two teacher networks are fused through the knowledge fusion module KFM to form a comprehensive pseudo label.

[0054] Pseudo Labels The screening process is as follows:

[0055] When pseudo labels Does not belong to the background category or the future category and its confidence exceeds the threshold τ 1 When , keep the predicted value; otherwise, set it to 0. This process can be expressed as:

[0056]

[0057] The threshold τ 1 =0.9, ^ represents the “AND” logical operation.

[0058] The specific operations of the knowledge fusion module KFM are:

[0059] When the teacher model generates credible predictions Pseudo labels for unlabeled data when it is not an all-zero label map The fusion result can be expressed as:

[0060]

[0061] The threshold τ 2 =0.7, ^ represents the “AND” logical operation.

[0062] Otherwise, the pseudo labels for the unlabeled data The fusion result is still a full zero label map, which is expressed as follows:

[0063]

[0064] Step 3): Train the model on both labeled data and unlabeled data, and use the trained model for semantic segmentation; Figure 1 As shown, the total training loss It is expressed as:

[0065]

[0066] in, represents the unsupervised loss calculated based on unlabeled data, represents the supervision loss calculated based on the labeled data, Follow and The same setting, with only slight differences in the input, λ u = 1 is the weighted coefficient that balances the contribution of the two losses. Unsupervised loss It is expressed as:

[0067]

[0068] in For contrast loss, p and q are two prediction distributions from different models with dimension N. It represents the segmentation result of the student model on the strongly enhanced unlabeled image. represents the strong enhancement method used in the present invention, that is, randomly adopting no more than 3 methods from the commonly used strong enhancement methods for enhancement, express Multiply it with the feature map of stage t and then smooth it. Similarly, express Multiply it with the feature map of stage t-1 and then smooth it. Indicates the total number of categories. Denotes the mask proposal B∈{0,1} N×H×W Multiply it with the feature map of stage t and then smooth it. Similarly, represents the multiplication of the mask proposal and the feature map of the t-1 stage and then smoothed, λ c =10 -2 .

[0069] The following is a simulation experiment on the knowledge fusion-assisted semi-supervised incremental semantic segmentation method of the present invention.

[0070] Experimental setup

[0071] We evaluated the proposed method on the Pascal VOC 2012 dataset and conducted experiments under different incremental learning settings, including 15-5, 15-1, and 10-1 scenarios. For each incremental scenario, experiments were conducted on data ratios of 1 / 16, 1 / 8, 1 / 4, and 1 / 2, respectively.

[0072] DeepLabV3+ and Swin Transformerbase (Swin-b) backbone network are used, pre-trained on ImageNet, and the window size of Swin Transformer is 12. Each training mini-batch contains 16 images, including 8 labeled images and 8 unlabeled images. The initial learning rate is set to 0.0001 using the ADAMW optimizer, and the model is trained 50 times using a multi-learning rate scheduler.

[0073] To ensure a fair comparison with previous methods, the same class-agnostic mask proposals are applied in the experiments. Specifically, the mask proposals are generated using a fixed-parameter Mask2Former pre-trained on MS-COCO, with N being 100 for all experiments. To prevent information leakage, Mask2Former is not fine-tuned on any benchmark datasets.

[0074] The mean intersection over union (mIoU) metric, which is widely used in semantic segmentation, is used for evaluation to quantify the overlap between the predicted segmentation and the true segmentation.

[0075] Comparison with state-of-the-art methods

[0076] like Figure 2 As shown in Figure 2, the performance of the method under different data ratios is compared with the previous method using the full amount of labeled data. Although only half or 1 / 4 of the labeled data is used, the method of the present invention performs well in all incremental scenarios.

[0077] In particular, in the 15-5 scenario, the method of the present invention outperforms the CoinSeg method using only 1 / 4 of the labeled data, and even when using 1 / 2 of the labeled data, the method of the present invention brings a 1% performance gap. In addition, focusing on the long-term scenarios (i.e., 15-1 and 10-1), the performance of the method of the present invention is comparable to that of CoinSeg even when using only 1 / 4 or 1 / 2 of the labeled data. Methods such as CoinSeg require a large amount of fully labeled data, but in real scenarios, labeled data is more difficult to obtain.

[0078] also, Figure 3 The performance of the three incremental learning scenarios with a smaller proportion of labeled data is described, where the three dashed lines from top to bottom are the SOTA in the 15-5 scenario, 15-1 scenario, and 10-1 scenario. Even with 1 / 8 or even 1 / 16 of the labeled data, the method of the present invention still has good robustness.

[0079] Ablation experiment

[0080] We evaluate the KFM module and the hyperparameter τ 1effectiveness.

[0081] Effectiveness of KFM Module

[0082] like Figure 4 As shown in Figure 2, a detailed experimental analysis of the impact of the KFM module is conducted, focusing on its performance changes under different data ratios. Through experiments conducted in VOC15-1 and VOC10-1 incremental learning scenarios, it can be seen that the KFM module has made significant contributions in enhancing the generalization ability of the model and improving the recognition performance of new categories.

[0083] First, in the VOC15-1 scenario, the results show that when the labeled data ratio is 1 / 8, the performance of incremental categories 16-20 is improved from 42.4 to 56.4, and the overall performance is improved from 69.7 to 73.4 after the introduction of the KFM module. Similarly, in the VOC10-1 scenario, when the labeled data ratio is 1 / 4, the performance of categories 11-20 is improved from 53.6 to 60.3 after the use of the KFM module, and the overall performance is improved by 3.5 points (from 66.5 to 70.0).

[0084] Hyperparameter τ 1 Effectiveness

[0085] like Figure 5 As shown, for the parameter τ 1 We conduct ablation studies on different values ​​of to evaluate its impact on the performance of the class-incremental semantic segmentation task. According to the results in the table, when τ 1 When τ is set to 0.9, the performance of the model on 0-10 categories, 11-20 categories, and all categories is 78.8, 60.3, and 70.0, respectively, which is the best choice for overall performance. Therefore, in subsequent experiments, τ 1 =0.9 as the optimal parameter value. In the experiment, the parameter τ 2 = 0.7 was not studied independently. This is because we followed the recommended setting of CoinSeg. This choice was based on the validation results in CoinSeg experiments, which have been shown to provide stable and good performance on similar tasks. Since this parameter has been well validated in existing studies, we prioritize other parameters (such as τ 1 ) to further optimize the model’s performance on our specific task.

[0086] Visual Comparison

[0087] like Figure 6 As shown, in order to better understand the correction process in the present invention, we intuitively display the visual experimental results of the performance of each stage in the 10-1 task compared with CoinSeg.

[0088] The first picture shows that the method of the present invention always accurately identifies all cows as "cows", while the CoinSeg method misidentifies the rightmost cow as "horse" in the 4th stage and misidentifies the middle cow as "sheep" in the 8th stage, which reflects the advantage of the method of the present invention in maintaining the accuracy of old categories. In the second picture, both methods successfully identify "potted plants" in the 7th stage, while CoinSeg mistakenly identifies the glass bottle as "potted plants" in the last stage and fails to detect the real target, showing the excellent recognition ability of our method in details. In the last picture, the method of the present invention accurately identifies "display", while CoinSeg fails to complete this task, further demonstrating the excellent performance of the method of the present invention in identifying new categories.

[0089] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A semi-supervised incremental semantic segmentation method based on knowledge fusion assistance, characterized in that: The following steps are involved: Step S1: Define a semi-supervised class-incremental semantic segmentation method: make and denote the sub-dataset of the t-th step of incremental learning and its corresponding category set, where t=1,…,T denotes a series of incremental learning steps, (x t ,y t ) represents the input image of the labeled data and its corresponding true label, u t Input image representing unlabeled data; make Represents all categories of the current stage, where represents all visible categories that need to be learned in the tth step, represents the category learned before step t, c u Indicates that the category has not been seen; Step S2: construct a semi-supervised class-incremental semantic segmentation framework, using a dual-teacher structure to generate pseudo labels for unlabeled data; Step S3: Train the model on the labeled data and the unlabeled data simultaneously, and use the trained model to perform semantic segmentation.

2. The knowledge fusion-assisted semi-supervised incremental semantic segmentation method according to claim 1, characterized in that: The specific generation process of generating pseudo labels from unlabeled data in step S2 is as follows: Taking the t-th step learning as an example, the first teacher model Perform parameter updates to generate pseudo labels for unlabeled data and its corresponding confidence map ρ t , this process is expressed as: Where σ(·) is the Sigmoid function, and the teacher model Through the student model The exponential moving average of the weights is gradually updated to keep pace with the student model. This process can be expressed as: θ t ←αθ t +(1-α)θ s (2) where α is a commonly used momentum parameter, which defaults to 0.999; The second teacher model Freeze the parameters and set them as the parameters of the first teacher model at step t-1 to generate pseudo labels for unlabeled data and its corresponding confidence map ρ t-1 , this process is expressed as: The second teacher model It also generates pseudo labels for the labeled data and consolidates the information of old categories. Finally, the pseudo labels generated by the two teacher networks are fused through the knowledge fusion module KFM to form comprehensive pseudo labels.

3. The knowledge fusion-assisted semi-supervised incremental semantic segmentation method according to claim 2, characterized in that: The specific operation of the knowledge fusion module KFM is: When the teacher model generates credible predictions Pseudo labels for unlabeled data when it is not an all-zero label map The fusion result is expressed as: Wherein the threshold value τ2 = 0.7, ^ represents the "AND" logic operation; Otherwise, the pseudo labels for the unlabeled data The fusion result is still a full zero label map, which is expressed as follows:

4. The knowledge fusion-assisted semi-supervised incremental semantic segmentation method according to claim 3 is characterized by: In step S3, when the model is trained on labeled data and unlabeled data at the same time, the total training loss It is expressed as: in, represents the unsupervised loss calculated based on unlabeled data, represents the supervision loss calculated based on the labeled data, Follow and The same setting, u =1 is the weighted coefficient that balances the contributions of the two losses; Unsupervised loss It is expressed as: in is the contrast loss, p and q are two prediction distributions from different models, with dimension N, It represents the segmentation result of the student model on the strongly enhanced unlabeled image. Indicates the strong enhancement method used, that is, randomly using no more than 3 methods from the commonly used strong enhancement methods for enhancement, express Multiply it with the feature map of stage t and then smooth it. express Multiply it with the feature map of stage t-1 and then smooth it. represents the total number of categories, Denotes the mask proposal B∈{0,1} N×H×W Multiply it with the feature map of stage t and then smooth it. represents the multiplication of the mask proposal and the feature map of the t-1 stage and then smoothed, H and W represent the height and width of the image respectively, and λ c =10 -2 .

Images