A class-incremental semantic segmentation method based on dynamic knowledge distillation

Abstract

This invention discloses a class incremental semantic segmentation method based on dynamic knowledge distillation. The class incremental semantic segmentation method is based on a two-layer optimization framework of learnable knowledge distillation. The two-layer optimization framework includes a frozen old model, a distillation scheme optimization module, a trainable new model, and a dynamic weight balancing module. The distillation scheme optimization model obtains the optimal distillation scheme for each output by greedily searching a set of distillation candidates according to the initialized weights and iterating every T / N periods. The optimal distillation scheme includes the objective function of the original dynamic weight balancing module to update the weights in each mini-batch iteration. This invention learns the optimal distillation scheme and weights in an iterative manner, aiming to solve the catastrophic forgetting and background class shift problems faced by CISS.

Description

Technical fields:

[0001] This invention belongs to the field of image processing, and particularly relates to a class incremental semantic segmentation method based on dynamic knowledge distillation. Background technology:

[0002] Unlike traditional semantic segmentation [1,2], class incremental semantic segmentation (CISS) [3] is geared towards open-world scenarios and is applied to many fields in the real world, including robot navigation, autonomous driving, medical diagnosis, etc. In recent years, people have shown great interest in CISS. The biggest challenge facing CISS is catastrophic forgetting, and knowledge distillation [4] technology has been widely used to solve this challenge. Knowledge distillation focuses on improving the performance of small student models by utilizing large pre-trained models. In recent years, CISS architectures have emerged one after another, among which the three mainstream trends are initializing classifiers [5,6], multi-scale spatial knowledge distillation [7], and introducing knowledge distillation loss based on contrastive learning [8]. In addition, existing methods [9] propose weighted knowledge distillation based on inter-class similarity. In addition, some studies focus on generating more accurate pseudo-labels [7,10], using replay techniques [5,11], increasing model capacity

[12] , etc. Among them, some dynamic knowledge distillation methods

[13] learn the weights of distillation loss by utilizing the prediction relationship between teacher models and student models to perform saliency detection.

[14] Dynamically select teacher models and data for pre-trained language models.

[0003] To address the catastrophic forgetting problem faced by CISS (Content-Based Scraping System), most existing research employs a knowledge distillation-based framework. However, despite significantly improving CISS performance, it still faces a stability-plasticity dilemma: rigidity refers to the ability to retain old knowledge, while plasticity refers to the ability to learn new knowledge. First, due to semantic transfer in the background class—where background pixels may include both previously learned and future foreground classes—current knowledge distillation methods cause model confusion, preventing the correct distinction between foreground and background classes. Second, existing methods apply the same distillation approach to different intermediate layer features and predicted outputs. However, deep models at different stages possess different properties; shallow features capture low-level local information, while deep features capture high-level semantic information. Therefore, the distillation effects and information differ between different outputs, and appropriate distillation schemes should be adopted for different network outputs. Finally, trade-off weights have a significant impact on CISS performance; suitable trade-off weights play a crucial role in balancing the model's ability to learn new knowledge (plasticity) and retain old knowledge (stability). Existing methods manually perform trade-off weight adjustments, and the parameters are fixed during model training, which limits the model's performance under varying scenarios and samples. Therefore, these limitations prevent existing knowledge distillation-based CISS methods from achieving a fully ideal balance between stability and plasticity. Summary of the Invention:

[0004] To overcome the shortcomings of existing methods and technologies, this invention proposes a class-incremental semantic segmentation method based on dynamic knowledge distillation, which addresses the stability-plasticity dilemma in a more flexible way and further improves the performance of CISS. Specifically, LKD learns dynamic trade-off weights while searching for the optimal distillation scheme for different outputs, achieving an adaptive trade-off between stability and plasticity.

[0005] To equip appropriate distillation schemes for different outputs, this invention constructs a set of distillation scheme candidates, specifically proposing two effective KD schemes to mitigate background offset and utilize richer contextual information. Furthermore, this invention employs a regularized gradient balancing formula to learn dynamic trade-off weights. Based on the above methods, LKD iteratively learns the optimal distillation scheme and weights.

[0006] The present invention solves its practical problem by adopting the following technical solution:

[0007] An incremental semantic segmentation method based on dynamic knowledge distillation is proposed. This method is based on a two-layer optimization framework of learnable knowledge distillation, which includes a frozen old model, a distillation scheme optimization model, a trainable new model, and a dynamic weight balancing model, wherein:

[0008] The preferred model for the distillation scheme is based on initialization. The weights are calculated by iterating through a set of distillation candidates every T / N periods using a greedy search to obtain the optimal distillation scheme for each output, where the optimal distillation scheme includes: Local POD. MD MCD And the original KD

[0009] The dynamic weight balancing module updates the weights during each mini-batch iteration using the following objective function. Right now:

[0010]

[0011] in, yes The mean, and H CE and H Lgt respectively by and It is calculated that W is the parameter of the layer preceding the classification layer and c is the ratio of the number of new categories to the number of old categories. α l and β o The relative reverse training rates are respectively.

[0012] Furthermore, the distillation scheme optimization model obtains the optimal distillation scheme MD through a mask. process:

[0013] The optimal distillation scheme MD is generated according to the following loss function.

[0014]

[0015] in: Features after mask You can also mask by scaling GT. The background category label is set to 0, and the foreground category label is set to 1.

[0016] Furthermore, the frozen old model for the optimal distillation scheme MD Perform background semantic transfer process:

[0017] In the t-th step, the original image I is masked with the background class to obtain the image I. + Image I obtained after masking the foreground class - Feed to the old model f t-1 In the process, the corresponding outputs of the l-th layer are obtained respectively. and

[0018] The region least affected by the background class is calculated using the following formula:

[0019]

[0020] in, It is along the channel dimension The result obtained after average pooling; ε and ξ are two boundary values; |U| ＜ε and |U| ＞ξ Let represent the elements in U that are less than ε and greater than ξ, respectively; and Eliminate regions that are significantly affected by the background class and find regions that are less affected by the background class, respectively.

[0021] The two regions above are merged using the union operation U;

[0022] Where: for a given region if We set To generate Furthermore, the distillation scheme optimization model obtains the optimal distillation scheme MCD through a mask. process:

[0023] Mask-guided context distillation is used to extract global contextual information and richer feature statistics.

[0024] right Perform context pooling, perform spatial average pooling; and... Perform channel segmentation to generate a set of features and

[0025] Use global second-order pooling to capture Chinese statistical information;

[0026] The following optimal distillation scheme (MCD) is generated according to the following loss function.

[0027]

[0028] Where G is the number of channel segmentation groups; through the introduction of context pooling, MCD can extract the inherent global and rich statistical information in the features.

[0029] Beneficial effects:

[0030] This invention proposes a two-layer optimization framework based on learnable knowledge distillation, which iteratively learns the optimal distillation scheme and weights, aiming to solve the catastrophic forgetting and background class shift problems faced by CISS. Extensive experiments clearly demonstrate the effectiveness of our proposed method, outperforming the current state-of-the-art methods.

[0031] In summary, this invention has the following two key inventive points:

[0032] (1) This invention is the first attempt to develop a learnable knowledge distillation (LKD) framework to address the stability-plasticity dilemma in the CISS task. The core of LKD is a two-layer optimization framework for searching distillation schemes and simultaneously learning dynamic trade-off weights to balance stability and plasticity in a more flexible way.

[0033] (2) This invention constructs a set of distillation candidate schemes, among which two effective distillation schemes (i.e., MD and MCD) are proposed to mitigate background offset and utilize richer contextual information. In addition, regularized gradient balancing is established to learn dynamic tradeoff weights. Attached image description:

[0034] Figure 1 This is a framework diagram of a class incremental semantic segmentation method based on dynamic knowledge distillation according to the present invention;

[0035] Figure 2 This invention relates to a schematic diagram of the mask region in MD calculation;

[0036] Figure 3 This invention relates to the feature A schematic diagram of context distillation. Detailed Implementation

[0037] The present invention will now be described in detail with reference to the accompanying drawings:

[0038] like Figure 1 As shown, this invention provides a class-incremental semantic segmentation method based on dynamic knowledge distillation. This method is based on a two-layer optimization model, which simultaneously searches for distillation schemes and learns dynamic trade-off weights during training, thus more flexibly addressing the stability-plasticity dilemma.

[0039] Specifically, the present invention provides a class incremental semantic segmentation method based on dynamic knowledge distillation, comprising the following steps:

[0040] [Step I]: Fix the trade-off weights. The two-layer optimization model performs a greedy search on a set of distillation scheme candidates to equip the best distillation scheme for different outputs. In particular, the present invention proposes two effective distillation schemes, namely mask-guided distillation (MD) and mask-guided context distillation (MCD), in which MD mitigates background offset by using features of the mask affected by background class semantic transfer, and MD is constructed based on multi-scale pooling distillation [7] (LocalPOD). In order to explore and utilize high-level semantic information in depth, MCD introduces global context pooling to extract global second-order statistical information of mask features. Therefore, the distillation candidate scheme of the present invention consists of the proposed MD and MCD, as well as the previous LocalPOD [7] and the original KD

[15] .

[0041] [Step II]: With the distillation scheme fixed, the bilayer optimization model, inspired by the gradient normalization method

[16] (GradNorm), formulates and minimizes the objective function on the regularized gradient balancing module (RGE) to learn the dynamic trade-off weights of cross-entropy loss and various KD losses. The bilayer optimization model performs the above two steps iteratively, as summarized below. Figure 1 As shown.

[0042] This invention proposes a candidate MD module and an MCD module for distillation schemes, and then proposes a formula for regularized gradient balancing to learn dynamic trade-off weights.

[0043] 2. Specific Plan

[0044] 2.1 Candidate Distillation Schemes

[0045] Since different outputs have different properties, they may require different distillation schemes. Inspired by different pooled distillation

[17] (POD) and pyramidal structures[18,19], the knowledge distillation loss of LocalPOD was proposed to solve the CISS problem. In particular, let's assume that... This represents the output of the CISS model in stage l, where W, H, and C represent... The width, height, and number of channels of the l-th layer. Then the output features of the l-th layer... The LocalPOD loss can be expressed as:

[0046]

[0047] Where L is the total number of layers in the CISS model. and These are the model's outputs at step (t-1) and step t, respectively. Specifically, This indicates the use of spatial pyramids with S scales. Perform multi-scale pooling operations to generate a set of connected features. For each Specific regions are pooled along both the horizontal and vertical directions. Finally... This forms a ((W+H)×C×S) dimensional feature representation. Therefore, the LocalPOD method can simultaneously capture... The long-range and short-range spatial relationships.

[0048] 2.1.2 Mask-guided distillation (MD)

[0049] As is well known, the semantics of the background class evolves with each step of continuous learning. As shown in Equation 1, LocalPOD distillation includes all features, including the current foreground and background classes. However, changing background classes can confuse the model's learning [7,15]. Therefore, this invention proposes a mask-guided distillation scheme that distills only features less affected by background class shifts. Specifically, our MD loss based on LocalPOD is represented as follows:

[0050]

[0051] in This indicates the features following the mask. You can also mask by scaling GT. We then set the background class label to 0 and the foreground class label to 1. However, the background class influences feature learning layer by layer, which cannot be solved by directly masking the intermediate layer features.

[0052] To address the aforementioned problems, this invention calculates the difference between the features obtained before and after masking the input image. like Figure 2 As shown, in the t-th step, the present invention respectively transforms the original image I and the image I obtained after masking the background class. + Image I obtained after masking the foreground class - Feed to the old model f t-1 In (θ), the corresponding outputs of the l-th layer are obtained respectively. and This invention calculates the region less affected by the background class using the following formula:

[0053]

[0054] in, It is along the channel dimension The result obtained after average pooling. ε and ξ are two boundary values. |U|＜ε and |U| ＞ξ Let represent the elements in U that are less than ε and greater than ξ, respectively. and Regions heavily influenced by the background class are eliminated separately, while regions less influenced by the background class are found. Then, these two regions are merged using a union operation U. For a given region... if We set To generate Therefore, our MD has the ability to mitigate the impact of context class semantic transfer.

[0055] 2.1.3 Mask-Guided Context Distillation (MCD)

[0056] Both LocalPOD and MD perform pooling summation along both horizontal and vertical directions. However, neither horizontal nor vertical pooling can fully simulate the global contextual information of features. Furthermore, pooling summation only computes first-order statistics, failing to capture the rich statistical information inherent in features.

[0057] Previous work has shown that global contextual information and rich statistical information are crucial for representing features, especially semantic features. Therefore, we introduce mask-guided contextual distillation to extract global contextual information and richer feature statistics.

[0058] like Figure 3 As shown, the MCD of the present invention... To perform context pooling, first, spatial average pooling is performed (pooling kernel size is 5×5, stride is 2), and then... Channel segmentation is performed to improve computational efficiency and generate a set of features. and Then, the present invention uses global second-order pooling to capture (Right now (This provides more comprehensive statistical information.) Through the The result is obtained by reshaping, where N = (W / 2) × (H / 2). Finally, the MCD loss is obtained as follows:

[0059]

[0060] Where G is the number of channel segmentation groups. Through the introduction of context pooling, the MCD of this invention can extract the inherent global and rich statistical information in the features.

[0061] 2.2 Regularized Gradient Balancing Module (RGE)

[0062] By using a fixed KD scheme, the CISS model typically uses calibrated cross-entropy loss. [7] with total distillation loss Combined training:

[0063]

[0064] Where, λ CE and λ KD They are and The trade-off weights. Specifically, This typically involves the KD loss, which predicts the score output (e.g., ) and a set of intermediate layer feature outputs of KD loss (e.g. Since trade-off weights have a significant impact on model training, existing methods manually adjust the weights and fix them during the training phase. These methods struggle to handle different scenarios and samples.

[0065] Therefore, this invention proposes learning a set of dynamic trade-off weights. Clearly, CISS can be viewed as a multi-task learning task, and an effective weighting strategy is to balance the gradients of different tasks during training. Based on this, this invention proposes an objective function to update the weights through regularized gradient balancing at each iteration:

[0066]

[0067] The upper part of Formula 6 balances the effects of different feature extraction methods. yes The mean of (such as the gradient of the loss with respect to the output parameters of the l-th layer), and

[0068] In the first aspect, the invention restricts the gradients of distillation losses of different characteristics to tend toward their average value. We note that, compared with

[16] Compared to the norm, the mean is less sensitive to the dimension of the gradient and offers more stable performance. Furthermore, we introduce λ in the second term. F Prior constraints on, where (a) 0，+ =max(a, 0), using larger weights on the KD loss of deeper features to extract key semantic information. In this work, we set η = 1.25.

[0069] Due to Logit distillation and cross-entropy loss Both are closely related to the predicted score, and the lower half of Equation 6 achieves a gradient balance between logit distillation and cross-entropy loss. Specifically, HCE and H Lgt Each by and It is calculated that W is the parameter of the layer preceding the classification layer and c is the ratio of the number of new categories to the number of old categories. α l and β o These are the relative reverse training rates mentioned in []. In this invention, γ is set to 1e-3 to 5e-3 using cross-validation. To learn the dynamic trade-off weights, we minimize Equation 6.

[0070] 2.3 Dual Optimization of LKD

[0071] To address the stability-plasticity dilemma in a more flexible way, our LKD develops a two-layer optimization framework to simultaneously search for distillation schemes for different outputs and learn dynamically weighted tradeoffs during model training. Specifically, given initialized weights... We obtain the optimal distillation scheme for each output every T / N periods (T is the total number of iterations), and perform a greedy search on a set of distillation candidates, including Local POD. MD MCD And the original KD Based on the distillation scheme found, we minimize the objective function in Equation 6 to update the weights during each mini-batch iteration. This method addresses the stability-plasticity problem of class incremental semantic segmentation (CISS) by proposing a learnable knowledge distillation approach based on existing deep convolutional network models. The method employs a two-layer optimization framework to learn dynamic weight trade-offs while searching for distillation schemes, thus providing a more flexible solution to the CISS problem.

[0072] We evaluated the widely used Pascal VOC 12

[20] and ADE20K

[21] , and the experimental results achieved state-of-the-art performance. Next, we will elaborate on the experimental details and implementation of our LKD method.

[0073] To apply our LKD to the CISS task and to compare it with existing work, we adopted Deeplab-v3

[23] with ResNet-101

[22] as the base segmentation model. Specifically, we set the output stride of Deeplab-v3 to 16, pre-trained the ResNet-101 model on ImageNet-1k

[24] , and used batch normalization with in-situ activation

[25] . In addition, we cropped the input images to 512×512 and performed the same data augmentation as the Deeplab-v3 network. To train our model, we used the stochastic gradient descent (SGD) optimizer with the same hyperparameters as in Deeplab-v3. Specifically, for fair comparison, all models were trained on PASCAL VOC 12 and ADE20K for 30 and 60 epochs respectively, with an initial learning rate of 0.01, a training batch size of 24, and a learning rate reduced to 0.001 after the first training step. Following the common settings of previous methods, we use unbiased knowledge distillation loss to assist training. We set G in Equation 4 to 8 and 20 on PASCAL VOC 12 and ADE20K, respectively. ε and ξ in Equation 3 represent the proportion of foreground or background pixels in the total number of pixels in the batch image, respectively.

[0074] According to the general protocol

[15] , the experiments were conducted in two scenarios: overlapping and disjoint. Specifically, the overlapping scenario allowed all images to contain at least one new class pixel, while the disjoint scenario assumed that the image in the current step did not contain pixels belonging to the future new class. Furthermore, we denoted the incremental learning scenario as "AB", where A represents the number of classes trained in the initial step. In each subsequent step, class B was sequentially labeled as the foreground until all classes were learned.

[0075] The experimental results of the embodiments of the present invention are shown in Tables 1, 2 and 3, respectively. These results demonstrate the performance of various state-of-the-art methods on the ADE20K and PASCALVOC 12 datasets. The CISS methods include those mentioned above, as well as SDR

[26] , ST

[27] , and the LKD method in this invention. The results show that the method proposed in this invention achieves better performance than other methods, reaching state-of-the-art performance. In particular, in more challenging scenarios, our method still possesses the ability to resist catastrophic forgetting.

[0076] In summary, this invention proposes a learnable knowledge distillation (LKD) method for the CISS task, which simultaneously searches for distillation schemes and learns dynamic trade-off weights within a two-layer optimization framework. In particular, we propose two efficient distillation schemes (MD and MCD) to mitigate background bias and leverage richer contextual information, while formulating regularized gradient balances to learn dynamic trade-off weights. Results on two widely used benchmarks demonstrate that our LKD significantly outperforms existing KD-based methods in a variety of scenarios, suggesting that LKD provides a promising and flexible solution to the stability-plasticity dilemma of CISS. An alternative to greedily searching for the optimal distillation scheme is to learn a weighted combination of all distillation candidates, such as differentiable architecture search

[28] ; however, this strategy requires significant computational cost and is generally unstable

[29] .

[0077]

[0078] Table 1 Comparison of mIoU in overlapping scenarios based on the ADE20K dataset.

[0079]

[0080] Table 2 compares mIoU in overlapping scenarios based on the PASCALVOC 12 dataset.

[0081]

[0082]

[0083] Table 3 compares mIoU in disjoint scenarios based on the PASCALVOC 12 dataset.

[0084] 7. Other materials

[0085] References

[0086] [1]Liang-Chieh Chen,George Papandreou,Florian Schroff,and HartwigAdam.Rethinking atrous convolution for semantic image segmentation.arXiv,abs / 1706.05587,2017.1,6

[0087] [2]Liang-Chieh Chen,Yukun Zhu,George Papandreou,Flo-rianSchroff,andHartwig Adam.Encoder-decoder with atrous separable convolution for semanticimage segmentation.In ECCV,2018.1

[0088] [3]Umberto Michieli and Pietro Zanuttigh.Incremental learningtechniques for semamntic segmentation.In ICCV Workshops,2019.1,3,6,7

[0089] [4]Geoffrey E.Hinton,Oriol Vinyals,and Jeffrey Dean.Distilling theknowledge in a neural network.arXiv,1503.02531,2015.1,2,3

[0090] [5]Sungmin Cha,YoungJoonYoo,Taesup Moon,et al.SSUL:Semanticsegmentation with unknown label for exemplar-based class-incrementallearning.In NeurIPS,2021.1,2,3,7

[0091] [6]Dipam Goswami,Rene′Schuster,Joost van de Weijer,and DidierStricker.Attribution-aware weight transfer:A warm-start initialization forclass-incremental semantic segmentation.In WACV,2023.1,3,6,7

[0092] [7]Arthur Douillard,Yifu Chen,Arnaud Dapogny,and Matthieu Cord.PLOP:Learning without forgetting for continual semantic segmentation.In CVPR,2021.1,2,3,4,5,6,7,8

[0093] [8]Guanglei Yang,Enrico Fini,Dan Xu,Paolo Rota,Min-gli Ding,MoinNabi,Xavier Alameda-Pineda,and Elisa Ricci.Uncertainty-aware contrastivedistillation for in-cremental semantic segmentation.IEEE Trans.PatternAnal.Mach.Intell.,2022.1,3,6,7

[0094] [9]Minh Hieu Phan,Son Lam Phung,Long Tran-Thanh,AbdesselamBouzerdoum,et al.Class similarity weighted knowledge distillation for continual semanticsegmentation.In CVPR,2022.1,3,6,7

[0095]

[10] Shipeng Yan,Jiale Zhou,JiangweiXie,Songyang Zhang,and XumingHe.An EM framework for online incremental learning of semanticsegmentation.In ACM MM,2021.1,2,3

[0096]

[11] Youngmin Oh,DonghyeonBaek,and Bumsub Ham.ALIFE:Adaptive logitregularizer and feature replay for incremental semantic segmentation.InNeurIPS,2022.1,3,6,7

[0097]

[12] Chang-Bin Zhang,Jia-Wen Xiao,Xialei Liu,Ying-Cong,Chen,and Ming-Ming Cheng.Representation compensation networks for continual semanticsegmentation.In CVPR,2022.2,3,6,7,8

[0098]

[13] Guangyu Ren and Tania Stathaki.Dynamic knowledge distillationwith a single stream structure for RGB-D salient object detection.arXiv,2106.09517,2021.3

[0099]

[14] Lei Li,Yankai Lin,Shuhuai Ren,Peng Li,Jie Zhou,and Xu Sun.Dynamicknowledge distillation for pre-trained language models.In EMNLP,2021.3

[0100]

[15] Fabio Cermelli,Massimiliano Mancini,Samuel Rota Bulo,Elisa Ricci,and Barbara Caputo.Modeling the background for incremental learning insemantic seg-mentation.In CVPR,2020.1,2,3,4,5,6,7,8

[0101]

[16] Zhao Chen,Vijay Badrinarayanan,Chen-Yu Lee,and AndrewRabinovich.GradNorm:Gradient normalization for adaptive loss balancing indeep multitask networks.In ICML,2018.2,5,7,8

[0102]

[17] Arthur Douillard,Matthieu Cord,Charles Ollion,Thomas Robert,andEduardo Valle.PODNet:Pooled outputs distillation for small-tasks incrementallearning.In ECCV,2020.3

[0103]

[18] Kaiming He,Xiangyu Zhang,Shaoqing Ren,and Jian Sun.Spatialpyramid pooling in deep convolutional networks for visual recognition.IEEETrans.Pattern Anal.Mach.Intell.,37(9):1904–1916,2015.3

[0104]

[19] Svetlana Lazebnik,Cordelia Schmid,and Jean Ponce.Beyond bags offeatures:Spatial pyramid matching for recognizing natural scene categories.InCVPR,2006.3

[0105]

[20] Mark Everingham,S.M.Ali Eslami,Luc Van Gool,ChristopherK.I.Williams,John M.Winn,and Andrew Zisserman.The Pascal visual objectclasses challenge:A retrospective.Int.J.Comput.Vis.,111(1):98–136,2015.2,5

[0106]

[21] Bolei Zhou,Hang Zhao,Xavier Puig,Sanja Fidler,Adela Barriuso,andAntonio Torralba.Scene parsing through ADE20K dataset.In CVPR,2017.2,5

[0107]

[22] Kaiming He,Xiangyu Zhang,Shaoqing Ren,and Jian Sun.Deep residuallearning for image recognition.In CVPR,2016.5.

[0108]

[23] Liang-Chieh Chen,George Papandreou,Florian Schroff,and HartwigAdam.Rethinking atrous convolution for semantic image segmentation.arXiv,abs / 1706.05587,2017.1,6

[0109]

[24] Jia Deng,Wei Dong,Richard Socher,Li-Jia Li,Kai Li,and Li Fei-Fei.ImageNet:A large-scale hierarchical im-age database.In CVPR,2009.6

[0110]

[25] Samuel Rota Bulo,Lorenzo Porzi,and Peter Kontschieder.In-placeactivated BatchNorm for memory-optimized training of DNNs.In CVPR,2018.6

[0111]

[26] Umberto Michieli and Pietro Zanuttigh.Continual semanticsegmentation via repulsion-attraction of sparse and disentangled latentrepresentations.In CVPR,2021.3,6,7

[0112]

[27] Lu Yu,Xialei Liu,and Joost Van de Weijer.Self-training for class-incremental semantic segmentation.IEEE TNNLS,2022.1,3,6,7

[0113]

[28] Hanxiao Liu,Karen Simonyan,and Yiming Yang.DARTS:differentiablearchitecture search.In ICLR,2019.8

[0114]

[29] Xiangxiang Chu, Xiaoxing Wang, Bo Zhang, Shun Lu, Xiaolin Wei, and Junchi Yan. DARTS: robustly stepping out of performance collapse without indicators. In ICLR, 2021.8

[0115] This invention is not limited to the embodiments described above. The above description of specific embodiments is intended to illustrate and explain the technical solutions of this invention. The specific embodiments described above are merely illustrative and not restrictive. Without departing from the spirit and scope of the claims, those skilled in the art can make many specific modifications based on the teachings of this invention, and these modifications all fall within the scope of protection of this invention.

Claims

1. A class-incremental semantic segmentation method based on dynamic knowledge distillation, wherein the class-incremental semantic segmentation method is based on a two-layer optimization framework of learnable knowledge distillation, characterized in that... The two-layer optimization framework includes a frozen old model, a distillation scheme optimization module, a trainable new model, and a dynamic weight balancing module, wherein: The preferred model for the distillation scheme is based on initialization. Weights, according to each Each iteration performs a greedy search on a set of distillation candidates to obtain the optimal distillation scheme for each output, where: the optimal distillation scheme includes: Local POD ( ), MD ( ), MCD ( ) and the original KD ( ); The dynamic weight balancing module updates the weights during each mini-batch iteration using the following objective function. ;Right now: ； in, yes The mean, and ; and Each by and It was calculated that, among which It is the parameter of the layer preceding the classification layer and ; The ratio of the number of new categories to the number of old categories; and Relative reverse training rates; The optimal distillation scheme model obtains the optimal distillation scheme MD through a mask. )process: The optimal distillation scheme MD is generated according to the following loss function. ): ； in: This indicates that the feature after masking is over-scaling GT to mask. The background category label is set to 0, and the foreground category label is set to 1.

2. The class incremental semantic segmentation method based on dynamic knowledge distillation according to claim 1, characterized in that, The optimal distillation scheme MD is generated using the frozen old model. To address the context class semantic transfer process: In the Step 1, to transform the original image The image obtained after masking the background class The image obtained after masking the foreground class Feed to the old model In the middle, the first one was obtained respectively. The corresponding output of the layer , and ; The region least affected by the background class is calculated using the following formula: ； in, It is along the channel dimension The result obtained after performing average pooling; and These are two boundary values; and They represent medium to small And greater than Element; and Eliminate regions that are significantly affected by the background class and find regions that are less affected by the background class, respectively. The above two regions are processed by taking the union operation. To merge; For a given region ,if And by setting To generate .

3. The class incremental semantic segmentation method based on dynamic knowledge distillation according to claim 1, characterized in that, The optimal distillation scheme model obtains the optimal distillation scheme MCD through a mask. )process: Mask-guided context distillation is used to extract global contextual information and richer feature statistics. right Perform context pooling; first, perform spatial average pooling, and then... Perform channel segmentation to generate a set of features and ; Use global second-order pooling to capture Chinese statistical information; The following optimal distillation scheme (MCD) is generated according to the following loss function. ): ； in, It refers to the number of channel segments; through the introduction of context pooling, MCD is able to extract the inherent global and rich statistical information in the features. .

Images