Class increment semantic segmentation method based on feature purification gradient projection

Abstract

The invention discloses a class increment semantic segmentation method based on feature purification gradient projection, which is characterized in that a model is trained to learn the optimal segmentation capability of a current task, and then feature purification and construction of a specific feature space of the task are carried out; the method specifically comprises the steps of base class model training, feature purification, task specific subspace construction, gradient projection optimization and the like. Compared with the prior art, the method has the advantages that dependence on complete historical data is reduced under an incremental learning framework, the model does not need to retrain all categories from zero, the storage and calculation cost is greatly reduced, meanwhile, the training efficiency and the resource utilization rate are improved, and the method is suitable for popularization and application. Large-scale class increment semantic segmentation can be efficiently realized under the condition of finite computing power, and a more economical solution is provided for model deployment in the fields of automatic driving, intelligent monitoring and the like.

Description

Technical Field

[0001] The present invention relates to the technical fields of incremental learning and semantic segmentation, and in particular to a class-incremental semantic segmentation method based on feature purification gradient projection. Background Art

[0002] Class-incremental semantic segmentation (CISS) is a semantic segmentation task aimed at gradually expanding the model's recognized categories, and is widely applied in fields such as autonomous driving, intelligent monitoring, and medical image analysis. Different from traditional semantic segmentation, CISS requires the model to learn new categories without accessing the data of old tasks, while maintaining the recognition ability of the learned categories. However, affected by the catastrophic forgetting problem, the model often forgets the categories learned in the early stage during the incremental training process, resulting in a decline in overall performance.

[0003] Existing class-incremental learning methods mainly include methods based on regularization loss, methods based on replay, and methods based on network structure optimization. Regularization loss methods balance the learning between new and old tasks by constructing special loss functions, but require complex hyperparameter adjustment and it is difficult to ensure stability. Replay methods reduce forgetting by storing old task data or generating old task samples for replay, but there may be a risk of privacy leakage. Network structure optimization methods adapt to the learning needs of new tasks by adjusting or expanding the structure of neural networks. However, this usually increases the computational overhead and reduces the training efficiency.

[0004] In recent years, methods based on gradient optimization have received extensive attention in class-incremental learning tasks. These methods store the feature space of historical tasks and project the gradients during the training of new tasks to ensure that the learning of new tasks does not interfere with the feature representation of old tasks. Compared with traditional methods, gradient optimization methods have better interpretability, do not rely on data storage, and do not require complex regularization loss design, so they show good application prospects in many incremental learning scenarios. However, in the CISS task, since the same image may contain multiple categories, the direct application of existing gradient optimization methods will introduce more background noise during the feature space decomposition process, thereby reducing the incremental learning effect of the model. Therefore, how to effectively apply gradient optimization methods in the CISS task has become a key issue in current research.

[0005] In summary, the semantic segmentation of the prior art has problems such as large computational overhead, poor incremental learning effect of the model, inability to obtain high-quality segmentation results, low training efficiency, and poor stability and generalization ability. Summary of the Invention

[0006] The object of the present invention is to provide a class-incremental semantic segmentation method based on feature purification and gradient projection in view of the deficiencies of the prior art. The DeepLabv3 model is combined with DINOv2 as the backbone network, and multi-scale feature extraction is performed through ASPP. The base class model is trained and the model structure, training weights, and dataset information are recorded. The gradient class activation map is used to screen features related to the current task category, the category feature prototype is calculated and the feature length information is stored to reconstruct pure category features. Then, the singular value decomposition (SVD) is used to construct a task-specific feature subspace to ensure feature separation. Finally, the gradient of the incremental task is orthogonally projected using this feature subspace to remove the influence of the gradient on the old task feature space, ensuring that the learning of the new task does not interfere with the feature expression of the learned categories, thereby alleviating catastrophic forgetting and improving the stability and generalization ability of incremental semantic segmentation. This method can reduce the dependence on the complete historical data in the incremental learning framework, enabling the model to avoid retraining all categories from scratch, greatly reducing the storage and computational costs, while improving the training efficiency and resource utilization rate, enabling the efficient implementation of large-scale class-incremental semantic segmentation under limited computing power conditions, providing a more economical solution for model deployment in fields such as autonomous driving and intelligent monitoring, and having good application prospects.

[0007] The specific technical solution for achieving the object of the present invention is: a class-incremental semantic segmentation method based on feature purification and gradient projection, which adopts a standard training process to train the model to learn the optimal segmentation ability of the current task, and records the model structure, training weights, and the current task dataset. After the task training is completed, feature purification and the construction of a task-specific feature space are carried out. First, the gradient class activation map is used to screen task-related features, removing the interference information from other tasks to ensure that only the feature representation of the current task is retained; then, based on the extracted feature set, the category prototype is constructed, and the diversity of the feature vectors is increased through a feature enhancement strategy to obtain a structured feature representation. Subsequently, the singular value decomposition (SVD) method is used to extract a task-specific feature subspace to ensure that the feature spaces of different tasks can be effectively separated, providing guidance for the gradient optimization of the next task. In the incremental task training stage, a gradient projection matrix is constructed using the feature subspace of the previous task to constrain the gradient of the new task learning, making its update direction orthogonal to the feature space of the historical tasks, ensuring that the learning of the new category does not interfere with the feature expression of the old categories, thereby effectively alleviating catastrophic forgetting and achieving a dynamic balance between stability and plasticity. This method specifically includes:

[0008] Step 1: Training of the base class model

[0009] 1.1) For the input RGB image, use the DeepLabv3 model, adopt DINOv2 as the backbone network, and perform multi-scale feature extraction on the features extracted by the backbone network through ASPP to train a base class semantic segmentation model;

[0010] 1.2) After the training is completed, record the model structure, training weights, and dataset information of the current task as the basis for the subsequent feature purification and incremental learning stages.

[0011] Step 2: Feature Purification and Construction of Task-Specific Subspaces

[0012] 2.1) Feed the input image into the base class model trained in Step 1 and set to inference mode to extract the input features of each layer of the ASPP module. The feature of the i-th layer is denoted as F i , whose dimension is (B, C, H, W);

[0013] 2.2) For each layer of the feature F i , use the Gradient-weighted Class Activation Mapping (GradCAM) method to calculate the class activation map for each category c ∈ C where GradCAM generates the heatmap for category c, and the value range is 0 to 1. According to the threshold λ (default is 0.5), select the feature set most relevant to the base category c ∈ C from the heatmap

[0014] 2.3) Based on the selected feature set calculate the feature prototype for each category c where, f c is the feature vector in the feature , and N represents the normalization operation. To retain the length information of the original feature vector, further calculate and store the mean μ c and variance σ c :

[0015] Finally, use the category prototype P c and the corresponding length information (μ c , σ c ) to construct the reconstructed feature to reconstruct the purified category feature R i : where, is the feature vector in the reconstructed feature R i . Thus, avoid the noise interference introduced by features that do not belong to the current category.

[0016] 2.4) Expand the extended reconstructed feature R i into matrix form and calculate the decentralized covariance matrix Ci : C i = R i (R i ) T . Among them, T represents the transpose operation of the matrix. On this basis, the singular value decomposition of the decentralized covariance matrix is performed, and its formula is: Among them, U i is the left singular matrix; V i is the right singular matrix.

[0017] According to the threshold ε, k eigenvectors in U i can be selected to form the feature subspace S i of the current task, and the selection formula is:

[0018] Step 3: Gradient projection optimization

[0019] 3.1) After steps 1 and 2, the feature subspace S i of the base class stage is obtained. In the incremental stage, calculate the loss gradient of the current task and use S i to construct a gradient projection matrix for orthogonal projection, and its formula is as follows: to remove the component of the gradient in the incremental stage on the feature space of the old task.

[0020] 3.2) Finally, update the model parameter W to ensure that the learning of the new task does not affect the feature expression of the learned categories, and its effect is as follows:

[0021] W new F i = W old F i + ΔWF i ≈ W old F i .

[0022] Among them, F i represents the feature of the i-th layer, W new represents the network weight of the new task, and W old represents the network weight of the old task.

[0023] The described feature purification method uses the gradient class activation map to screen the features related to the current task category, calculates the category feature prototype, and stores the feature length information to ensure the accuracy of feature extraction and constructs a pure feature that removes background noise.

[0024] The present invention is characterized in that a task-specific feature subspace is constructed through singular value decomposition (SVD), and the subspace data is stored to perform an orthogonal projection on the gradient of the incremental task, removing the influence of the gradient on the feature space of the old task, so as to ensure that the learning of the new task does not interfere with the feature expression of the learned categories.

[0025] Compared with the prior art, the present invention has the following beneficial technical effects and remarkable technical progress:

[0026] 1) Optimize model update using gradient projection, only store the feature subspace data without retaining the original training data, and avoid complex regularization loss design at the same time, realizing efficient incremental learning, alleviating catastrophic forgetting, and enhancing the long-term scalability and generalization ability of semantic segmentation tasks.

[0027] 2) Effectively solve the problems of feature mixing and catastrophic forgetting in the process of incremental semantic segmentation, remove background noise through feature purification, and use gradient projection to ensure that the learning of the new task does not interfere with the learned categories, improving the stability and generalization ability of the model, reducing storage and computational costs, and enhancing the efficiency of incremental learning.

[0028] 3) Enable large-scale class-incremental semantic segmentation to be efficiently implemented under limited computing power conditions, provide a more efficient solution for the long-term scalable model deployment in scenarios such as autonomous driving and intelligent monitoring, and have good application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 is a flowchart of the present invention;

[0030] Figure 2 is a flowchart of feature reconstruction and subspace construction;

[0031] Figure 3 is a flowchart of gradient projection optimization. DETAILED DESCRIPTION OF THE INVENTION

[0032] The present invention includes three steps: First, in the current task training phase, a standard training process is adopted to train the model to learn the optimal segmentation ability of the current task, and the model structure, training weights, and the current task dataset are recorded to provide a basis for subsequent steps. After the task training is completed, feature purification and the construction of a task-specific feature space are carried out. First, gradient-based class activation maps are used to screen task-related features, removing interference information from other tasks to ensure that only the feature representation of the current task is retained; then, based on the extracted feature set, class prototypes are constructed, and the diversity of feature vectors is increased through a feature enhancement strategy to obtain a structured feature representation. Subsequently, the singular value decomposition (SVD) method is used to extract the task-specific feature subspace, ensuring that the feature spaces of different tasks can be effectively separated to provide guidance for the gradient optimization of the next task. In the incremental task training phase, a gradient projection matrix is constructed using the feature subspace of the previous task to constrain the gradient learned by the new task, making its update direction orthogonal to the feature space of the historical task, ensuring that the learning of new classes does not interfere with the feature expression of old classes, thus effectively alleviating catastrophic forgetting and achieving a dynamic balance between stability and plasticity.

[0033] To facilitate the understanding of the present invention, the present invention will be described in detail below in conjunction with the accompanying drawings and embodiments.

[0034] Embodiment 1

[0035] Refer to Figure 1 , the present invention performs class-incremental semantic segmentation according to the following steps:

[0036] S100: For the input RGB image, use the DeepLabv3 model for semantic segmentation to obtain the initial semantic segmentation ability of the model on the base classes.

[0037] S110: Refer to Figure 2 , the intermediate layer features of the model obtained in step S100 are purified through the GradCAM method to extract the features that best represent the base classes.

[0038] S120: Refer to Figure 2 , use the features purified in step S110 to construct a feature subspace.

[0039] S130: Input the RGB image into the DeepLabv3 model again to train the semantic segmentation ability of the model on the incremental classes.

[0040] S140: Refer to Figure 3 , for the gradient generated in step S130, orthogonal gradient projection optimization is performed using the feature subspace constructed in step S120.

[0041] Refer to Figure 2, the specific processes of feature reconstruction and subspace construction in the present invention are as follows:

[0042] S200: Use the GradCAM method to eliminate features that do not belong to the current task category and retain the purified features that are most representative of the category.

[0043] S210: Calculate the category feature prototype and feature length information using the purified features obtained in step S200.

[0044] S220: Reconstruct the purified features based on the prototype and length information calculated in step S210.

[0045] S230: Use the features reconstructed in step S220 to perform singular value decomposition and construct the feature subspace of the current task.

[0046] Refer to Figure 3 , the specific process of gradient projection optimization in the present invention is as follows:

[0047] S300: Obtain the basic category feature subspace constructed in step 2 and the gradients generated by the model during the incremental category training phase.

[0048] S310: Construct a projection matrix for gradient projection based on the feature subspace in step S300.

[0049] S320: Use the projection matrix constructed in step S310 to perform an orthogonal projection of the gradient in step S300 onto the basic category feature subspace to determine the orthogonal direction of the gradient.

[0050] S330: Update the model parameters using the result of step S320 to obtain the orthogonal gradient, ensuring that the learning of new categories does not interfere with the feature expression of the categories that the model has learned.

[0051] The present invention can reduce the dependence on complete historical data in the incremental learning framework, enabling the model to avoid retraining all categories from scratch, greatly reducing the storage and computational costs, while improving the training efficiency and resource utilization rate, and enabling the efficient implementation of large-scale class incremental semantic segmentation under limited computing power conditions, providing a more economical solution for model deployment in fields such as autonomous driving and intelligent monitoring.

[0052] The above is only a further explanation of the present invention and is not intended to limit this patent. All equivalent implementations of the present invention should be included within the scope of the claims of this patent.

Claims

1. A class incremental semantic segmentation method based on feature purification gradient projection, characterized in that: The method specifically includes: Step 1: Base model training 1.1) For the input RGB image, the DeepLabv3 model is used, DINOv2 is adopted as the backbone network, and multi-scale feature extraction is performed on the features extracted by the backbone network through ASPP to train the base class semantic segmentation model; 1.2) After completing the base class semantic segmentation model training, record the model structure, training weights and data set information of the current task; Step 2: Feature purification and task-specific subspace construction 2.1) Send the input image to the base model trained in step 1 and set to inference mode to extract the input features F of each layer of the ASPP module i , whose dimensions are (B, C, H, W); 2.2) Input feature F i Using the gradient class activation map method, the class activation map of each category c∈C is calculated by the following formula Among them, GradCAM is the heat map of generated category c; C is the basic category set; 2.3) From the heat map, select the most relevant feature set for the basic category c∈c by the following formula: Among them, λ is the threshold; 2.4) Calculate the feature prototype P of category c by the following formula c : Among them, f c is a feature set The eigenvector in; N is the normalization operation; 2.5) Calculate the mean μ for the stored characteristic lengths using the following formula c and variance σ c : 2.6) Use the category prototype P c and the corresponding length information (μ c , σ c + The reconstruction feature R is constructed by the following formula i The eigenvectors in 2.7) The expanded reconstructed feature R i Expand into matrix form, and calculate the decentralized covariance matrix C by the following formula i : C i =R i (R i ) T ; Where T is the transpose operation of the matrix; 2.8) For the decentralized covariance matrix C i The following formula is used for singular value decomposition: Among them, U i is a left singular matrix; V i is a right singular matrix; 2.9) Select k U from the following formula i The feature vector in forms the feature subspace S of the current task i : Among them, ε is the threshold; Step 3: Gradient Projection Optimization 3.1) Calculate the loss gradient of the current task by the following formula: And using the characteristic subspace S i Construct the gradient projection matrix for orthogonal projection: Among them, S i is the characteristic subspace of the base class stage; 3.2) Orthogonalization is used to update the gradient of the new task and update the model parameters W to obtain the class incremental semantic segmentation effect expressed as follows: W new F i =W old F i +ΔWF i ≈W old F i ; Among them, F i is the feature of the i-th layer; W new is the network weight for the new task; W old is the network weight of the old task.

2. The class incremental semantic segmentation method based on feature purification gradient projection according to claim 1, characterized in that: The feature purification stage only selects feature vectors that are highly relevant to the current task category for spatial construction, and screens the most discriminative feature regions through the gradient class activation map.

3. The class incremental semantic segmentation method based on feature purification gradient projection according to claim 1, characterized in that: The characteristic subspace S i The construction adopts the method of feature prototype calculation and length information storage, and extracts key feature subspaces through singular value decomposition.

Images