An incremental semantic segmentation method based on meta-learning and pseudo-labeling strategy

Abstract

The invention discloses an incremental semantic segmentation method based on meta-learning and pseudo-label strategy, including: establishing a model; characterization training, at the beginning of each incremental step, freezing other network structures except the feature extractor, and using a temporary segmentation head to train the feature extractor; knowledge connection, only updating the parameters of the mask generator and the segmentation head, freezing the parameters of other structures, the feature extractor extracts the features of the training data, and the mask generator generates a mask to segment the semantic segmentation network; knowledge updating, using the meta-learning strategy to train a shared knowledge base, updating parameters, and then obtaining reusable knowledge to face new tasks; knowledge consolidation, training the mask generator and the segmentation head again to consolidate the new knowledge in the shared knowledge base. The invention adopts a meta-learning algorithm to alleviate the model's forgetting of old tasks, and adopts a pseudo-label strategy to solve the problem of background semantic drift.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to an incremental semantic segmentation method based on meta-learning and pseudo-labeling strategy. Background Art

[0002] With the continuous development of artificial intelligence technology, computer vision has attracted much attention as an important branch of computer application technology. Semantic segmentation is an extremely challenging pixel-level classification task in the field of computer vision. Traditional semantic segmentation methods mostly use graph partitioning based on graph theory to solve the segmentation problem, set an objective function, and segment the image through optimization. However, the excessive amount of calculation and the bottleneck accuracy cannot meet the expectations of users. The emergence of deep learning methods has greatly improved the accuracy of semantic segmentation and met the needs of industrial applications.

[0003] After the semantic segmentation method based on deep learning annotates the large-scale data set at the pixel level, it uses the deep neural network to learn and fit the training data, so as to have a good classification accuracy on unknown data. However, it is obviously unreasonable for the neural network to learn all the knowledge of the categories at one time. In the real semantic segmentation application scenario, new categories are always generated in an endless stream, which requires the semantic segmentation model to have the ability to continuously learn new knowledge. Incremental learning is the continuous learning of new knowledge. However, the current knowledge learning strategy makes it very easy to forget the knowledge of old categories after learning the old knowledge and in the process of learning new knowledge. The model after incremental learning can usually have a high accuracy in the classification of the categories involved in the new task, but the classification accuracy of the categories designed for the old task is greatly reduced, which leads to the catastrophic forgetting problem in the incremental learning process.

[0004] In addition, the new knowledge does not contain the old knowledge, that is, the training samples of the earlier incremental step may contain unlabeled categories that need to be learned in the later incremental step, and the training samples of the later incremental step may also contain unlabeled categories that have been learned in the earlier incremental step. The wrong label information of the old category pixels will seriously affect the model's absorption of new knowledge, and thus lead to the background drift problem in the incremental learning of semantic segmentation. Summary of the invention

[0005] The purpose of the present invention is to provide an incremental semantic segmentation method based on meta-learning and pseudo-labeling strategy. The meta-learning algorithm is adopted to encourage the reusability of model parameters when learning new tasks, rather than directly overwriting the parameters, thereby alleviating the catastrophic forgetting problem of the model for old tasks; the pseudo-labeling strategy is adopted to provide pseudo-labels of old categories for the learning of new tasks by utilizing the prediction output of the old model to solve the drift problem of background semantics.

[0006] To achieve the above object, the present invention provides an incremental semantic segmentation method based on meta-learning and pseudo-labeling strategy, comprising the following steps:

[0007] S1, establishing a model, the model includes a mask generation module and a semantic segmentation network, the mask generation module includes a feature extractor and a mask generator, the feature extractor extracts features from an image, the mask generator takes the features as input, generates a set of task-related masks, which are used to query and select relevant weights of the semantic segmentation network to solve the corresponding task; the semantic segmentation network includes a shared knowledge base and a segmentation head, and the semantic segmentation network is trained in combination with a meta-learning method and the mask generator. During the training process, pseudo labels are generated for the training set in combination with the output of the old model;

[0008] S2, representation training, at the beginning of each incremental step, freeze the other network structures except the feature extractor, and use the temporary segmentation head to train the feature extractor;

[0009] S3, knowledge connection, only updates the parameters of the mask generator and the segmentation head, and freezes the parameters of other structures; the feature extractor extracts the features of the training data, and the mask generator generates a mask to segment the semantic segmentation network;

[0010] S4, knowledge updating, uses a meta-learning strategy to train the shared knowledge base, update parameters, and then obtain reusable knowledge to face new tasks;

[0011] S5, knowledge consolidation, training the mask generator and the segmentation head again to consolidate the new knowledge in the shared knowledge base.

[0012] Furthermore, before all tasks begin, the semantic segmentation network is trained based on the training data using a traditional training method, and steps S2 to S5 are repeated in each subsequent incremental step.

[0013] Furthermore, the feature extractor is implemented using a deep convolutional network method.

[0014]

[0015] Among them, F t is the feature extractor corresponding to the incremental task t, X i is the input image, It is the feature map extracted after the input image passes through the deep convolutional network.

[0016] Furthermore, the mask generator is implemented using the following formula:

[0017]

[0018] The feature map extracted by the feature extractor is passed through a single layer of convolution conv and activated by the activation function ρ to obtain the input image X corresponding to the increment step t i Mask

[0019] Furthermore, the semantic segmentation network is trained with the mask generator in combination with a meta-learning method, and the shared knowledge base is composed of n residual blocks with residual connections connected in series, each of which is implemented using a deep convolutional network.

[0020]

[0021] Among them, φ is the shared knowledge base, For the input image X i The mask generated at increment step t, f i For the input image X i The feature map after passing through the i-th residual block in sequence;

[0022] The deepest feature map f n Send it to the segmentation head to get the semantic segmentation result

[0023] o=H(f n )

[0024] Among them, o is the semantic segmentation result, and H is the semantic segmentation head.

[0025] Furthermore, the deepest feature map f n The calculation method is as follows:

[0026] Fixed the weight of the shared knowledge base, for the input image X i , through the feature extractor and the mask generator, the corresponding mask M is obtained i t ;

[0027] X i Through the shared knowledge base, in the process of convolution calculation, the relevant knowledge in the shared knowledge base is selected according to the information of the mask, so as to obtain the feature map f n :

[0028]

[0029] Among them, b i is the i-th residual block of the shared knowledge base, δ is the adaptive average pooling layer, Mask The jth component of .

[0030] Further, in step S4, the method of using the meta-learning strategy to train the shared knowledge base to update parameters includes an inner loop and an outer loop. The inner loop creates and trains task-related models, and the outer loop integrates and generalizes the task-related models to make them applicable to all tasks. The specific updating method is as follows:

[0031] When the inner loop updates the model parameters, the parameters of the current shared knowledge base Φ base As the initialization parameters for each mini-batch, the task-dependent model is trained using cross entropy loss;

[0032] The outer loop uses all the small batch parameters and takes their average direction as the final update direction of the shared knowledge base to obtain the following model

[0033]

[0034] where η is a momentum-based controller, Allows you to speed up the training at the beginning and slow down the training at the end.

[0035] Furthermore, the pseudo-label generation strategy is as follows:

[0036] In the current training step, the old model is used to forward infer the training set to obtain the semantic segmentation results of the old model of the training data, which are compared with the labels of the training data. The categories of non-background classes in the labels remain unchanged. For the background classes in the labels, if the confidence of the result of the old model semantic segmentation at this point is greater than the threshold, the old model semantic segmentation result is used to cover the label value; otherwise, the label category is still retained.

[0037] The beneficial effects of the present invention are as follows: the present invention adopts a meta-learning algorithm and a pseudo-label strategy to realize incremental learning semantic segmentation, the mask generation module generates a set of task-related masks for solving corresponding tasks in incremental learning semantic segmentation; the semantic segmentation network gradually learns new knowledge from new tasks through a meta-learning strategy to expand the knowledge base without affecting its performance on old tasks; the mask generation module and the semantic segmentation network cooperate with each other to promote the reusability of model weights, rather than directly covering and causing catastrophic forgetting. In addition, in order to solve the background drift problem in the incremental learning semantic segmentation task, the prior knowledge output by the old model is used to provide pseudo-labels for the learning of new tasks, further improving the average intersection-over-union ratio of the model on all semantic categories. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a schematic diagram of the incremental learning principle of semantic segmentation of the present invention;

[0039] Figure 2 This is a flowchart of the semantic segmentation incremental learning training of the present invention; DETAILED DESCRIPTION

[0040] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0041] like Figure 1 The figure shows the principle diagram of incremental learning for semantic segmentation of the present invention. First, the feature extractor extracts features from the image, and the mask generator takes the features as input to generate a set of task-related masks, which are used to query and select the relevant weights of the semantic segmentation network to solve the corresponding tasks; then, the semantic segmentation network is trained by combining the meta-learning method with the mask generator to predict the segmentation results. Finally, pseudo labels are generated for the training set in combination with the output of the old model to supplement the labels of the old categories in the training data.

[0042] like Figure 2 The figure shows the flowchart of the semantic segmentation incremental learning training of the present invention. The present invention is different from the traditional training process. Before all tasks start, the model is first established, and the semantic segmentation network is trained based on the training data using the traditional training method. Each subsequent incremental step repeats the following S2-S5 steps. The traditional training method is to input the model for training, execute the S2-S3 steps, without the knowledge update and consolidation process, and train the semantic segmentation model at the same time.

[0043] The steps of establishing the model are as follows: S1: the model includes a mask generation module and a semantic segmentation network, the mask generation module includes a feature extractor and a mask generator, the feature extractor extracts features from the image, the mask generator takes the features as input, generates a set of task-related masks, which are used to query and select relevant weights of the semantic segmentation network to solve the corresponding tasks; the semantic segmentation network includes a shared knowledge base and a segmentation head, and the semantic segmentation network is trained by combining the meta-learning method and the mask generator. During the training process, the output of the old model is combined to generate pseudo labels for the training set;

[0044] S2: Representation training: At the beginning of each incremental step, freeze all network structures except the feature extractor, and use the temporary segmentation head to train the feature extractor;

[0045] S3: Knowledge connection, only the parameters of the mask generator and the segmentation head are updated, and the parameters of other structures are frozen. The feature extractor extracts the features of the training data, and the mask generator generates masks for segmentation of the semantic segmentation network. Since the parameters of the shared knowledge base are frozen, the model completes the training using the previously accumulated knowledge, reuses and consolidates the old knowledge.

[0046] The mask generator is implemented as follows

[0047]

[0048] The feature map extracted by the feature extractor is passed through a single layer of convolution conv and activated by the activation function ρ to obtain the input image X corresponding to the increment step t i Mask

[0049] Masks are generated through a mask generation module to query and select relevant weights of the semantic segmentation network to solve the corresponding tasks.

[0050] Among them, the feature extractor in the mask generation module is implemented using a deep convolutional network method:

[0051]

[0052] Among them, F t is the feature extractor corresponding to the incremental task t, X i is the input image, It is the feature map extracted after the input image passes through the deep convolutional network.

[0053] The mask generator in the mask generation module is implemented as follows:

[0054]

[0055] The feature map after the feature extractor passes through a single layer of convolution conv and passes through the activation function ρ to obtain the input image X corresponding to the incremental step t i Mask

[0056] S4: Knowledge update, using meta-learning strategy to train the shared knowledge base, update parameters, and then obtain reusable knowledge to face new tasks; the semantic segmentation network consists of a shared knowledge base and a segmentation head, and is trained by combining meta-learning methods and mask generators. The shared knowledge base is composed of n residual blocks with residual connections connected in series, and each residual block is implemented using a deep convolutional network:

[0057]

[0058] Among them, φ is the shared knowledge base, For the input image X i The mask generated at increment step t, f i For the input image X i The feature maps after passing through the i-th residual block in sequence.

[0059] The deepest feature map f n Send it to the segmentation head to get the semantic segmentation result:

[0060] o=H(f n )

[0061] Among them, o is the semantic segmentation result, and H is the semantic segmentation head.

[0062] The deepest feature map f n The calculation method is as follows: fix the weight of the shared knowledge base, for the input image X i , through the feature extractor and mask generator, the corresponding mask is obtained At the same time, X i Through the shared knowledge base, in the process of convolution calculation, the relevant knowledge in the knowledge base is selected according to the information of the mask, and then the feature map f is obtained. n :

[0063]

[0064] Among them, b i is the i-th residual block of the shared knowledge base, δ is the adaptive average pooling layer, Mask The jth component of .

[0065] The update method based on meta-learning is used to update the parameters of the shared knowledge base. The update method includes an inner loop and an outer loop. The inner loop creates and trains task-related models, and the outer loop integrates and generalizes task-related models to make them applicable to all tasks. When the inner loop updates the model parameters, the parameters of the current shared knowledge base are Φ base As the initialization parameters of each small batch, the cross entropy loss is used to train the task-related model; the outer loop uses all the small batch parameters and takes their average direction as the final update direction of the shared knowledge base to obtain a more prosperous and general model:

[0066]

[0067] Among them, η is a momentum-based controller, and its formula is η is a parameter that controls each small batch of tasks in meta-learning. Its value changes with the task. The purpose is to update faster in the early stages of incremental learning (larger η value) and slower in the later stages (smaller η value), that is, to allow training to be accelerated at the beginning and slowed down at the end.

[0068] S5: Knowledge consolidation, the mask generator and segmentation head are trained again to consolidate the new knowledge in the shared knowledge base.

[0069] The pseudo-label generation strategy is as follows: in the current training step, the old model is used to forward reason the training set to obtain the semantic segmentation result of the old model of the training data, which is compared with the label of the training data. The category of the non-background class in the label remains unchanged. For the background class in the label, if the confidence of the result of the semantic segmentation of the old model at this point is greater than the threshold, the label value is covered by the semantic segmentation result of the old model; otherwise, the label category is retained. The present invention adopts a pseudo-label strategy, and provides pseudo-labels of the old category for new task learning by using the prediction output of the old model, which solves the background drift problem in incremental learning of semantic segmentation and further improves the prediction performance of the model. Specific embodiments

[0071] The following is a specific embodiment of the semantic segmentation model based on meta-learning and incremental learning in the above method, so that those skilled in the art can better understand the effect of the present invention.

[0072] The following simulation experiment is conducted based on the incremental learning semantic segmentation method based on meta-learning and pseudo-labeling strategy. The implementation method of this embodiment is as described in S1-S5 above, and the specific steps will not be elaborated in detail. The following only shows its effect based on the experimental results.

[0073] The model in this example consists of a shared knowledge base, a segmentation head, a feature extractor, and a mask generator.

[0074] The shared knowledge base and feature extractor both use the resnet50 skeleton network to extract deep semantic features of the image; the mask generator is implemented using convolutional layers and adaptive average pooling layers of appropriate channels to compress the spatial dimension; the segmentation head uses the DeeplabV3 segmentation head structure to make full use of semantic information of different scales in the feature map. The semantic segmentation task carried out on the PASCAL-VOC 2012 dataset has two incremental settings for incremental learning, namely 10-10 (two steps) and 5-5-5-5 (four steps). Under the 10-10 setting, 20 classes are learned in two steps, 10 categories each time. Under the 5-5-5-5 setting, 20 classes are learned in four steps, 5 categories each time.

[0075] At the same time, both incremental settings study the incremental learning effects in the case of disjoint categories and overlapping categories. In the case of disjoint categories, the categories in the training images only include new categories in the current step and old categories that have been learned; in the case of overlapping categories, there is no restriction on the categories of the training images, including at least one new category, and other categories may include old categories that have been learned or categories that may appear in future steps. The implementation effects of incremental learning in two steps and four steps of this implementation example are shown in Table 1 and Table 2 respectively.

[0076] Table 1 Average Intersection-over-Union of different incremental learning schemes in two steps on the PASCAL-VOC 2012 dataset

[0077]

[0078] Table 2 Average IoU of different incremental learning schemes in four steps on PASCAL-VOC 2012 dataset

[0079]

[0080] As shown in Table 1, the average intersection over union (mIoU) of the model of the present invention (Ours) reached 62.8% and 65.1% in the 10-10 category disjoint and category overlapping scenarios, surpassing the results of the most recent state-of-the-art methods (MiB, PLOP). As shown in Table 2, the average intersection over union (mIoU) of the present invention (Ours) was improved by 2.6 and 3.5 respectively compared with the most recent state-of-the-art method (MiB) in two different scenarios of the 5-5-5-5 experimental setting. At the same time, the performance differences of all the compared methods are large in different incremental steps. In contrast, the present invention can maintain the stability of the performance of the model at all steps during the incremental learning process, and can achieve the learning of new tasks by using meta-learning methods and pseudo-label strategies, while better alleviating the problem of catastrophic forgetting.

[0081] In summary, the implementation example of the present invention is different from the prior art. In the incremental learning process, mask queries and meta-learning strategies are generated to update the semantic segmentation network shared knowledge base, thereby promoting the reusability of weights. Compared with the prior art, the present invention can alleviate the model's forgetting of old task knowledge and overfitting of new tasks. The model has a stable average intersection-over-union ratio on all semantic categories and is more robust.

[0082] The embodiments of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited thereto. Various changes that can be made within the knowledge scope of technicians in the relevant technical field without departing from the spirit of the present invention are all within the protection scope of the claims of the present invention.

Claims

1. An incremental semantic segmentation method based on meta-learning and pseudo-labeling strategy, characterized in that: The following steps are involved: S1, establishing a model, the model includes a mask generation module and a semantic segmentation network, the mask generation module includes a feature extractor and a mask generator, the feature extractor extracts features from an image, the mask generator takes the features as input, generates a set of task-related masks, which are used to query and select relevant weights of the semantic segmentation network to solve the corresponding task; the semantic segmentation network includes a shared knowledge base and a segmentation head, and the semantic segmentation network is trained in combination with a meta-learning method and the mask generator. During the training process, pseudo labels are generated for the training set in combination with the output of the old model; S2, representation training, at the beginning of each incremental step, freeze the other network structures except the feature extractor, and use the temporary segmentation head to train the feature extractor; S3, knowledge connection, only updates the parameters of the mask generator and the segmentation head, and freezes the parameters of other structures; the feature extractor extracts the features of the training data, and the mask generator generates a mask to segment the semantic segmentation network; S4, knowledge updating, uses a meta-learning strategy to train the shared knowledge base, update parameters, and then obtain reusable knowledge to face new tasks; S5, knowledge consolidation, training the mask generator and the segmentation head again to consolidate the new knowledge in the shared knowledge base.

2. The incremental semantic segmentation method based on meta-learning and pseudo-labeling strategy according to claim 1, characterized in that: Before all tasks begin, the semantic segmentation network is trained based on the training data using a traditional training method, and steps S2 to S5 are repeated in each subsequent incremental step.

3. The incremental semantic segmentation method based on meta-learning and pseudo-labeling strategy according to claim 1, characterized in that: The feature extractor is implemented using a deep convolutional network method. Among them, F t is the feature extractor corresponding to the incremental task t, X i is the input image, It is the feature map extracted after the input image passes through the deep convolutional network.

4. The incremental semantic segmentation method based on meta-learning and pseudo-labeling strategy according to claim 1, characterized in that: The mask generator is implemented as follows The feature map extracted by the feature extractor is passed through a single layer of convolution conv and activated by the activation function ρ to obtain the input image X corresponding to the increment step t i Mask 5. The incremental semantic segmentation method based on meta-learning and pseudo-labeling strategy according to claim 1, characterized in that: The semantic segmentation network is trained with the mask generator in combination with a meta-learning method. The shared knowledge base is constructed by connecting n residual blocks with residual connections in series. Each residual block is implemented using a deep convolutional network. Among them, φ is the shared knowledge base, For the input image X i The mask generated at increment step t, f i For the input image X i The feature map after passing through the i-th residual block in sequence; The deepest feature map f n Send it to the segmentation head to get the semantic segmentation result o=H(f n ) Among them, o is the semantic segmentation result, and H is the semantic segmentation head.

6. The incremental semantic segmentation method based on meta-learning and pseudo-labeling strategy according to claim 5, characterized in that: The deepest feature map f n The calculation method is as follows: Fixed the weight of the shared knowledge base, for the input image X i , through the feature extractor and the mask generator, the corresponding mask is obtained X i Through the shared knowledge base, in the process of convolution calculation, the relevant knowledge in the shared knowledge base is selected according to the information of the mask, so as to obtain the feature map f n : Among them, b i is the i-th residual block of the shared knowledge base, δ is the adaptive average pooling layer, Mask The jth component of .

7. The incremental semantic segmentation method based on meta-learning and pseudo-labeling strategy according to claim 2, characterized in that: In step S4, the method of using the meta-learning strategy to train the shared knowledge base to update parameters includes an inner loop and an outer loop, wherein the inner loop creates and trains task-related models, and the outer loop integrates and generalizes the task-related models to make them applicable to all tasks. The specific updating method is as follows: When the inner loop updates the model parameters, the parameters of the current shared knowledge base Φ base As the initialization parameters for each mini-batch, the task-dependent model is trained using cross entropy loss; The outer loop uses all the small batch parameters and takes their average direction as the final update direction of the shared knowledge base to obtain the following model where η is a momentum-based controller, Allows you to speed up the training at the beginning and slow down the training at the end.

8. The incremental semantic segmentation method based on meta-learning and pseudo-labeling strategy according to claim 1, characterized in that: The pseudo-label generation strategy is as follows: In the current training step, the old model is used to forward infer the training set to obtain the semantic segmentation results of the old model of the training data, which are compared with the labels of the training data. The categories of non-background classes in the labels remain unchanged. For the background classes in the labels, if the confidence of the result of the old model semantic segmentation at this point is greater than the threshold, the old model semantic segmentation result is used to cover the label value; otherwise, the label category is still retained.

Images