An incremental semantic segmentation method with biased context information correction

Abstract

The present invention discloses an incremental semantic segmentation method for correction of biased context information. The method first obtains semantic segmentation data streams of multiple categories and divides them into multiple training data sets; in the first incremental semantic segmentation learning step, an initial semantic segmentation network model is obtained by learning with one training data set; in the next incremental semantic segmentation learning step, a new class pixel point is erased for the newly obtained training data set containing the new category, a pair of images with biased context information correction is generated, a training data set with biased context information correction is constructed, and based on the training data set with biased context information correction, the latest incremental semantic segmentation network model is updated with a learning method of biased context information correction and adaptive class balance. The present invention can effectively correct the context information of old class pixels that is biased towards the new class and alleviate the problem of biased class distribution, thereby reducing the forgetting of old class knowledge and the semantic drift of background classes.

Description

Technical Field

[0001] The present invention relates to the field of incremental semantic segmentation, and in particular to an incremental semantic segmentation method for correcting biased context information. Background Art

[0002] Semantic segmentation is a classic pixel-level classification problem in computer vision. When given a large-scale pixel-level labeled dataset, deep learning methods have produced amazing results on semantic segmentation tasks. However, in more practical incremental semantic segmentation scenarios, deep neural networks are needed to learn a series of tasks with incremental classes and data. Research on incremental semantic segmentation aims to alleviate the network's forgetfulness of past tasks and over-matching of current tasks without past data. There are two main challenges: first, catastrophic forgetting. When learning the current task, past data cannot be obtained, and the performance of the model on the old task will be greatly reduced; second, semantic drift of background classes. In the current incremental semantic segmentation task, only new class pixels are labeled as semantic classes, while other pixels including old class pixels are labeled as background classes. In the incremental learning process, this semantic change of pixel-level labels will lead to unclear meanings of old class pixels and bring obstacles to correct model predictions. Since semantic segmentation is usually considered a context-dependent pixel-level classification task, we find that there is another specific challenge of incremental semantic segmentation that has not yet attracted attention from the perspective of context. Compared with old images, the context of old class pixels in new images is more biased towards new classes, which may lead to a sharp deterioration of forgetting of old classes and overmatching of new classes. In incremental semantic segmentation, we call this challenge “biased context”.

[0003] Some existing incremental semantic segmentation methods try to use pseudo-labeling technology to solve the above challenges, that is, using the old model obtained in the previous learning step to pseudo-label the old class pixels that are currently mislabeled as background classes. However, the semantic segmentation model updated by this incremental learning method is still prone to bias towards the new class, which is specifically reflected in the following two aspects: first, in the newly acquired image data in the current learning step, the contextual information of the old class pixels is biased towards the new class; second, the number of new class pixels contained in the new image data is much larger than the number of old class pixels. Summary of the invention

[0004] In order to solve the problems existing in the prior art, the present invention aims to study and design a method for correcting biased context information for incremental semantic segmentation. The method constructs an incremental semantic segmentation learning strategy for correcting biased context information, which is insensitive to the biased context information designed for old class pixels in incremental images. Through a consistency loss function that is insensitive to biased context information, the consistency of the context information of old class pixels in a picture pair (composed of an original sample containing new class pixels and an erased sample corresponding to erasing all new class pixels) is maintained, and the context information of old class pixels biased towards the new class can be corrected. The adaptive class-balanced incremental semantic segmentation learning strategy proposed in the present invention can effectively deal with the problem of biased class distribution by increasing the learning weights of old class pixels. The present invention can alleviate the problems of forgetting old class knowledge and semantic drift of background categories.

[0005] The technical solutions specifically adopted in the present invention are as follows:

[0006] An incremental semantic segmentation method with biased context information correction, comprising the following steps:

[0007] S1. Obtain a semantic segmentation data stream containing multiple categories, and divide the data stream into a series of training data sets for semantic segmentation, wherein the image samples in each training data set contain one semantic category or multiple semantic categories, and the semantic categories contained in different training data sets do not overlap;

[0008] S2. In the first incremental semantic segmentation learning step, a semantic segmentation network is trained with one of the training data sets to learn and obtain an initial semantic segmentation network model; the initial semantic segmentation network model is used as the incremental semantic segmentation network model of the next incremental semantic segmentation learning step;

[0009] S3. In the next incremental semantic segmentation learning step, a new training data set is obtained, and the newly obtained training data set containing the new semantic category is used to generate a pair of images corrected with deviation context information by erasing the new category pixels, thereby constructing a training data set corrected with deviation context information, and then based on the training data set corrected with deviation context information, the incremental semantic segmentation network model trained in the previous incremental semantic segmentation learning step is updated by a learning method of deviation context information correction and adaptive class balance;

[0010] S4: When there is a training dataset that has not yet been used for learning the incremental semantic segmentation network model, S3 is repeated continuously to use the training dataset containing the new semantic category to perform incremental learning on the incremental semantic segmentation network model until all training datasets are learned through the incremental semantic segmentation learning steps, and the final incremental semantic segmentation network model is used to perform semantic segmentation tasks.

[0011] Preferably, in S1, the semantic segmentation data stream D containing multiple categories is composed of N training data sets, D = {D1, D2, ..., D N}, where each incremental semantic segmentation learning step corresponds to a training dataset, and the training dataset obtained by the tth incremental semantic segmentation learning step is D t , D t A series of labeled images Composition, of which represents the i-th input image sample of size W×H, express The corresponding correctly labeled segmentation label, D t It is only acquired in the tth incremental semantic segmentation learning step and used for training the semantic segmentation network model; C t is the set of semantic categories obtained in the tth incremental semantic segmentation learning step. The data categories contained in different incremental semantic segmentation learning steps do not overlap, where C0 represents the background category.

[0012] Preferably, the goal of the incremental semantic segmentation method is defined as: in each incremental semantic segmentation learning step, updating the semantic segmentation model, eliminating the biased context information and biased unbalanced class distribution problems of old class pixels, so as to alleviate the forgetting of old class knowledge and the semantic drift of background categories, and realize pixel semantic segmentation in all semantic categories and background category data that have appeared.

[0013] Preferably, in S2, the initial semantic segmentation network model is composed of an encoder-decoder reference network F1 and a convolution head G1; the encoder-decoder reference network is used as a feature extractor to extract a dense feature map F1(x) for an input image sample x; the convolution head is used to generate a segmentation score map S1(x)=G1·F1(x) for a feature map corresponding to a sample, Elements in It indicates that the pixel at (w, h) in sample x is predicted to be scored as category c. The final output of the initial semantic segmentation network is

[0014] In the first incremental semantic segmentation learning step, the semantic segmentation network is trained to minimize the loss function For the purpose of training and learning, the initial semantic segmentation network model is obtained, where the loss function for:

[0015]

[0016] Where: l(x, Θ1) is the cross entropy loss function, |D1| represents the number of images contained in the training data set D1, (x, y) represents the labeled image samples in the training data set, and y is the semantic segmentation label of sample x.

[0017] Preferably, in S3, the method for erasing new class pixels is as follows: first, for a newly acquired training data set D containing a new semantic category, t , the training data set D t The pixels of the new semantic category in each image sample in the image are replaced with black pixels, thereby generating an image pair with the deviation context information corrected. in The original image sample The image corrected by context information, for The corresponding correctly labeled segmentation labels are used to construct a training dataset with corrected deviation context information. in:

[0018]

[0019] Where: |D t | represents the training data set D t The number of images contained in the .

[0020] Preferably, in S3, the learning method for correcting the deviation context information and adaptive class balance is performed by minimizing the loss function This is achieved by updating the incremental semantic segmentation network model trained in the previous incremental semantic segmentation learning;

[0021] The loss function for:

[0022]

[0023] Where: γ is used for balancing and The coefficient of

[0024] Loss Item The calculation formula is:

[0025]

[0026] Where α is used to balance and The coefficient of

[0027] Loss term l bps (x, θ t ) is calculated as:

[0028]

[0029] Where: β is the proportion of old class pixels in the new semantic class sample that are incorrectly labeled as background class, C 0：t is the set of semantic categories and background categories obtained in the first t incremental semantic segmentation learning steps, S t-1 and S t represents the incremental semantic segmentation model for the t-1th and tth incremental learning steps, Indicates S t Pseudo label output for sample x; To use the standard true label y and pseudo label The corrected segmentation labels, and Respectively based on and The pixel at (w, h) in the determined sample x is predicted to be the score of category c; η w，h (x) is the weight of the pixel of sample x at position (w, h), and the calculation formula is:

[0030]

[0031] Where N old (x) is the number of old class pixels belonging to the old semantic category in sample x, N new (x) is the number of new class pixels belonging to the new semantic category in sample x, σ(·) is the Sigmoid function, and O(x) is the position set of old class pixels in sample x;

[0032] Loss Item The calculation formula is:

[0033]

[0034] Where: Indicates based on Confirmed samples The pixel at position (w, h) in is predicted to be scored as category c;

[0035] Loss term l kd (x, θ t ) is calculated as:

[0036] l kd (x, θ t )=||Φ(F t (x))-Φ(F t-1 (x))|| 2

[0037] In the formula For concatenation operation:

[0038]

[0039] Where F ：，w，： (x) represents the eigenvalue of the second dimension w in the dense feature map extracted from sample x; F t (x) and F t-1 (x) denotes F(x) corresponding to the t-1th and tth incremental semantic segmentation learning steps, respectively; [·||·] denotes the concatenation operation in the channel dimension.

[0040] Loss Item The calculation formula is:

[0041]

[0042] Where: C 1：t-1 is the set of semantic categories obtained in the first t-1 incremental semantic segmentation learning steps.

[0043] Preferably, in addition to the semantic segmentation data stream containing multiple categories obtained in S1, new image samples are continuously obtained from the outside. If image samples containing new semantic categories are obtained, they are constructed as new training data sets, and the latest incremental semantic segmentation network model is continuously incrementally learned according to S3. The incrementally learned model is used to perform semantic segmentation tasks on images containing new semantic categories.

[0044] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0045] Compared with the traditional incremental semantic segmentation method for deep neural network models, the present invention provides a method for correcting biased context information for incremental semantic segmentation, which can effectively correct the context information of old class pixels that is biased towards new classes and alleviate the biased class distribution problem, reduce the forgetting of old class knowledge and the semantic drift of background categories, and is simple to implement. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 Schematic diagram of the incremental semantic segmentation method corrected for biased context information;

[0047] Figure 2 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0048] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments.

[0049] The principle of the incremental semantic segmentation method with deviation context information correction provided by the present invention is as follows Figure 1As shown: first, in the first incremental semantic segmentation learning step, an initial semantic segmentation network model is learned on a training dataset containing one or several semantic categories, and the incremental semantic segmentation network models are initialized with the initial semantic segmentation network model; when entering the second incremental semantic segmentation learning step, only the newly arrived training dataset containing new categories and the incremental semantic segmentation network model obtained in the previous learning step can be obtained, and the new category pixels are erased for the newly acquired training dataset containing new categories to generate image pairs with bias context information correction, and a training dataset with bias context information correction is constructed. The incremental semantic segmentation network model is updated based on the training dataset corrected with bias context information by using a learning method of bias context information correction and adaptive class balance; similarly, each time a dataset containing new semantic categories arrives, that is, when entering the next incremental semantic segmentation learning step, the model is updated based on the training dataset corrected with bias context information by using a learning method of bias context information correction and adaptive class balance, and a new incremental semantic segmentation network model can be obtained. In this way, the context information of the old class pixels that is biased towards the new class can be effectively corrected and the biased class distribution problem can be alleviated, thereby reducing the forgetting of old class knowledge and the semantic drift of background classes. The specific implementation process of the present invention is described in detail below.

[0050] like Figure 1 and Figure 2 As shown, an incremental semantic segmentation method for correcting biased context information is provided, comprising the following steps:

[0051] S1. Obtain a semantic segmentation data stream containing multiple categories, and divide the data stream into a series of training data sets for semantic segmentation, wherein the image samples in each training data set contain one semantic category or multiple semantic categories, and the semantic categories contained in different training data sets do not overlap.

[0052] In the present invention, the incremental semantic segmentation learning step is defined as: the time interval from the arrival of the segmentation data containing the current semantic category to the arrival of the segmentation data containing the new semantic category is an incremental semantic segmentation learning step. Each training data set formed by the above division can be divided according to the total number of incremental semantic segmentation learning steps, and each training data set corresponds to an incremental semantic segmentation learning step, so that only one or several categories of semantic segmentation training data sets can be obtained in each incremental semantic segmentation learning step.

[0053] In the present invention, the semantic segmentation data stream containing multiple categories is defined as follows:

[0054] The semantic segmentation data stream D consists of N training data sets, D = {D1, D2, ..., D N}, where each incremental semantic segmentation learning step corresponds to a training dataset, and the training dataset obtained by the tth incremental semantic segmentation learning step is D t , D t A series of labeled images Composition, of which represents the i-th input image sample of size W×H, express The corresponding correctly labeled segmentation label, D t It is only acquired in the tth incremental semantic segmentation learning step and used for training the semantic segmentation network model; C t is the set of semantic categories obtained in the tth incremental semantic segmentation learning step. The data categories contained in different incremental semantic segmentation learning steps do not overlap, where C0 represents the background category.

[0055] The goal of the incremental semantic segmentation method of the present invention is defined as: in each incremental semantic segmentation learning step, the semantic segmentation model is updated to eliminate the biased context information and biased unbalanced class distribution problems of old class pixels, so as to alleviate the forgetting of old class knowledge and the semantic drift of background categories, and realize pixel semantic segmentation in all semantic categories and background category data that have appeared.

[0056] S2. In the first incremental semantic segmentation learning step, a semantic segmentation network is trained with one of the training data sets to learn and obtain an initial semantic segmentation network model; the initial semantic segmentation network model is used as the incremental semantic segmentation network model of the next incremental semantic segmentation learning step.

[0057] In this step, the initial semantic segmentation network model is composed of an encoder-decoder reference network F1 and a convolution head G1. The encoder-decoder reference network adopts an encoder-decoder structure and can be used as a feature extractor to extract a dense feature map F1(x) for an input image sample x; the convolution head is used to generate a segmentation score map S1(x) = G1·F1(x) for the feature map corresponding to a sample. Elements in It indicates that the pixel at (w, h) in sample x is predicted to be scored as category c. The final output of the initial semantic segmentation network is

[0058] In the first incremental semantic segmentation learning step, the semantic segmentation network is trained to minimize the loss function For the purpose of training and learning, the initial semantic segmentation network model is obtained, where the loss function for:

[0059]

[0060] Among them: l(x, Θ1) is the cross entropy loss function, |D1| represents the number of images in the training data set D1; (x, y) represents the labeled image samples in the training data set, and y is the semantic segmentation label of sample x. In this loss function and subsequent loss functions, (x, y) is used as the general representation of the input image sample to be labeled, which is essentially equivalent to

[0061] S3. In the next incremental semantic segmentation learning step, a new training data set is obtained, and the newly obtained training data set containing the new semantic category is used to generate a pair of images corrected for the deviation context information by erasing the new category pixels, thereby constructing a training data set corrected for the deviation context information, and then based on the training data set corrected for the deviation context information, the incremental semantic segmentation network model trained in the previous incremental semantic segmentation learning step is updated by a learning method of deviation context information correction and adaptive class balance.

[0062] It should be noted that the incremental semantic segmentation network model trained in the previous incremental semantic segmentation learning step is different in different incremental semantic segmentation learning steps. Figure 1 As shown, in the second incremental semantic segmentation learning, the incremental semantic segmentation network model trained in the previous incremental semantic segmentation learning step is the initial semantic segmentation network model trained in the first incremental semantic segmentation learning step, and in the third incremental semantic segmentation learning step, the incremental semantic segmentation network model trained in the previous incremental semantic segmentation learning step is the incremental semantic segmentation network model trained in the second incremental semantic segmentation learning step, and so on.

[0063] In this step, the method for erasing new class pixels is as follows:

[0064] First, for the newly acquired training data set D containing new semantic categories t , the training data set D t The pixels of the new semantic category in each image sample in the image are replaced with black pixels, thereby generating an image pair with the deviation context information corrected. in The original image sample The image corrected by context information, for The corresponding correctly labeled segmentation labels are used to construct a training dataset with corrected deviation context information. in:

[0065]

[0066] Where: |Dt | represents the training data set D t The number of images contained in the .

[0067] In addition, in this step, the above-mentioned deviation context information correction and adaptive class balance learning method is to minimize the loss function This is achieved by updating the incremental semantic segmentation network model trained in the previous incremental semantic segmentation learning.

[0068] The loss function for:

[0069]

[0070] Where: γ is used for balancing and The coefficient of .

[0071] For the original sample x and the corresponding erased sample Adopt cross entropy loss function based on pseudo label And the knowledge distillation loss function l kd (x, θ t ). The loss item The calculation formula is:

[0072]

[0073] Where α is used to balance and The coefficient of bps (x, θ t )and It is used to maintain the old category knowledge of the segmentation model and reduce the background offset of the old category pixels. bps (x, θ t ) is the class-balanced pseudo-label-based cross entropy loss function used for the original sample x with unbalanced class distribution. The loss term l bps (x, θ t ) is calculated as:

[0074]

[0075] Where: β is the proportion of old class pixels in the new semantic class sample that are incorrectly labeled as background class, C 0：t is the set of semantic categories and background categories obtained in the first t incremental semantic segmentation learning steps, S t-1 and S t represents the incremental semantic segmentation model for the t-1th and tth incremental learning steps, Indicates S tPseudo label output for sample x; To use the standard true label y and pseudo label The corrected segmentation labels, and Respectively based on and The pixel at (w, h) in the determined sample x is predicted to be the score of category c; η w，h (x) is the weight of the pixel of sample x at position (w, h), and the calculation formula is:

[0076]

[0077] Where N old (x) is the number of old class pixels belonging to the old semantic category in sample x, N new (x) is the number of new class pixels belonging to the new semantic category in sample x, σ(·) is the Sigmoid function, and O(x) is the position set of old class pixels in sample x.

[0078] is an erased sample containing only pixels of the old category The pseudo-label-based cross entropy loss function is used. The calculation formula is:

[0079]

[0080] Where: Indicates based on Confirmed samples The pixel at position (w, h) in is predicted to be scored as category c; Definition The only difference is that the input sample is instead of x.

[0081] l kd (x, θ t ) is used to maintain the reference network F t-1 The old class knowledge, the loss term l kd (x, θ t ) is calculated as:

[0082] l kd (x, θ t )=||Φ(F t (x))-Φ(F t-1 (x))|| 2

[0083] In the formula For concatenation operation:

[0084]

[0085] Where F ：，w，： (x) represents the eigenvalue of the second dimension w in the dense feature map extracted from sample x; F t (x) and F t-1 (x) denotes F(x) corresponding to the t-1th and tth incremental semantic segmentation learning steps, respectively; [·||·] denotes the concatenation operation in the channel dimension.

[0086] In the formula This loss term is used to maintain the prediction consistency between the original sample and the old class pixels in the corresponding erased sample. The calculation formula is:

[0087]

[0088] Where: C 1：t-1 is the set of semantic categories obtained in the first t-1 incremental semantic segmentation learning steps. The semantic categories included in different incremental semantic segmentation learning steps do not overlap. O(x) is the set of pixel positions in the sample where x belongs to the old semantic category. |O(x)| represents the number of pixels of the old semantic category contained in the pixel position set O(x).

[0089] It should be noted that the new semantic categories and old semantic categories in this step are distinguished based on whether they have been learned by the model in the previously completed incremental semantic segmentation learning step. The semantic categories that have been learned by the incremental semantic segmentation network model are the old semantic categories, and the semantic categories that have not been learned by the incremental semantic segmentation network model are the new semantic categories. For ease of description, samples and pixels with new semantic categories are called new class sample points and new class pixel points, respectively, and samples and pixels with old semantic categories are called old class sample points and old class pixel points, respectively.

[0090] S4: When there is a training data set that has not been used for learning the incremental semantic segmentation network model, S3 is repeated continuously to use the training data set containing new semantic categories to incrementally learn the incremental semantic segmentation network model. Each execution of S3 is regarded as an incremental semantic segmentation learning step. Through this incremental semantic segmentation learning step, one or more new semantic categories can be learned. The learned incremental semantic segmentation network model can be used to perform semantic segmentation on samples containing these new semantic categories. After executing N incremental semantic segmentation learning steps in total, all N training data sets in the semantic segmentation data stream D are learned through these N incremental semantic segmentation learning steps. Therefore, the final incremental semantic segmentation network model can be used to perform semantic segmentation tasks on images of all known semantic categories.

[0091] However, in actual applications, in addition to the semantic segmentation data stream containing multiple categories obtained in S1, image samples containing updated semantic categories may still be generated later. Therefore, the present invention can continuously obtain new image samples from the outside. If image samples containing new semantic categories are obtained, they will continue to be constructed as new training data sets, and then the latest incremental semantic segmentation network model will be incrementally learned again according to S3. The model after incremental learning is used to perform semantic segmentation tasks on images containing new semantic categories, thereby ensuring real-time updating of the model.

[0092] The above method is applied to a specific embodiment below so that those skilled in the art can better understand the effect of the present invention.

[0093] Example

[0094] The following simulation experiment is carried out based on the incremental semantic segmentation method corrected by the above-mentioned deviation context information. The implementation method of this embodiment is as described in S1 to S4 above, and the specific steps are not elaborated in detail. The following only shows its effect based on the experimental results.

[0095] This embodiment uses the original complex Deeplab-V3 network for semantic segmentation tasks on the PASCAL VOC dataset to carry out incremental semantic segmentation tasks. On the PASCAL VOC dataset, there are three task scenarios, namely VOC19-1, VOC15-5, and VOC15-1. In the VOC19-1 scenario, there are 2 incremental semantic segmentation learning steps, and the training data reached in the first incremental semantic segmentation learning step contains 19 semantic categories, and the data set reached in each subsequent incremental semantic segmentation learning step contains 1 semantic category; in the VOC15-5 scenario, there are 2 incremental semantic segmentation learning steps, and the training data reached in the first incremental semantic segmentation learning step contains 15 semantic categories, and the data set reached in each subsequent incremental semantic segmentation learning step contains 5 semantic categories; in the VOC15-1 scenario, there are 6 incremental semantic segmentation learning steps, and the training data reached in the first incremental semantic segmentation learning step contains 15 semantic categories, and the data set reached in each subsequent incremental semantic segmentation learning step contains 1 semantic category; the implementation effect of this embodiment is shown in Table 1.

[0096] Table 1 Implementation effect of the method of the present invention on the PASCAL VOC dataset

[0097]

[0098] As shown in Table 1, on the PASCAL VOC dataset, the average intersection over union (mIoU) of the model of ours in the VOC19-1 scenario reaches 75.01%, surpassing the results of the most advanced methods (PLOP, SDR). In the VOC15-5 and VOC15-1 scenarios, the effect of ours after each incremental semantic segmentation learning step surpasses the most advanced methods. In principle, the correction of biased context information in the incremental semantic segmentation learning process can not only reduce the forgetting of old class knowledge, but also better learn new class knowledge. Compared with the results of other advanced methods, it can be observed that the average intersection over union of the model of the proposed method after the incremental semantic segmentation learning session is the highest.

[0099] In summary, the embodiment of the present invention is different from the prior art in that the model is updated through the learning method of bias context information correction and adaptive class balance in each incremental semantic segmentation learning step. Compared with the prior art, it can reduce the forgetting of old task knowledge and prevent overfitting of new task knowledge, and the average intersection-over-union ratio of the model on all semantic categories is higher.

[0100] It should be understood that the present embodiment is only used to illustrate the present invention and is not used to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms also fall within the scope limited by the appended claims of the application.

Claims

1. An incremental semantic segmentation method with biased context information correction, characterized in that: The following steps are involved: S1. Obtain a semantic segmentation data stream containing multiple categories, and divide the data stream into a series of training data sets for semantic segmentation, wherein the image samples in each training data set contain one semantic category or multiple semantic categories, and the semantic categories contained in different training data sets do not overlap; S2. In the first incremental semantic segmentation learning step, a semantic segmentation network is trained with one of the training data sets to learn and obtain an initial semantic segmentation network model; the initial semantic segmentation network model is used as the incremental semantic segmentation network model of the next incremental semantic segmentation learning step; S3. In the next incremental semantic segmentation learning step, a new training data set is obtained, and the newly obtained training data set containing the new semantic category is used to generate a pair of images corrected with deviation context information by erasing the new category pixels, thereby constructing a training data set corrected with deviation context information, and then based on the training data set corrected with deviation context information, the incremental semantic segmentation network model trained in the previous incremental semantic segmentation learning step is updated by a learning method of deviation context information correction and adaptive class balance; S4: When there is a training data set that has not been used for learning the incremental semantic segmentation network model, S3 is repeated continuously to use the training data set containing the new semantic category to incrementally learn the incremental semantic segmentation network model until all training data sets are learned through the incremental semantic segmentation learning steps, and the resulting incremental semantic segmentation network model is used to perform the semantic segmentation task; In S2, the initial semantic segmentation network model is composed of an encoder-decoder reference network F1 and a convolution head G1; the encoder-decoder reference network is used as a feature extractor to extract a dense feature map F1(x) for an input image sample x; the convolution head is used to generate a segmentation score map S1(x)=G1·F1(x) for a feature map corresponding to a sample, Elements in It indicates that the pixel at (w,h) in sample x is predicted to be scored as category c. The final output of the initial semantic segmentation network is In the first incremental semantic segmentation learning step, the semantic segmentation network is trained to minimize the loss function For the purpose of training and learning, the initial semantic segmentation network model is obtained, where the loss function for: Where: l(x, Θ1) is the cross entropy loss function, |D1| represents the number of images contained in the training data set D1, (x, y) represents the labeled image samples in the training data set, and y is the semantic segmentation label of sample x; In S3, the method for erasing new class pixels is as follows: first, for a newly obtained training data set D containing new semantic categories, t , the training data set D t The pixels of the new semantic category in each image sample in the image are replaced with black pixels, thereby generating an image pair with the deviation context information corrected. in The original image sample The image corrected by context information, for The corresponding correctly labeled segmentation labels are used to construct a training dataset with corrected deviation context information. in: Where: |D t | represents the training data set D t The number of images contained in the .

2. The incremental semantic segmentation method with bias context information correction according to claim 1, characterized in that: In S1, the semantic segmentation data stream D containing multiple categories is composed of N training data sets, D = {D1, D2, ..., D N }, where each incremental semantic segmentation learning step corresponds to a training dataset, and the training dataset obtained by the tth incremental semantic segmentation learning step is D t , D t A series of labeled images Composition, of which represents the i-th input image sample of size W×H, express The corresponding correctly labeled segmentation label, D t It is only acquired in the tth incremental semantic segmentation learning step and used for the training and learning of the semantic segmentation network model; C t is the set of semantic categories obtained in the tth incremental semantic segmentation learning step. The data categories contained in different incremental semantic segmentation learning steps do not overlap, where C0 represents the background category.

3. The incremental semantic segmentation method with bias context information correction according to claim 1, characterized in that: The goal of the incremental semantic segmentation method is defined as: in each incremental semantic segmentation learning step, the semantic segmentation model is updated to eliminate the biased context information and biased imbalanced class distribution problems of old class pixels, so as to alleviate the forgetting of old class knowledge and the semantic shift of background categories, and realize pixel semantic segmentation in all semantic categories and background category data that have appeared.

4. The incremental semantic segmentation method with bias context information correction according to claim 1, characterized in that: In S3, the bias context information correction and adaptive class balancing learning method is implemented by minimizing the loss function This is achieved by updating the incremental semantic segmentation network model trained in the previous incremental semantic segmentation learning; The loss function for: Where: γ is used for balancing and The coefficient of Loss Item The calculation formula is: Where α is used to balance and The coefficient of Loss term l bps (x,Θ t ) is calculated as: Where: β is the proportion of old class pixels in the new semantic class sample that are incorrectly labeled as background class, C 0:t is the set of semantic categories and background categories obtained in the first t incremental semantic segmentation learning steps, S t-1 and S t represents the incremental semantic segmentation model of the t-1th and tth incremental learning steps, where the pixel at (w,h) in sample x is predicted to have a score of category c Where N old (x) is the number of old class pixels belonging to the old semantic category in sample x, N new (x) is the number of new class pixels belonging to the new semantic category in sample x, σ(·) is the Sigmoid function, and O(x) is the position set of old class pixels in sample x; Loss Item The calculation formula is: Where: Indicates based on Confirmed samples The pixel at position (w,h) in is predicted to be scored as category c; Loss term l kd (x,Θ t ) is calculated as: l kd (x,Θ t )=||Φ(F t (x))-Φ(F t-1 (x))|| 2 In the formula For concatenation operation: Where F :,w,: (x) represents the eigenvalue of the second dimension w in the dense feature map extracted from sample x; F t (x) and F t-1 (x) represents F(x) corresponding to the t-1th and tth incremental semantic segmentation learning steps, respectively; [·||·] represents the concatenation operation in the channel dimension; Loss Item The calculation formula is: Where: C 1:t-1 is the set of semantic categories obtained in the first t-1 incremental semantic segmentation learning steps.

5. The incremental semantic segmentation method with bias context information correction according to claim 1, characterized in that: In addition to the semantic segmentation data stream containing multiple categories obtained in S1, new image samples are continuously obtained from the outside. If image samples containing new semantic categories are obtained, they are constructed as new training data sets, and the latest incremental semantic segmentation network model is continuously incrementally learned according to S3. The incrementally learned model is used to perform semantic segmentation tasks on images containing new semantic categories.

Images