Image increment semantic segmentation method based on image-level posterior guidance and knowledge decoupling

Abstract

The invention provides an image increment semantic segmentation method based on image-level posterior guidance and knowledge decoupling. The method comprises the following steps: an image-level posterior guidance branch processes a feature map of original image data by using dynamic average pooling and a multilayer perceptron to obtain image-level category information; the permanent learning branch processes the feature map by using a convolutional layer and an up-sampling layer to obtain a segmentation result of the permanent learning branch; the temporary learning branch learns category knowledge existing in the feature map in the current incremental stage by using an incremental learning head to obtain a segmentation result of the temporary learning branch; integrating the segmentation results of the permanent learning branch and the temporary learning branch to obtain an aggregated intermediate prediction result; and optimizing the aggregated intermediate prediction result by using image-level category information to obtain an incremental semantic segmentation result. According to the method, complex image category knowledge is decoupled into a permanent static concept and a temporary dynamic concept, and the semantic offset resistance of the model is effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of class incremental learning technology, and in particular to an image incremental semantic segmentation method based on image-level posterior guidance and knowledge decoupling. Background Art

[0002] Continuous learning (CL) originates from cognitive neuroscience research on the mechanisms of memory and forgetting. It is a method that focuses on sequential acquisition of knowledge. It is also called incremental learning (IL) or lifelong learning. It has made significant progress in the past decade. Traditional machine learning is usually based on fixed predefined data sets and cannot cope with new data and tasks that appear continuously. The core challenge of continuous learning is how to balance stability and plasticity of the model under limited computing and storage resources. Stability refers to the ability of the model to retain previous knowledge, and plasticity refers to the ability of the model to integrate new knowledge.

[0003] Class Incremental Learning (CIL) is a method that continuously acquires knowledge in a categorical order, aiming to alleviate the problem of catastrophic forgetting while continuously learning new categories. Existing research can be divided into three main categories: Replay methods involve storing data or features of old categories, or generating data containing old categories to alleviate catastrophic forgetting. This category of methods can be further divided into experience replay, generation replay, and feature replay. Regularization methods focus on designing loss functions that contain penalties based on the contribution of parameters to different tasks. Such methods also rely on knowledge distillation, usually using the model of the previous stage as a teacher model to constrain the model of the current stage. Dynamic architecture methods dynamically adjust model parameters according to new data, including assigning specific parameters to different data and decomposing model parameters into task-specific or shared parts.

[0004] Class Incremental Semantic Segmentation (CISS) is similar to class incremental learning, but extends the task to pixel-level prediction. Class incremental semantic segmentation faces the challenge of semantic drift, which refers to the gradual change and evolution of the content contained in background categories as new classes are gradually learned. Semantic drift is usually caused by mixed semantics between background, old classes, and future classes.

[0005] Regarding the study of incremental semantic segmentation, some scholars first adopted a distillation strategy to alleviate this problem. Other scholars used pseudo-labeling technology for incremental segmentation to solve the problem of background transfer. Some scholars further combined saliency information, introduced the concept of "unknown categories" in each learning stage, and used a memory pool to store old data to prevent catastrophic forgetting. Some scholars used proposal generators to simulate unseen categories. Some scholars emphasized the differences within and between classes, and designed a contrast loss to adjust the feature distribution of the class. Some scholars emphasized pre-learning of future knowledge to enhance the model's ability to distinguish between new and old categories.

[0006] Most of the incremental semantic segmentation methods in the above-mentioned prior art adopt a dynamic architecture, and different parameters are assigned to categories at different stages to learn new knowledge. However, there is a lack of unified and global optimization methods in the model output stage. This independent optimization method is prone to strong responses to errors in old categories, seriously disrupting the model's ability to make correct predictions.

[0007] The incremental semantic segmentation methods in the above-mentioned prior art only partially alleviate the semantic offset problem and mostly focus on the foreground area of ​​the image, but do not solve the impact of noise semantic information caused by unknown categories in the background on the model prediction effect. Summary of the invention

[0008] The present invention provides an image incremental semantic segmentation method based on image-level posterior guidance and knowledge decoupling, so as to realize effective incremental semantic segmentation processing of images.

[0009] In order to achieve the above object, the present invention adopts the following technical scheme.

[0010] An image incremental semantic segmentation method based on image-level posterior guidance and knowledge decoupling, comprising:

[0011] Performing data enhancement processing on the original image data, obtaining a feature map from the image data after the data enhancement processing using a feature extractor, and inputting the feature map into the image-level posterior guidance branch, the permanent learning branch, and the temporary learning branch in parallel;

[0012] The image-level posterior guidance branch processes the feature map using dynamic average pooling and a multi-layer perceptron to obtain image-level category information;

[0013] The permanent learning branch processes the feature map using a convolution layer and an upsampling layer to obtain a segmentation result of the permanent learning branch;

[0014] The temporary learning branch uses the incremental learning head to learn the category knowledge existing in the feature map in the current incremental stage to obtain the segmentation result of the temporary learning branch;

[0015] Integrate the segmentation result of the permanent learning branch and the segmentation result of the temporary learning branch to obtain an aggregated intermediate prediction result;

[0016] The aggregated intermediate prediction result is optimized using the image-level category information to obtain an incremental semantic segmentation result of the original image data.

[0017] Preferably, the data enhancement processing is performed on the original image data, a feature map is obtained from the image data after the data enhancement processing by using a feature extractor, and the feature map is input into the image-level posterior guidance branch, the permanent learning branch and the temporary learning branch in parallel, including:

[0018] In each stage of incremental learning, the image data from the corresponding stage and the image data from the memory buffer are received, the input original image data is flipped, randomly cropped and data enhanced, and the image data after data enhancement is input into the system model. The system model uses the feature extractor to obtain the feature map from the image data. The feature map contains various abstract information of the image.

[0019] The feature maps are input in parallel into the following image-level posterior guidance branch, permanent learning branch and temporary learning branch.

[0020] Preferably, the image-level posterior guidance branch processes the feature map using dynamic average pooling and a multi-layer perceptron to obtain image-level category information, including:

[0021] After the feature map is input into the image-level posterior guidance branch, it is flattened into a one-dimensional vector through dynamic average pooling and then input into the multi-layer perceptron to be processed into a shape of [B,C 1:t ]’s category prediction results Where B represents the batch size, C 1:t Represents all categories that the model has touched up to the current incremental stage;

[0022] Use the category labels when training the image-level posterior guidance branch model calculate Constrain and calculate the binary cross entropy loss function;

[0023]

[0024] Among them, BCE represents the binary cross entropy loss function, φ c represents the classification head of the branch, Represents the pseudo category information obtained by the image after passing through the model in the previous stage.

[0025] Preferably, the permanent learning branch processes the feature map using a convolution layer and an upsampling layer to obtain a segmentation result of the permanent learning branch, including:

[0026] Reassign the image labels of the original image data as:

[0027]

[0028] in, represents the complement operator, φ t―1 (x i ) represents the prediction result of the previous model, S(·) is the saliency detector, is assigned to the permanent learning branch φ p Labels;

[0029] Input the feature map into the permanent learning branch φ p After that, the feature map passes through the permanent learning branch φ p After three convolutional layers and one upsampling layer, the pure background c′ is generated b and fixed foreground c f The segmentation result of the permanent learning branch φ p The output data is The shape is [B, 2, H, W], where B, H, W represent the batch size, the height and width of the input image, respectively, and 2 represents the dimensions of pure background and fixed foreground;

[0030] In the permanent learning branch φ p During the training process, pixel-level labels are used Calculate using the following formula (3) Make constraints and calculate the binary cross entropy loss function;

[0031]

[0032] Preferably, the temporary learning branch uses the incremental learning head to learn the category knowledge existing in the feature map in the current incremental stage to obtain the segmentation result of the temporary learning branch, including:

[0033] The image labels of the original image data are additionally reassigned as:

[0034]

[0035] in, represents the complement operator and S(·) is the saliency detector used in the previous work SSUL. is the incremental learning head corresponding to the current task t Labels;

[0036] The incremental learning head As a temporary learning branch, the temporary learning branch is used to learn the category knowledge C existing in the current incremental stage. t After the feature map passes through the three convolutional layers and one upsampling layer of the current incremental learning head, it generates the current stage category C t , the rest of the foreground area (c f ―C t ), pure background area c′ b The segmentation result of the temporary learning branch is The shape is [B,C t +2,H,W], where B,H,W represent the batch size, the height and width of the input image respectively; during the training process of the temporary learning branch, pixel-level labels are used Calculate using the following formula (5) Make constraints and calculate the binary cross entropy loss function;

[0037]

[0038] Among them, C inner Indicates the foreground category corresponding to each incremental stage.

[0039] Preferably, the step of integrating the segmentation result of the permanent learning branch and the segmentation result of the temporary learning branch to obtain an aggregated intermediate prediction result includes:

[0040] From the permanent learning branch φ p The output data [B,2,H,W] of each temporary branch is used to extract the segmentation result [B,1,H,W] for the pure background. i +2,H,W]} i＝1:t Extract the segmentation results for each category {[B,C i ,H,W]} i＝1:t , after integration, we get the aggregated intermediate prediction results The shape is [B,C 1:t +1,H,W].

[0041] Preferably, the step of optimizing the aggregated intermediate prediction result by using the image-level category information to obtain the incremental semantic segmentation result of the original image data includes:

[0042] Image-level category information provided by the image-level posterior guidance branch The shape is [B,C 1:t ], after adding the background dimension, it becomes [B,C 1:t +1], and further expand the dimension to [B,C 1:t +1,1,1], change [B,C 1:t+1,1,1] and the shape is [B,C 1:t +1,H,W] intermediate prediction result Performing element-by-element multiplication to obtain an incremental semantic segmentation result of the original image data, the incremental semantic segmentation result including segmentation results of a background area and each category area in the image;

[0043] The next round of system iteration processing is performed, and the above processing process is repeated until all incremental learning stages are completed to obtain the final incremental semantic segmentation result of the original image data.

[0044] It can be seen from the technical solutions provided by the above embodiments of the present invention that the present invention uses image category information to guide incremental semantic segmentation tasks, applies image-level knowledge to pixel-level tasks, and solves the problem of semantic drift. The method of the present invention decouples complex image category knowledge into permanent static concepts and temporary dynamic concepts, divides and conquers, and finally integrates the output results, effectively improving the model's ability to resist semantic drift.

[0045] Additional aspects and advantages of the present invention will be given in part in the following description, which will become obvious from the following description, or may be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0047] Figure 1 A schematic diagram of an implementation principle of an image incremental semantic segmentation method based on image-level posterior guidance and knowledge decoupling provided in an embodiment of the present invention;

[0048] Figure 2 A processing flow chart of an image incremental semantic segmentation method based on image-level posterior guidance and knowledge decoupling provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0049] The embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention.

[0050] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, the "connection" or "coupling" used herein may include wireless connection or coupling. The term "and / or" used herein includes any unit and all combinations of one or more associated listed items.

[0051] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless defined as herein.

[0052] To facilitate understanding of the embodiments of the present invention, several specific embodiments will be further explained below with reference to the accompanying drawings, and each embodiment does not constitute a limitation on the embodiments of the present invention.

[0053] To address the problem of erroneous strong responses caused by independent optimization, the present invention designs an image-level posterior guidance strategy, which fully utilizes the image-level posterior category information as global guidance, strengthens the association between different tasks, and constrains the model's predictions for all categories to the same scale.

[0054] In response to the impact of noise semantics, the present invention proposes a knowledge and temporary knowledge decoupling strategy (Permanent-Tmporary Knowledge Decoupling), which decouples noise semantics into simple, static, permanent semantics and complex, dynamic, temporary semantic information, and uses different branches for learning, divide and conquer, which effectively alleviates the semantic drift problem and improves the model effect.

[0055] The present invention uses two strategies to solve the semantic drift problem: image-level posterior guidance and permanent and temporary knowledge decoupling. The implementation principle diagram of an image incremental semantic segmentation method based on image-level posterior guidance and knowledge decoupling provided by an embodiment of the present invention is shown in FIG. Figure 1As shown in the figure. The image-level posterior guidance branch uses a small amount of old task training samples with rich categories stored in the memory mechanism to explore the potential guidance capabilities implied by the global category information of these training samples, and uses additional class increment classifiers to make up for the limitations of independent optimization. At the same time, the knowledge decoupling branch decouples complex semantic knowledge into permanent and temporary knowledge to alleviate the impact of semantic drift.

[0056] Image-level posterior guidance: Independent optimization will lead to inconsistent model output scales, and the direct way to solve this problem is to scale the image using global statistics. To this end, the present invention proposes to use image-level posterior probabilities as global guidance information, based on the following facts:

[0057] For any image, if its image-level category domain is C I , the pixel-level category domain is C P , then the category domain C I and C P are the same, that is, C I =C P .

[0058] Therefore, the embodiment of the present invention designs a branch to predict the category of the image and train it in an incremental learning manner. The current task and the total task are denoted as t and T respectively. This branch is implemented using a simple multi-layer perceptron (MLP) with an input dimension of 4096 and an output dimension of |C 1:t |. During testing, it will predict and output the entire category set C 1:T The image-level posterior probability of .

[0059] In task t (t>1), the model accesses the training data D from the memory buffer M and the current task t The image from the memory buffer is denoted by It comes with the corresponding image-level true category label Its category label depends on the specific task, for example: if So The current task image is recorded as It has a class from the current task domain C t Image-level category labels

[0060] During the incremental training of task t, the data samples x from these two sources k By the network backbone θ Processed into image feature h θ (x k). This feature is further processed by Adaptive Average Pooling and a Multi-Layer Perceptron (MLP) to generate the final prediction Parameter optimization is achieved by the following formula:

[0061]

[0062] Among them, BCE represents the binary cross entropy loss function, φ c represents the classification head of the branch, Indicates the pseudo category information obtained by the model in the previous stage, compared with directly using part of the category label y k , pseudo labels are preferred here Because it can provide richer supervision information.

[0063] During testing, the image posterior branch predicts all classes C 1:t―1 The posterior probability is used as a guide. At test time, for the test image x i The final predicted segmentation result is determined by the image posterior probability and from the segmentation header The intermediate result of the segmentation is obtained by performing element-by-element multiplication between the two:

[0064]

[0065] Among them, σ is the Sigmoid function, α BR It is a hyperparameter used to compensate for the missing background category in the posterior probability. In the present invention, α is set BR =0.9. The result p i is a tensor of shape [C, HW], p i,j represents the C-dimensional prediction result of the j-th pixel. The prediction of the j-th pixel is:

[0066]

[0067] Permanent and temporary knowledge decoupling: In order to further solve the problem of disordered background categories c b , unknown category that is difficult to determine c u and the correct but incomplete current category C t To solve the semantic drift problem caused by the coupled learning of the three, the present invention proposes a decoupling method for separating different categories of knowledge based on semantic content.

[0068] Compared with previous work that classified foreground categories, the method of the present invention focuses on dividing all categories in the image into two categories: static category groups and dynamic category groups. And it is based on the following considerations: in different incremental stages, the foreground categories in the image will change dynamically, for example: the foreground category set in the current task may be "cat, horse" (corresponding to the category concepts that need to be learned in this stage), and in the next task it will become "cow, sofa" (corresponding to the category concepts to be learned in the next stage). At the same time, there are also static group categories in the image that can span different incremental stages, such as background and some fixed foreground concepts.

[0069] Therefore, all categories can be decomposed into two groups: c′ b ∪c f and c b ∪C t , where c′ b and c f denote pure background and fixed foreground concepts, respectively, which represent permanent concepts that can span the entire incremental phase. b and C t It includes the categories and corresponding backgrounds that need to be learned at the current stage, representing temporary concepts that will change with different stages.

[0070] In order to better learn these two concepts, the present invention uses two independent branches for training: based on the existing incremental architecture, a permanent learning branch φ is introduced. p to learn permanent concepts and transform the existing incremental learning head As a temporary learning branch, it is only used to learn the category knowledge C existing in the current incremental stage t . Permanent learning branch φ p The life cycle of spans the entire incremental phase, from the 1st task to the Tth task, while the temporary learning branch Will only be updated at task t.

[0071] Based on the above decoupling strategy, the image label x i Redistribute to:

[0072]

[0073] in, represents the complement operator and S(·) is the saliency detector used in the previous work SSUL. is assigned to the permanent learning branch φ p The label, is the incremental learning head corresponding to the current task t .

[0074] The optimization objectives of these two branches are:

[0075]

[0076]

[0077] Among them, C inner Indicates the foreground category corresponding to each incremental stage.

[0078] The overall optimization goal can be expressed as:

[0079]

[0080] Among them, λ 1 and λ 2 To balance the hyperparameters of different training functions.

[0081] In addition to the above unique design, the present invention also improves the memory buffer M which plays an important role. The memory buffer stores samples or categories learned in the past with a fixed small capacity, which is crucial to improving incremental learning. The present invention improves the utilization efficiency of the memory buffer through class-balanced sampling strategy and storage cost optimization.

[0082] Class-balanced design: To ensure that the image-level posterior branch has sufficient access to samples of all categories and is not biased towards specific categories during training, a class-balanced strategy is used to select representative samples: For a given memory size |M| and the number of known categories |C 1:t |, ensure that each category has at least |M| / |C samples 1:t In addition, the number of categories and the number of pixels of foreground objects are also taken into account in the selection criteria to further ensure balance.

[0083] Storage cost optimization: The incremental semantic segmentation method designed by the present invention only requires image-level labels and saliency masks for samples in the memory buffer. The image-level labels will be used as constraint information in the image-level posterior branch, while the saliency masks will be used in the permanent and temporary learning branches. It only needs to divide the image into background and foreground objects and store them with 0 and 1 marks respectively. Compared with the previous method of storing pixel-level annotations of all categories, this storage simplification mechanism effectively reduces storage costs.

[0084] The processing flow chart of an image incremental semantic segmentation method based on image-level posterior guidance and knowledge decoupling provided by an embodiment of the present invention is as follows: Figure 2 As shown, the processing steps include the following:

[0085] Step S1: Data input: At each stage of incremental learning, the model receives image data from the corresponding stage and image data from the memory buffer.

[0086] Step S:2: Data processing: The input image is scaled to 513×513 after flipping, random cropping and other data enhancement processes.

[0087] Step S3: Feature extraction: The image data after data enhancement is input into the system model. The system model first uses a feature extractor to obtain a feature map from the image data. The feature map contains various abstract information of the image and is crucial for subsequent model prediction. The above feature map is input in parallel into the following image-level posterior guidance branch, permanent learning branch, and temporary learning branch.

[0088] Step S4: Image-level posterior guidance branch: After the feature map enters this branch, it will go through the network structure such as Adaptive Average Pooling and Multi-layer Perceptron (MLP) to obtain the image-level category information. calculate To constrain.

[0089] After the feature map enters this branch, it first undergoes adaptive average pooling, then flattens into a one-dimensional vector, and then inputs into the MLP (Multilayer Perceptron), and is finally processed into a shape of [B, C 1:t ]’s category prediction results Where B represents the batch size, C 1:t Represents all categories that the model has contacted up to the current incremental stage.

[0090] Use the category labels when training the image-level posterior guidance branch model calculate Constrain and calculate the binary cross entropy loss function;

[0091]

[0092] Among them, BCE represents the binary cross entropy loss function, φ c represents the classification head of the branch, Indicates the pseudo category information obtained by the model in the previous stage, compared with directly using part of the category label y k , pseudo labels are preferred here Because it can provide richer supervision information.

[0093] Step S5: Permanent learning branch: Introduce a permanent learning branch φ based on the existing architecturep , for background and some fixed foreground category concepts that can span different incremental stages, the pixel-level labels calculate To constrain.

[0094] The feature map is passed through the permanent learning branch φ p After three convolutional layers and one upsampling layer, the pure background c′ is generated b and fixed foreground c f The segmentation result Permanent learning branch φ p The output data shape is [B, 2, H, W], where B, H, W represent the batch size, the height and width of the input image, respectively, and 2 represents the two dimensions of pure background and fixed foreground.

[0095] During training, the model uses pixel-level labels Using the above formula (7) to calculate Constraints are imposed and the binary cross entropy (BCE) loss function is calculated. During inference, only the segmentation results of the pure background are merged into the intermediate prediction results.

[0096] Step S6: Temporary learning branch: convert the existing incremental learning head As a temporary learning branch, it is used to learn the category knowledge C existing in the current incremental stage t , by pixel-level label calculate To constrain.

[0097] After the feature map passes through the three convolutional layers and one upsampling layer of the current incremental learning head, it generates the current stage category C t , the rest of the foreground area (c f ―C t ), pure background area c′ b The segmentation result of the temporary learning branch is [B,C t +2,H,W], where B,H,W represent the batch size and the height and width of the input image respectively.

[0098] After the feature map passes through the three convolutional layers and one upsampling layer of the current incremental learning head, it generates the current stage category C t , the rest of the foreground area (c f ―C t ), pure background area c′ b The segmentation result The output data shape of the temporary learning branch is [B,C t +2,H,W], where B,H,W represent the batch size, the height and width of the input image, respectively, and C t+2 indicates the category of the current incremental stage and the additional remaining foreground and pure background dimensions.

[0099] During the training process, the model integrates the segmentation results of each stage category by the incremental learning head at each stage, and uses pixel-level labels Using the above formula (6) to calculate Constraints are imposed and the binary cross entropy (BCE) loss function is calculated. During inference, only the segmentation results of the categories at each stage will be merged into the intermediate prediction results.

[0100] Step S7: Result integration: Permanent learning branch φ p The output data is And the output data of the temporary branch Integrate to obtain the aggregated intermediate prediction results

[0101] From the permanent learning branch φ p The output data [B,2,H,W] of each temporary branch is used to extract the segmentation result [B,1,H,W] for the pure background. i +2,H,W]} i＝1:t Extract the segmentation results for each category {[B,C i ,H,W]} i＝1:t , after integration, we get the aggregated intermediate prediction results The shape is [B,C 1:t +1,H,W].

[0102] The intermediate prediction results include predictions for the background and all categories, with a shape of [B,C 1:t +1,H,W], where the background prediction comes from the permanent learning branch φ p , the category prediction comes from all incremental learning heads

[0103] Step S8: Guided Optimization: Using the image-level posterior guidance branch to generate image-level category information Optimize the intermediate prediction results of the above aggregation Get the final incremental semantic segmentation result of the image,

[0104] The image-level posterior guidance branch provides image-level category information Compared with the above intermediate prediction results Element-wise multiplication is performed along the category dimension to obtain the final result of incremental semantic segmentation, which includes the segmentation results of the background area and each category area in the image.

[0105] Image-level category information provided by the image-level posterior guidance branch The shape is [B,C1:t ], after adding the background dimension, it becomes [B,C 1:t +1], and further expand the dimension to [B,C 1:t +1,1,1], and then with the shape [B,C 1:t +1,H,W] intermediate prediction result Element-wise multiplication is performed to obtain the final result of incremental semantic segmentation, which includes the segmentation results of the background area and each category area in the image.

[0106] Step S9: Next iteration: When the system model is trained, iterations will be performed continuously until all incremental learning stages are completed. This process does not occur during testing.

[0107] In summary, the method of the embodiment of the present invention uses image category information to guide the incremental semantic segmentation task, cleverly applies image-level knowledge to pixel-level tasks, and achieves the current optimal effect.

[0108] Compared with the existing methods that only partially solve the problem of semantic shift, the method of the present invention comprehensively analyzes the causes of semantic shift and clarifies the impact of noise semantics under semantic shift.

[0109] The method of the present invention decouples complex image category knowledge into permanent static concepts and temporary dynamic concepts, divides and conquers them, and finally integrates the output results, effectively improving the model's ability to resist semantic drift.

[0110] Those skilled in the art can understand that the accompanying drawings are only schematic diagrams of an embodiment, and the modules or processes in the accompanying drawings are not necessarily required to implement the present invention.

[0111] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present invention can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present invention or certain parts of the embodiments.

[0112] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0113] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. An incremental image semantic segmentation method based on image-level posterior guidance and knowledge decoupling, characterized in that: include: Performing data enhancement processing on the original image data, obtaining a feature map from the image data after the data enhancement processing using a feature extractor, and inputting the feature map into the image-level posterior guidance branch, the permanent learning branch, and the temporary learning branch in parallel; The image-level posterior guidance branch processes the feature map using dynamic average pooling and a multi-layer perceptron to obtain image-level category information; The permanent learning branch processes the feature map using a convolution layer and an upsampling layer to obtain a segmentation result of the permanent learning branch; The temporary learning branch uses the incremental learning head to learn the category knowledge existing in the feature map in the current incremental stage to obtain the segmentation result of the temporary learning branch; Integrate the segmentation result of the permanent learning branch and the segmentation result of the temporary learning branch to obtain an aggregated intermediate prediction result; The aggregated intermediate prediction result is optimized using the image-level category information to obtain an incremental semantic segmentation result of the original image data.

2. The method according to claim 1, characterized in that: The data enhancement processing is performed on the original image data, a feature map is obtained from the image data after the data enhancement processing by using a feature extractor, and the feature map is input into the image-level posterior guidance branch, the permanent learning branch and the temporary learning branch in parallel, including: In each stage of incremental learning, the image data from the corresponding stage and the image data from the memory buffer are received, the input original image data is flipped, randomly cropped and data enhanced, and the image data after data enhancement is input into the system model. The system model uses the feature extractor to obtain the feature map from the image data. The feature map contains various abstract information of the image. The feature maps are input in parallel into the following image-level posterior guidance branch, permanent learning branch and temporary learning branch.

3. The method according to claim 2, characterized in that The image-level posterior guidance branch processes the feature map using dynamic average pooling and a multi-layer perceptron to obtain image-level category information, including: After the feature map is input into the image-level posterior guidance branch, it is flattened into a one-dimensional vector through dynamic average pooling and then input into the multi-layer perceptron to be processed into a shape of [B,C 1:t ]’s category prediction results Where B represents the batch size, C 1:t Represents all categories that the model has touched up to the current incremental stage; Use the category labels when training the image-level posterior guidance branch model calculate Constrain and calculate the binary cross entropy loss function; Among them, BCE represents the binary cross entropy loss function, φ c represents the classification head of the branch, Represents the pseudo category information obtained by the image after passing through the model in the previous stage.

4. The method according to claim 2, characterized in that: The permanent learning branch processes the feature map using a convolution layer and an upsampling layer to obtain a segmentation result of the permanent learning branch, including: Reassign the image labels of the original image data as: in, represents the complement operator, φ t―1 (x i ) represents the prediction result of the previous model, S(·) is the saliency detector, is assigned to the permanent learning branch φ p Labels; Input the feature map into the permanent learning branch φ p After that, the feature map passes through the permanent learning branch φ p After three convolutional layers and one upsampling layer, the pure background c′ is generated b and fixed foreground c f The segmentation result of the permanent learning branch φ p The output data is The shape is [B, 2, H, W], where B, H, W represent the batch size, the height and width of the input image, respectively, and 2 represents the dimensions of pure background and fixed foreground; In the permanent learning branch φ p During the training process, pixel-level labels are used Calculate using the following formula (3) Make constraints and calculate the binary cross entropy loss function; 5. The method according to claim 2, characterized in that: The temporary learning branch uses the incremental learning head to learn the category knowledge existing in the feature map in the current incremental stage to obtain the segmentation result of the temporary learning branch, including: The image labels of the original image data are additionally reassigned as: in, represents the complement operator and S(·) is the saliency detector used in the previous work SSUL. is the incremental learning head corresponding to the current task t Labels; The incremental learning head As a temporary learning branch, the temporary learning branch is used to learn the category knowledge C existing in the current incremental stage. t After the feature map passes through the three convolutional layers and one upsampling layer of the current incremental learning head, it generates the current stage category C t , the rest of the foreground area (c f ―C t ), pure background area c′ b The segmentation result of the temporary learning branch is The shape is [B,C t +2,H,W], where B,H,W represent the batch size, the height and width of the input image respectively; during the training process of the temporary learning branch, pixel-level labels are used Calculate using the following formula (5) Make constraints and calculate the binary cross entropy loss function; Among them, C inner Indicates the foreground category corresponding to each incremental stage.

6. The method according to claim 2, characterized in that The step of integrating the segmentation result of the permanent learning branch and the segmentation result of the temporary learning branch to obtain an aggregated intermediate prediction result includes: From the permanent learning branch φ p The output data [B,2,H,W] of each temporary branch is used to extract the segmentation result [B,1,H,W] for the pure background. i +2,H,W]} i＝1:t Extract the segmentation results for each category {[B,C i ,H,W]} i＝1:t , after integration, we get the aggregated intermediate prediction results The shape is [B,C 1:t +1,H,W].

7. The method according to claim 6, characterized in that The step of optimizing the aggregated intermediate prediction result by using the image-level category information to obtain an incremental semantic segmentation result of the original image data includes: Image-level category information provided by the image-level posterior guidance branch The shape is [B,C 1:t ], after adding the background dimension, it becomes [B,C 1:t +1], and further expand the dimension to [B,C 1:t +1,1,1], change [B,C 1:t +1,1,1] and the shape is [B,C 1:t +1,H,W] intermediate prediction result Performing element-by-element multiplication to obtain an incremental semantic segmentation result of the original image data, the incremental semantic segmentation result including segmentation results of a background area and each category area in the image; The next round of system iteration processing is performed, and the above processing process is repeated until all incremental learning stages are completed to obtain the final incremental semantic segmentation result of the original image data.

Images