Image incremental semantic segmentation method based on image-level posterior guidance and knowledge decoupling

Abstract

The application provides an image incremental semantic segmentation method based on image-level posterior guidance and knowledge decoupling. The method comprises the following steps: an image-level posterior guidance branch processes a feature map of original image data by using a dynamic average pooling and a multilayer perception to obtain image-level class information; a permanent learning branch processes the feature map by using a convolution layer and an up-sampling layer to obtain a segmentation result of the permanent learning branch; a temporary learning branch learns class knowledge existing in the feature map in the current incremental stage by using an incremental learning head to obtain a segmentation result of the temporary learning branch; the segmentation results of the permanent learning branch and the temporary learning branch are integrated to obtain an aggregated intermediate prediction result; and the aggregated intermediate prediction result is optimized by using the image-level class information to obtain an incremental semantic segmentation result. The method decouples complex image class knowledge into permanent static concepts and temporary dynamic concepts, and effectively improves the ability of the model to resist semantic shift.

Description

Technical Field

[0001] This invention relates to the field of incremental learning technology, and in particular to an image incremental semantic segmentation method based on image-level posterior guidance and knowledge decoupling. Background Technology

[0002] Continuous learning (CL), originating from cognitive neuroscience research on the mechanisms of memory and forgetting, is a method focused on sequential acquisition of knowledge. Also known as incremental learning (IL) or lifelong learning, it has seen significant development over the past decade. Traditional machine learning is typically built on fixed, predefined datasets, making it unable to handle continuously emerging new data and tasks. The core challenge of continuous learning is how to balance stability and plasticity with limited computational and storage resources. Stability refers to the model's ability to retain previous knowledge, while plasticity refers to the model's ability to integrate new knowledge.

[0003] Class Incremental Learning (CIL) is a method for acquiring knowledge sequentially by category, aiming to mitigate catastrophic forgetting while continuously learning new categories. Existing research mainly falls into three categories: Replay methods involve storing data or features of old categories or generating data containing old categories to mitigate catastrophic forgetting. This category can be further divided into experience replay, generative replay, and feature replay. Regularization methods focus on designing loss functions that include penalties based on the parameters' contribution to different tasks. These methods also rely on knowledge distillation, typically using the model from the previous stage as a teacher model to constrain the model in the current stage. Dynamic architecture methods dynamically adjust model parameters based on new data, including assigning specific parameters to different data and decomposing model parameters into task-specific or shared components.

[0004] Class Incremental Semantic Segmentation (CISS), similar to class incremental learning, extends the task to pixel-level prediction. CISS faces the challenge of semantic drift, a phenomenon that refers to the gradual change and evolution of the context category's content as new classes are learned. Semantic drift is typically caused by the mixed semantics between the context, old classes, and future classes.

[0005] In research on incremental semantic segmentation, some scholars have first employed distillation strategies to alleviate this problem. Others have used pseudo-labeling techniques for incremental segmentation to address the background transfer issue. Still others have further incorporated saliency information, introducing the concept of "unknown categories" at each learning stage and using a memory pool to store old data to prevent catastrophic forgetting. Some scholars have used proposal generators to simulate unseen categories. Others have emphasized intra- and inter-class differences, designing a contrastive loss to adjust the feature distribution of classes. Still others have stressed pre-learning future knowledge to enhance the model's ability to distinguish between new and old categories.

[0006] Most of the incremental semantic segmentation methods in the above-mentioned technologies adopt a dynamic architecture, which assigns different parameters to different categories at different stages to learn new knowledge. However, there is a lack of unified and global optimization methods in the model output stage. This independent optimization method is prone to strong erroneous responses to old categories, which seriously disrupts the model from making correct predictions.

[0007] The incremental semantic segmentation methods in the above-mentioned existing technologies only alleviate part of the semantic offset problem and mostly focus on the foreground region of the image, but do not solve the impact of noise semantic information caused by unknown categories in the background on the model's prediction performance. Summary of the Invention

[0008] This invention provides an image incremental semantic segmentation method based on image-level posterior guidance and knowledge decoupling, so as to achieve effective incremental semantic segmentation of images.

[0009] To achieve the above objectives, the present invention adopts the following technical solution.

[0010] An incremental semantic segmentation method for images based on image-level posterior guidance and knowledge decoupling includes:

[0011] The original image data is subjected to data augmentation processing. A feature extractor is used to obtain feature maps from the data augmented image data. The feature maps are then input in parallel into the image-level posterior guidance branch, the permanent learning branch, and the temporary learning branch.

[0012] The image-level posterior guidance branch uses dynamic average pooling and a multilayer perceptron to process the feature map to obtain image-level category information;

[0013] The permanent learning branch processes the feature map using convolutional layers and upsampling layers to obtain the segmentation result of the permanent learning branch;

[0014] The temporary learning branch uses the incremental learning head to learn the category knowledge existing in the feature map of the current incremental stage, and obtains the segmentation result of the temporary learning branch;

[0015] The segmentation results of the permanent learning branch and the segmentation results of the temporary learning branch are integrated to obtain the aggregated intermediate prediction result.

[0016] The intermediate prediction results of the aggregation are optimized using the image-level category information to obtain the incremental semantic segmentation results of the original image data.

[0017] Preferably, the step of performing data augmentation on the original image data, obtaining feature maps from the data-augmented image data using a feature extractor, and inputting the feature maps in parallel into the image-level posterior guidance branch, the permanent learning branch, and the temporary learning branch includes:

[0018] At each stage of incremental learning, image data from the corresponding stage and image data from the memory buffer are received. The original input image data is flipped and randomly cropped for data augmentation. The augmented image data is then input into the system model. The system model uses a feature extractor to obtain feature maps from the image data. The feature maps contain various abstract information about the image.

[0019] The feature maps are input in parallel into the following image-level posterior guidance branch, permanent learning branch, and temporary learning branch.

[0020] Preferably, the image-level posterior guidance branch processes the feature map using dynamic average pooling and a multilayer perceptron to obtain image-level category information, including:

[0021] After the feature map is input into the image-level posterior guidance branch, it undergoes dynamic average pooling to flatten it into a one-dimensional vector, which is then input into the multilayer perceptron and processed into a shape [B, C]. 1:t Category prediction results Where B represents the batch size, C 1:t This represents all categories that the model has encountered up to the current incremental phase;

[0022] Class labels are used when training the image-level posterior-guided branch model. calculate Apply constraints and calculate the binary cross-entropy loss function;

[0023]

[0024] Where BCE represents the binary cross-entropy loss function, φ c This indicates the category header of this branch. This represents the pseudo-class information obtained by the model in the previous stage.

[0025] Preferably, the permanent learning branch processes the feature map using convolutional layers and upsampling layers to obtain the segmentation result of the permanent learning branch, including:

[0026] The image labels of the original image data are reassigned as follows:

[0027]

[0028] in, The complement operator, φ t―1 (x i ) represents the model prediction result of the previous stage, and S(·) is the significance detector. It is assigned to the permanent learning branch φ p Tags;

[0029] The feature map is input into the permanent learning branch φ p Afterwards, the feature map undergoes permanent learning branch φ p After three convolutional layers and one upsampling layer, a c′ for pure background is generated. b and fixed foreground c f The segmentation result, permanent learning branch φ p The output data is The shape is [B,2,H,W], where B,H,W represent the batch size, the height and width of the input image, respectively, and 2 represents the two dimensions of pure background and fixed foreground.

[0030] In the permanent learning branch φ p During training, pixel-level labels are used. Calculate using the following formula (3) Apply constraints and calculate the binary cross-entropy loss function;

[0031]

[0032] Preferably, the temporary learning branch uses the incremental learning head to learn the category knowledge existing in the feature map of the current incremental stage to obtain the segmentation result of the temporary learning branch, including:

[0033] The image labels of the original image data are additionally reassigned as follows:

[0034]

[0035] in, The complement operator is represented by S(·), which is the saliency detector used in the previous work SSUL. It is the incremental learning head corresponding to the current task t. Tags;

[0036] Incremental learning head As a temporary learning branch, the temporary learning branch is used to learn the category knowledge C that exists in the current incremental phase. t After the feature map passes through three convolutional layers and one upsampling layer of the current incremental learning head, a feature map is generated for the current stage category C. t The remaining foreground areas (c f —C t ), pure background area c′ b The segmentation result, the output data of the temporary learning branch is The shape is [B,C] t +2,H,W], where B, H, and W represent the batch size, the height of the input image, and the width of the input image, respectively; during the training of the temporary learning branch, pixel-level labels are used. Calculate using the following formula (5) Apply constraints and calculate the binary cross-entropy loss function;

[0037]

[0038] Among them, C inner This indicates the foreground category corresponding to each incremental stage.

[0039] Preferably, the step of integrating the segmentation results of the permanent learning branch and the segmentation results of the temporary learning branch to obtain the aggregated intermediate prediction result includes:

[0040] From the permanent learning branch φ p Extract the segmentation result [B,1,H,W] for pure background from the output data [B,2,H,W], and extract the segmentation result [B,1,H,W] from the output data {[B,C] of each temporary branch. i +2,H,W]} i＝1:t Extract the segmentation results for each category {[B,C i ,H,W]} i＝1:t After integration, the aggregated intermediate prediction results are obtained. The shape is [B,C] 1:t +1,H,W].

[0041] Preferably, the step of optimizing the aggregated intermediate prediction results using the image-level category information to obtain the incremental semantic segmentation result of the original image data includes:

[0042] Image-level category information provided by the image-level posterior guidance branch The shape is [B,C] 1:t After adding the background dimension, it becomes [B,C]. 1:t +1], further expanding the dimensions to [B,C] 1:t +1,1,1], [B,C 1:t+1,1,1] and shape [B,C] 1:t Intermediate prediction results of +1,H,W] Element-wise multiplication is performed to obtain the incremental semantic segmentation result of the original image data, which includes the segmentation results of the background region and each category region in the image.

[0043] The next round of system iteration is performed, and the above processing procedure is repeated until all incremental learning stages are completed, and the final incremental semantic segmentation result of the original image data is obtained.

[0044] As can be seen from the technical solutions provided by the embodiments of the present invention described above, the present invention uses image category information to guide incremental semantic segmentation tasks, applying image-level knowledge to pixel-level tasks, thus solving the semantic shift problem. The method of the present invention decouples complex image category knowledge into permanent static concepts and temporary dynamic concepts, dividing and conquering them, and finally integrating the output results, effectively improving the model's ability to resist semantic shift.

[0045] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and will become apparent from the description or may be learned by practice of the invention. Attached Figure Description

[0046] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 A schematic diagram illustrating the implementation principle of an image incremental semantic segmentation method based on image-level posterior guidance and knowledge decoupling, provided for an embodiment of the present invention;

[0048] Figure 2 The flowchart illustrates an image incremental semantic segmentation method based on image-level posterior guidance and knowledge decoupling, as provided in this embodiment of the invention. Detailed Implementation

[0049] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0050] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this specification means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or couplings. The term “and / or” as used herein includes any and all combinations of one or more of the associated listed items.

[0051] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless defined as herein.

[0052] To facilitate understanding of the embodiments of the present invention, the following will provide further explanation and description with reference to the accompanying drawings and several specific embodiments. These embodiments do not constitute a limitation on the embodiments of the present invention.

[0053] To address the problem of strong erroneous responses caused by independent optimization, this invention designs an image-level posterior guidance strategy. This strategy fully utilizes image-level posterior category information as global guidance, strengthens the correlation between different tasks, and constrains the model's predictions for all categories to the same scale.

[0054] To address the impact of noisy semantics, this invention proposes a Permanent-Tmporary Knowledge Decoupling strategy. This strategy decouples noisy semantics into simple, static, and permanent semantics and complex, dynamic, and temporary semantic information. Different branches are used for learning, employing a divide-and-conquer approach to effectively alleviate the semantic shift problem and improve model performance.

[0055] This invention employs two strategies to address the resulting semantic offset problem: image-level posterior guidance and decoupling of permanent and temporary knowledge. The implementation principle diagram of an image incremental semantic segmentation method based on image-level posterior guidance and knowledge decoupling provided by an embodiment of this invention is shown below. Figure 1As shown, the image-level posterior guidance branch utilizes a small number of old task training samples stored in the memory mechanism, but rich in categories, to mine the potential guidance capabilities implied by the global category information of these training samples. An additional class increment classifier is used to compensate for the limitations of independent optimization. Simultaneously, the knowledge decoupling branch decouples complex semantic knowledge into permanent and temporary knowledge, mitigating the impact of semantic shifts.

[0056] Image-level posterior guidance: Independent optimization can lead to inconsistent model output scales, and a direct solution is to scale the image using global statistics. Therefore, this invention proposes using image-level posterior probabilities as global guidance information, based on the following facts:

[0057] For any image, if its image-level category domain is C I The pixel-level category domain is C P Then category domain C I and C P They are the same, namely C. I =C P .

[0058] Therefore, this embodiment of the invention designs a branch to predict the image category, while simultaneously training it using incremental learning. Let the current task and the total task be denoted as t and T, respectively. This branch is implemented using a simple multilayer perceptron (MLP), with an input dimension of 4096 and an output dimension of |C|. 1:t During testing, it will predict and output the entire category set C. 1:T Image-level posterior probability.

[0059] In task t (t>1), the model accesses data from the memory buffer M and the current task training data D. t The data. Images from the memory buffer are denoted as... It comes with corresponding image-level real category labels. Its category label depends on the specific task, for example: if So The current task image is denoted as It has class domain C from the current task. t Image-level category tags

[0060] During the incremental training of task t, data samples x from these two sources... k By network backbone h θ Processed into image features h θ (x kThis feature is further processed using an adaptive average pooling function and a multilayer perceptron (MLP) to generate the final prediction. Parameter optimization is achieved using the following formula:

[0061]

[0062] Where BCE represents the binary cross-entropy loss function, φ c This indicates the category header of this branch. This represents the pseudo-class information obtained by the model in the previous stage, compared to directly using partial class labels y. k Pseudo-tags are preferred here. Because it can provide richer monitoring information.

[0063] During testing, the image posterior branch predicts all classes C. 1:t―1 The posterior probability serves as a guide. During testing, for the test image x... i The final predicted segmentation result is determined by the posterior probability of the image. and from the split head The intermediate results are obtained by performing element-wise multiplication between them:

[0064]

[0065] Where σ is the Sigmoid function, α BR It is a hyperparameter used to compensate for the missing background category in the posterior probability. In this invention, α is set... BR =0.9. Result p i It is a tensor of shape [C, HW], p i,j Let represent the C-dimensional prediction result of the j-th pixel. The prediction for the j-th pixel is:

[0066]

[0067] Decoupling permanent and temporary knowledge: To further address the issue of disordered background categories c b Unknown categories that are difficult to determine u And the current category C, which is correct but incomplete. t To address the semantic drift problem caused by the coupled learning of these three elements, this invention proposes a decoupling method based on the separation of different categories of knowledge through semantic content.

[0068] Compared to previous works that classified foreground categories, this invention focuses on dividing all categories in an image into two main groups: static category groups and dynamic category groups. This is based on the following considerations: During different incremental stages, the foreground categories in an image continuously and dynamically change. For example, the foreground category set in the current task might be "cat, horse" (corresponding to the category concepts to be learned in this stage), while in the next task it might become "cow, sofa" (corresponding to the category concepts to be learned in the next stage). Simultaneously, static category groups also exist in the image that can span different incremental stages, such as background and certain fixed foreground concepts.

[0069] Therefore, all categories can be decoupled into two groups: c′ b ∪c f and c b ∪C t , where c′ b and c f These represent the concepts of pure background and fixed foreground, respectively; they represent permanent concepts that can span the entire incremental phase. And c... b and C t This includes the categories and corresponding backgrounds that need to be learned at the current stage, representing temporary concepts that will change as the stage progresses.

[0070] To better learn these two concepts, this invention utilizes two independent branches for training: a permanent learning branch is introduced on top of the existing incremental architecture. p To learn permanent concepts and integrate existing incremental learning heads As a temporary learning branch, it is only used to learn the category knowledge C that exists in the current incremental stage. t Permanent learning branch φ p The lifecycle spans the entire incremental phase, from task 1 to task T, while the temporary learning branch... It will only update during task t.

[0071] Based on the above decoupling strategy, the image label x can be... i Reassigned as:

[0072]

[0073] in, The complement operator is represented by S(·), which is the saliency detector used in the previous work SSUL. It is assigned to the permanent learning branch φ p The tag, and It is the incremental learning head corresponding to the current task t. The tag.

[0074] The optimization objectives for these two branches are as follows:

[0075]

[0076]

[0077] Among them, C inner This indicates the foreground category corresponding to each incremental stage.

[0078] The overall optimization objective can be expressed as:

[0079]

[0080] Where λ1 and λ2 are hyperparameters for balancing different training functions.

[0081] In addition to the unique design described above, this invention also improves upon the crucial memory buffer M. The memory buffer stores previously learned samples or categories in a fixed small capacity, which is essential for improving incremental learning. This invention improves the utilization efficiency of the memory buffer through a class-balanced sampling strategy and storage cost optimization.

[0082] Class-balanced design: To ensure that the image-level posterior branch has sufficient access to samples from all classes and is not biased towards any particular class during training, a class-balanced strategy is employed to select representative samples: given the memory size |M| and the number of known classes |C|. 1:t |, ensure that samples of each category appear at least |M| / |C 1:t In addition, the number of foreground object categories and the number of pixels are also taken into account in the selection criteria to further ensure balance.

[0083] Storage cost optimization: The incremental semantic segmentation method designed in this invention requires only image-level labels and saliency masks for samples in the memory buffer. Image-level labels serve as constraint information in the image-level posterior branch, while saliency masks are used in permanent and temporary learning branches. The image is simply divided into background and foreground objects, stored using 0 and 1 respectively. Compared to previous methods that stored pixel-level annotations for all categories, this simplified storage mechanism effectively reduces storage costs.

[0084] The processing flowchart of an image incremental semantic segmentation method based on image-level posterior guidance and knowledge decoupling provided in this embodiment of the invention is as follows: Figure 2 As shown, the processing steps include the following:

[0085] Step S1: Data Input: At each stage of incremental learning, the model receives image data from the corresponding stage and image data from the memory buffer.

[0086] Step S:2: Data processing: The input image is scaled to 513×513 after data enhancement processing such as flipping and random cropping.

[0087] Step S3: Feature Extraction: The data-augmented image data is input into the system model. The system model first uses a feature extractor to obtain feature maps from the image data. These feature maps contain various abstract information about the image and are crucial for subsequent model predictions. The aforementioned feature maps are then input in parallel into the image-level posterior guidance branch, the permanent learning branch, and the temporary learning branch.

[0088] Step S4: Image-level posterior guidance branch: After the feature map enters this branch, it will undergo network structures such as Adaptive Average Pooling and Multilayer Perceptron (MLP) to obtain image-level category information, which is then determined by the category label. calculate Apply constraints.

[0089] After entering this branch, the feature map first undergoes Adaptive Average Pooling, then is flattened into a one-dimensional vector, and then input into an MLP (Multilayer Perceptron), where it is finally processed into a shape of [B, C]. 1:t Category prediction results Where B represents the batch size, C 1:t This represents all categories that the model has encountered up to the current incremental phase.

[0090] Class labels are used when training the image-level posterior-guided branch model. calculate Apply constraints and calculate the binary cross-entropy loss function;

[0091]

[0092] Where BCE represents the binary cross-entropy loss function, φ c This indicates the category header of this branch. This represents the pseudo-class information obtained by the model in the previous stage, compared to directly using partial class labels y. k Pseudo-tags are preferred here. Because it can provide richer monitoring information.

[0093] Step S5: Permanent Learning Branch: Introduce a permanent learning branch φ based on the existing architecture. p It learns from pixel-level labels, targeting background and certain fixed foreground category concepts that can span different incremental stages. calculate Apply constraints.

[0094] Feature maps are learned through a permanent learning branch φ p After three convolutional layers and one upsampling layer, a c′ for pure background is generated. b and fixed foreground c f Segmentation results Permanent learning branch φ p The output data shape is [B,2,H,W], where B,H,W represent the batch size, the height and width of the input image, respectively, and 2 represents the two dimensions of pure background and fixed foreground.

[0095] During training, the model uses pixel-level labels. Calculate using the above formula (7) Constraints are applied, and the binary cross-entropy (BCE) loss function is calculated. During inference, only the segmentation results of the pure background are incorporated into the intermediate prediction results.

[0096] Step S6: Temporary learning branch: Transfer the existing incremental learning head As a temporary learning branch, it is used to learn the category knowledge C that exists in the current incremental stage. t , composed of pixel-level tags calculate Apply constraints.

[0097] After the feature map is processed through three convolutional layers and one upsampling layer of the current incremental learning head, a feature map is generated for the current stage category C. t The remaining foreground areas (c f —C t ), pure background area c′ b The segmentation result, the output data of the temporary learning branch is [B, C] t +2,H,W], where B, H, and W represent the batch size, the height of the input image, and the width of the input image, respectively.

[0098] After the feature map is processed through three convolutional layers and one upsampling layer of the current incremental learning head, a feature map is generated for the current stage category C. t The remaining foreground areas (c f —C t ), pure background area c′ b Segmentation results The output data of the temporary learning branch has the shape [B, C] t +2,H,W], where B, H, and W represent the batch size, the height and width of the input image, respectively, and C t +2 indicates the category of the current incremental phase and the additional foreground and background dimensions.

[0099] During training, the model integrates the segmentation results of each stage's incremental learning heads for each stage's categories and uses pixel-level labels. Calculate using the above formula (6) Constraints are applied, and the binary cross-entropy (BCE) loss function is calculated. During inference, only the segmentation results of each stage category are incorporated into the intermediate prediction results.

[0100] Step S7: Result Integration: Integrate the permanent learning branch φ p The output data is and output data of temporary branches The results are integrated to obtain aggregated intermediate predictions.

[0101] From the permanent learning branch φ p Extract the segmentation result [B,1,H,W] for pure background from the output data [B,2,H,W], and extract the segmentation result [B,1,H,W] from the output data {[B,C] of each temporary branch. i +2,H,W]} i＝1:t Extract the segmentation results for each category {[B,C i ,H,W]} i＝1:t After integration, the aggregated intermediate prediction results are obtained. The shape is [B,C] 1:t +1,H,W].

[0102] Intermediate prediction results include predictions for background and all categories, with shapes [B, C]. 1:t +1,H,W], where the background prediction comes from the permanent learning branch φ p Category predictions come from all incremental learning heads

[0103] Step S8: Guided Optimization: Utilize image-level posterior guidance to generate image-level category information for the branch. Optimize the intermediate prediction results of the above aggregation. The final incremental semantic segmentation result of the image is obtained.

[0104] The image-level posterior guidance branch provides image-level category information. Compared with the above intermediate prediction results The incremental semantic segmentation final result is obtained by performing element-wise multiplication along the category dimension, which includes the segmentation results of the background region and each category region in the image.

[0105] Image-level category information provided by the image-level posterior guidance branch The shape is [B,C] 1:t After adding the background dimension, it becomes [B,C]. 1:t +1], further expanding the dimensions to [B,C]1:t +1,1,1], then with shape [B,C] 1:t Intermediate prediction results of +1,H,W] Element-wise multiplication is performed to obtain the final incremental semantic segmentation result, which includes the segmentation results of the background region and each category region in the image.

[0106] Step S9: Next Iteration: The system model will continuously iterate during training until all incremental learning phases are completed. This process does not occur during testing.

[0107] In summary, the method of this invention uses image category information to guide incremental semantic segmentation tasks, cleverly applying image-level knowledge to pixel-level tasks, and achieving the best results to date.

[0108] Compared to existing methods that only partially solve the semantic shift problem, the method of this invention comprehensively analyzes the causes of semantic shift and clarifies the impact of noisy semantics under semantic shift.

[0109] The method of this invention decouples complex image category knowledge into permanent static concepts and temporary dynamic concepts, divides and conquers, and finally integrates the output results, effectively improving the model's ability to resist semantic shift.

[0110] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of one embodiment, and the modules or processes shown in the drawings are not necessarily essential for implementing the present invention.

[0111] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that the present invention can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of the present invention.

[0112] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for apparatus or system embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. The apparatus and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0113] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An image incremental semantic segmentation method based on image-level posterior guidance and knowledge decoupling, characterized in that, include: The original image data is subjected to data augmentation processing. A feature extractor is used to obtain feature maps from the data augmented image data. The feature maps are then input in parallel into the image-level posterior guidance branch, the permanent learning branch, and the temporary learning branch. The image-level posterior guidance branch uses dynamic average pooling and a multilayer perceptron to process the feature map to obtain image-level category information; The permanent learning branch processes the feature map using convolutional layers and upsampling layers to obtain the segmentation result of the permanent learning branch; The temporary learning branch uses the incremental learning head to learn the category knowledge existing in the feature map of the current incremental stage, and obtains the segmentation result of the temporary learning branch; The segmentation results of the permanent learning branch and the segmentation results of the temporary learning branch are integrated to obtain the aggregated intermediate prediction result. The intermediate prediction results of the aggregation are optimized using the image-level category information to obtain the incremental semantic segmentation results of the original image data; The permanent learning branch processes the feature map using convolutional layers and upsampling layers to obtain the segmentation result of the permanent learning branch, including: The image labels of the original image data are reassigned as follows: wherein, represents a complement operator, represents a previous stage model prediction, is a saliency detector, is a label assigned to the permanent learning branch . inputting the feature map into the permanent learning branch After that, the feature map goes through the permanent learning branch After three layers of convolutional layers and one layer of up-sampling layers, the permanent learning branch generates segmentation results for pure background and fixed foreground The output data of the permanent learning branch is [B, 2, H, W], where B, H, and W represent batch size, height, and width of the input image, respectively, and 2 represents two dimensions of pure background and fixed foreground. In the permanent learning branch During training, pixel-level labels are used. Calculate using the following formula (3) Apply constraints and calculate the binary cross-entropy loss function; ； The temporary learning branch uses the incremental learning head to learn the category knowledge existing in the feature map of the current incremental stage, and obtains the segmentation result of the temporary learning branch, including: The image labels of the original image data are additionally reassigned as follows: in, This represents the complement operator. It is the saliency detector used in the previous work SSUL. It corresponds to the current task. Incremental learning head Tags; Incremental learning head As a temporary learning branch, the temporary learning branch is used to learn the category knowledge that exists in the current incremental phase. After the feature map passes through three convolutional layers and one upsampling layer of the current incremental learning head, a feature map is generated for the current stage category. Other foreground areas Pure background area The segmentation result, the output data of the temporary learning branch is The shape is [B, +2, H, W], where B, H, W represent the batch size, the height of the input image, and the width of the input image, respectively; during the training of the temporary learning branch, pixel-level labels are used. Calculate using the following formula (5) Apply constraints and calculate the binary cross-entropy loss function; （5） in, Indicates the foreground category corresponding to each incremental stage; The process of integrating the segmentation results of the permanent learning branch and the segmentation results of the temporary learning branch to obtain the aggregated intermediate prediction result includes: From the permanent learning branch Extract the segmentation result [B, 1, H, W] for pure background from the output data [B, 2, H, W], and extract the segmentation result [B, 1, H, W] from the output data {[B, 2, H, W] of each temporary branch. +2, H, W] Extract the segmentation results for each category {[B, , H, W] After integration, the aggregated intermediate prediction results are obtained. The shape is [B, +1, H, W].

2. The method according to claim 1, characterized in that, The process of performing data augmentation on the original image data, obtaining feature maps from the augmented image data using a feature extractor, and inputting the feature maps in parallel into an image-level posterior guidance branch, a permanent learning branch, and a temporary learning branch includes: At each stage of incremental learning, image data from the corresponding stage and image data from the memory buffer are received. The original input image data is flipped and randomly cropped for data augmentation. The augmented image data is then input into the system model. The system model uses a feature extractor to obtain feature maps from the image data. The feature maps contain various abstract information about the image. The feature maps are input in parallel into the following image-level posterior guidance branch, permanent learning branch, and temporary learning branch.

3. The method according to claim 2, characterized in that, The image-level posterior guidance branch processes the feature map using dynamic average pooling and a multilayer perceptron to obtain image-level category information, including: After the feature map is input into the image-level posterior guidance branch, it undergoes dynamic average pooling to flatten it into a one-dimensional vector, which is then input into the multilayer perceptron and processed into a shape [B, Category prediction results Where B represents the batch size. This represents all categories that the model has encountered up to the current incremental phase; Class labels are used when training the image-level posterior-guided branch model. calculate Apply constraints and calculate the binary cross-entropy loss function; Where BCE represents the binary cross-entropy loss function, This indicates the category header of this branch. This represents the pseudo-class information obtained by the model in the previous stage.

4. The method according to claim 1, characterized in that, The optimization of the aggregated intermediate prediction results using the image-level category information to obtain the incremental semantic segmentation result of the original image data includes: Image-level category information provided by the image-level posterior guidance branch The shape is [B, After adding the background dimension, it becomes [B, +1], further expanding the dimension to [B, +1, 1, 1], will [B, +1, 1, 1] and shape [B, Intermediate prediction results of +1, H, W] Element-wise multiplication is performed to obtain the incremental semantic segmentation result of the original image data, which includes the segmentation results of the background region and each category region in the image. The next round of system iteration is performed, and the above processing procedure is repeated until all incremental learning stages are completed, and the final incremental semantic segmentation result of the original image data is obtained.

Images