An interactive image segmentation method based on incremental learning

Abstract

This invention discloses an interactive image segmentation method based on incremental learning, comprising the following steps: dividing the image segmentation data stream into multiple training datasets according to the task; providing the first click, positive click, and negative click of the target category, and generating a distance map and Gaussian map based on the clicks; in the first interactive image segmentation task, learning the training dataset of the first task and the distance map and Gaussian map based on user clicks, and updating the interactive image segmentation model of the first task with the first click loss and general click loss; treating the interactive image segmentation model learned in the previous task as the old model, and using the old model to initialize the interactive image segmentation model of the current task, learning the training dataset of the current task and the distance map and Gaussian map based on user clicks, and updating the interactive image segmentation model of the current task with the first click loss, general click loss, and knowledge distillation loss.

Description

Technical Field

[0001] This invention relates to the fields of human-computer interaction and computer vision, and more specifically to an interactive image segmentation method based on incremental learning. Background Technology

[0002] Interactive segmentation aims to segment target categories with minimal user input, and it holds broad application prospects in image editing and medical image analysis. In recent years, with the widespread adoption of data-driven deep learning technologies, the demand for pixel-level annotation has increased dramatically in several fields, such as salient object detection, semantic segmentation, instance segmentation, camouflaged object detection, and video processing. Therefore, an increasing number of researchers are dedicated to exploring efficient interactive segmentation techniques to reduce annotation costs. Current interactive segmentation techniques learn all target categories at once during the training phase and do not introduce new target categories during the testing phase. However, in real-world applications, models continuously encounter new target categories, making existing interactive segmentation techniques unsuitable for incremental learning settings. Currently, image interactive segmentation methods based on incremental learning have not yet received sufficient attention and exploration from researchers.

[0003] Interactive image segmentation methods based on incremental learning face two main challenges: low user interaction efficiency and the problem of forgetting old target categories during incremental learning. For interactive segmentation, most current methods indiscriminately use all interaction points to generate segmentation results, but not all interaction points produce the same segmentation effect. The first click often plays a crucial role in interactive segmentation, significantly impacting the performance of the interactive segmentation model. For incremental learning, most current methods suffer from the problem of forgetting old categories, meaning that the model parameters fit new knowledge, leading to the loss of old knowledge.

[0004] Existing interactive segmentation methods for multi-foreground target images (Zhou Jie, Hu Han, Feng Jianjiang, Yu Chuan. Interactive segmentation method for multi-foreground target images [P]) can only interactively segment already learned categories. If a new target category appears, all data needs to be retrained, and it does not have the ability to learn incrementally. Summary of the Invention

[0005] To address the problems existing in current technologies, this invention aims to research and design an incremental learning method for interactive image segmentation. This method constructs an interactive image segmentation strategy based on incremental learning. The proposed first-click loss uses the user's first click as the anchor point for interactive segmentation, which can better guide the location and subject information of the target category. General click losses consider all user clicks to guide the interactive segmentation model to more accurately segment the target category. Furthermore, the knowledge distillation loss proposed in this invention ensures the consistency of category semantic relationships across different incremental learning tasks. This invention can effectively improve the user interaction efficiency in interactive image segmentation and reduce the forgetting of old categories during incremental learning.

[0006] The present invention is achieved by at least one of the following technical solutions.

[0007] An interactive image segmentation method based on incremental learning includes the following steps:

[0008] S1. Obtain the image segmentation data stream and divide the image segmentation data stream into multiple training datasets containing non-overlapping target categories according to the task. Each task contains only one training dataset, and each training dataset contains one or more target categories.

[0009] S2. Provide users with the first click and positive click within the target category and the negative click outside the target category in the training dataset image, and generate distance maps and Gaussian maps based on the first click, positive click and negative click respectively;

[0010] S3. In the first interactive image segmentation task, learn the training dataset for the first task and the distance map and Gaussian map based on user clicks, and update the interactive image segmentation model for the first task with the first click loss and the general click loss.

[0011] S4. In the next interactive image segmentation task, the interactive image segmentation model learned in the previous task is regarded as the old model, and the interactive image segmentation model of the current task is initialized using the old model. The training dataset of the current task and the distance map and Gaussian map based on user clicks are learned. The interactive image segmentation model of the current task is updated with the first click loss, general click loss and knowledge distillation loss.

[0012] S5. When there is an unlearned interactive image segmentation task, repeat step S3 continuously to learn a training dataset containing new non-overlapping target categories to update the interactive image segmentation model until all training datasets for interactive image segmentation tasks have been learned. Then, use the final interactive image segmentation model to perform interactive segmentation on the image to obtain the interactive segmentation result.

[0013] Further, in step S1, the image segmentation data stream D containing non-overlapping target categories includes training datasets D = {D} for M interactive image segmentation tasks. 1 D 2 ,…,D M Each interactive image segmentation task contains a training dataset, and the training dataset corresponding to the t-th interactive image segmentation task is... in This represents the i-th image in the t-th training dataset. Represents the corresponding true segmentation label, |D t | Represents the training dataset D t The number of samples in the middle; the goal of interactive image segmentation based on incremental learning is to accurately segment all previously learned target categories through user interaction.

[0014] Furthermore, in step S2, the first click within the target category in the training dataset is 'a'. f The set of positive clicks within the target category is A. p The set of negative clicks outside the target category is A. n Let A be the set of all clicks, and G be the set of all pixels within the target category. p The set of all pixels outside the target category is G. n Let G be the set of all pixels; calculate the distance from pixel p to pixel a in the image. f The distance to the first click is obtained from the map, and the distance from pixel p in the image to set A is calculated. p The minimum distance from the midpoint is used to obtain the distance map of the frontal click, and the distance from pixel p in the image to set A is calculated. n The minimum distance from the midpoint is used to obtain the distance map for negative clicks.

[0015] Furthermore, in step S2, a Gaussian function is applied to the distance map to soften the distance information, resulting in a Gaussian map.

[0016] Furthermore, in step S3, in the first interactive image segmentation task, the first click loss L f The calculation formula is:

[0017]

[0018] Where: N represents the total number of pixels; G represents the set of all pixels;

[0019] Binary cross-entropy loss L p The calculation formula is:

[0020]

[0021] in: This represents the prediction value of the current interactive segmentation model for the p-th pixel of the i-th image in the first training dataset. This represents the true label of the p-th pixel in the i-th image of the first training dataset;

[0022] Weighted weight of the first click The calculation formula is:

[0023]

[0024] Where α and β are constants;

[0025] Pixel p is far from the first click a f The distance ψ(p,{a f The formula for calculating}) is:

[0026]

[0027] Where τ is a constant;

[0028] Pixel p is far from the first click a f The minimum distance φ(p,a) between the midpoints f The formula for calculating ) is:

[0029]

[0030] Where: d(p,p) s ) represents the distance from point p to point p. s Euclidean distance.

[0031] Furthermore, in step S3, in the first interactive image segmentation task, the typical click loss L... g The calculation formula is:

[0032]

[0033] Where N represents the total number of pixels, and G represents the set of all pixels; L p This represents the binary cross-entropy loss;

[0034] Weighted weight of general clicks The calculation formula is:

[0035]

[0036] Where α and β are constants; the distance set A of pixel point p p and A n The distance ψ(p,A) p ) and ψ(p,A n The calculation formulas for ) are as follows:

[0037]

[0038]

[0039] Where τ is a constant; the distance of pixel p from set A p and A n The minimum distance φ(p,A) between the midpoints p ) and φ(p,A n The calculation formulas for ) are as follows:

[0040]

[0041]

[0042] p s Represents the set of positive clicks, A. p Or negative click set A n Any point in the array.

[0043] Furthermore, in step S4, in the next interactive image segmentation task, the first click loss L f The calculation formula is:

[0044]

[0045] Where N represents the total number of pixels, and G represents the set of all pixels; L p This represents the binary cross-entropy loss;

[0046] Binary cross-entropy loss L p The calculation formula is:

[0047]

[0048] in: This represents the prediction value of the current interactive segmentation model for the p-th pixel of the i-th image in the t-th training dataset. This represents the true label of the p-th pixel in the i-th image of the t-th training dataset;

[0049] Weighted weight of the first click The calculation formula is:

[0050]

[0051] Where α and β are constants, and the distance from pixel p to the first click a is... f The distance ψ(p,{a f The formula for calculating}) is:

[0052]

[0053] Where τ is a constant.

[0054] Furthermore, in step S4, in the next interactive image segmentation task, the typical click loss L... g The calculation formula is:

[0055]

[0056] Weighted weight of general clicks The calculation formula is:

[0057]

[0058] Pixel p distance set A p and A n The distance ψ(p,A) p ) and ψ(p,A n The calculation formulas for ) are as follows:

[0059]

[0060]

[0061] Pixel p distance set A p and A n The minimum distance φ(p,A) between the midpoints p ) and φ(p,A n The calculation formulas for ) are as follows:

[0062]

[0063]

[0064] Where d(p,p) s ) represents the distance from point p to point p. s Euclidean distance.

[0065] Furthermore, in step S4, in the next interactive image segmentation task, the knowledge distillation loss L... d The calculation formula is:

[0066]

[0067] in: Let KL(·||·) represent the prediction value of the old model for the p-th pixel of the i-th image in the t-th training dataset, and let KL(·||·) represent the Kullback-Leibler divergence.

[0068] Furthermore, when there are multiple training datasets for interactive image segmentation that have not been learned in step S1, the training dataset containing new non-overlapping target categories is learned in step S4 to update the interactive image segmentation model until all training datasets for interactive image segmentation are learned. The final interactive image segmentation model is then used to perform interactive segmentation on all previously learned target categories to obtain the interactive segmentation result.

[0069] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0070] Compared with traditional interactive semantic segmentation methods, the incremental learning method for interactive image segmentation provided by this invention can improve the user interaction efficiency in interactive image segmentation and reduce the forgetting of old categories during the incremental learning process, and is simple to implement. Attached Figure Description

[0071] Figure 1 This is a schematic diagram illustrating the principle of an interactive image segmentation method based on incremental learning, as shown in the example.

[0072] Figure 2 Here is a flowchart of an interactive image segmentation method based on incremental learning, as shown in the example.

[0073] Figure 3 This is a schematic diagram of the first click in an example. Detailed Implementation

[0074] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0075] The interactive image segmentation method based on incremental learning provided by this invention, such as Figure 1 As shown, the method includes the following steps: First, provide the user's first click, positive click, and negative click for the target category, and generate a distance map and a Gaussian map based on the clicks; In the first interactive image segmentation task, learn the training dataset of the first task and the distance map and Gaussian map based on the user clicks, and update the interactive image segmentation model of the first task with the first click loss and general click loss; In the next interactive image segmentation task, treat the interactive image segmentation model learned in the previous task as the old model, and use the old model to initialize the interactive image segmentation model of the current task, learn the training dataset of the current task and the distance map and Gaussian map based on the user clicks, and update the interactive image segmentation model of the current task with the first click loss, general click loss, and knowledge distillation loss. This invention can effectively improve the user's interaction efficiency in interactive image segmentation and reduce the forgetting of old categories during incremental learning.

[0076] like Figure 1 and Figure 2 As shown in the embodiment, an interactive image segmentation method based on incremental learning includes the following steps:

[0077] S1. Download the semantic segmentation data stream containing multiple target categories (such as PASCAL VOC 2012). Divide the image segmentation data stream into multiple training datasets containing non-overlapping target categories according to the task. Each task contains only one training dataset, and each training dataset contains one or more target categories.

[0078] In this embodiment, an interactive image segmentation task is defined as follows: An interactive image segmentation task refers to the data from the arrival of the current target category in the segmentation data stream to the arrival of the new target semantic category. The training dataset for each interactive image segmentation task is divided according to the total number of interactive image segmentation tasks, and each interactive image segmentation task contains one training dataset.

[0079] Specifically, the segmented data stream is defined as follows:

[0080] In this step, the image segmentation data stream D containing non-overlapping target categories is derived from the training dataset D = {D} of M interactive image segmentation tasks. 1 D 2 ,…,D M The dataset is composed of}, where each interactive image segmentation task contains a training dataset, and the training dataset corresponding to the t-th interactive image segmentation task is}. in This represents the i-th image in the t-th training dataset. Represents the corresponding true segmentation label, |D t | Represents the training dataset D t The number of samples in the middle; the goal of interactive image segmentation based on incremental learning is to segment all previously learned target categories through user interaction;

[0081] S2. Provide users with the first click and positive click within the target category and the negative click outside the target category in the training dataset image, and generate distance maps and Gaussian maps based on the first click, positive click and negative click respectively;

[0082] In this step, the first click within the target category in the training dataset is 'a'. f The set of positive clicks within the target category is A. p The set of negative clicks outside the target category is A. n Let A be the set of all clicks, and G be the set of all pixels within the target category. p The set of all pixels outside the target category is G. n Let G be the set of all pixels; calculate the distance from pixel p to pixel a in the image.f The distance is obtained from the first click on the map, and the distance from pixel p in the image to A is calculated. p The minimum distance from the midpoint is used to obtain the distance map of the frontal click, and the distance from pixel p in the image to A is calculated. n The minimum distance from the midpoint is used to obtain a distance map of negative clicks. A Gaussian function is then applied to this distance map to soften the distance information, resulting in a Gaussian map.

[0083] S3. In the first interactive image segmentation task, learn the training dataset for the first task and the distance map and Gaussian map based on user clicks, and update the interactive image segmentation model for the first task with the first click loss and the general click loss.

[0084] As one embodiment, the interactive segmentation model is a combination of the existing ResNet model, ASPP module, and decoder;

[0085] In this step, in the first interactive image segmentation task, the first click loss L f The calculation formula is:

[0086]

[0087] Where: N represents the total number of pixels;

[0088] Binary cross-entropy loss L p The calculation formula is:

[0089]

[0090] in: This represents the prediction value of the current interactive segmentation model for the p-th pixel of the i-th image in the first training dataset. This represents the true label (0 or 1) of the p-th pixel in the i-th image of the first training dataset;

[0091] Weighted weight of the first click The calculation formula is:

[0092]

[0093] Among them: α=0.8, β=20;

[0094] Pixel p is far from the first click a f The distance ψ(p,{a f The formula for calculating}) is:

[0095]

[0096] As one example, τ is 100.

[0097] Pixel p is far from the first click a f The minimum distance φ(p,a) between the midpoints f The formula for calculating ) is:

[0098]

[0099] Where: d(p,p) s ) represents the distance from point p to point p. s Euclidean distance;

[0100] Typical click loss L g The calculation formula is:

[0101]

[0102] Weighted weight of general clicks The calculation formula is:

[0103]

[0104] Pixel p distance set A p and A n The distance ψ(p,A) p ) and ψ(p,A n The calculation formulas for ) are as follows:

[0105]

[0106]

[0107] Pixel p distance set A p and A n The minimum distance φ(p,A) between the midpoints p ) and φ(p,A n The calculation formulas for ) are as follows:

[0108]

[0109]

[0110] p s Represents the set of positive clicks, A. p Or negative click set A n Any point in the array.

[0111] S4. In the next interactive image segmentation task, the interactive image segmentation model learned in the previous task is regarded as the old model, and the interactive image segmentation model of the current task is initialized using the old model. The training dataset of the current task and the distance map and Gaussian map based on user clicks are learned. The interactive image segmentation model of the current task is updated with the first click loss, general click loss and knowledge distillation loss.

[0112] In this step, in the next interactive image segmentation task, the first click loss L f The calculation formula is:

[0113]

[0114]

[0115] Binary cross-entropy loss L p The calculation formula is:

[0116]

[0117] in: This represents the prediction value of the current interactive segmentation model for the p-th pixel of the i-th image in the t-th training dataset. This represents the true label (0 or 1) of the p-th pixel in the i-th image of the t-th training dataset;

[0118] Weighted weight of the first click The calculation formula is:

[0119]

[0120] Typical click loss L g The calculation formula is:

[0121]

[0122] Weighted weight of general clicks The calculation formula is:

[0123]

[0124] Knowledge distillation loss L d The calculation formula is:

[0125]

[0126] in: Let KL(·||·) represent the prediction value of the old model for the p-th pixel of the i-th image in the t-th training dataset, and let KL(·||·) represent the Kullback-Leibler divergence.

[0127] S5. When there is an unlearned interactive image segmentation task, repeat step S3 continuously to learn a training dataset containing new non-overlapping target categories to update the interactive image segmentation model until all training datasets for interactive image segmentation tasks have been learned. Then, use the final interactive image segmentation model to perform interactive image segmentation and obtain the interactive segmentation result.

[0128] In real-world scenarios, when there are multiple training datasets for interactive image segmentation that have not been learned in step S1, the interactive image segmentation model continues to learn training datasets containing new, non-overlapping target categories according to step S4 until all training datasets for interactive image segmentation have been learned. Finally, the interactive image segmentation model is used to perform interactive segmentation on all previously learned target categories to obtain interactive segmentation results.

[0129] It should be understood that this embodiment is for illustrative purposes only and is not intended to limit the scope of the invention. Furthermore, it should be understood that after reading the teachings of this invention, those skilled in the art can make various alterations or modifications to the invention, and these equivalent forms also fall within the scope defined by the appended claims.

Claims

1. An interactive image segmentation method based on incremental learning, characterized in that, Includes the following steps: S1. Obtain the image segmentation data stream and divide the image segmentation data stream into multiple training datasets containing non-overlapping target categories according to the task. Each task contains only one training dataset, and each training dataset contains one or more target categories. S2. Provide users with the first click and positive click within the target category and the negative click outside the target category in the training dataset image, and generate distance maps and Gaussian maps based on the first click, positive click and negative click respectively; S3. In the first interactive image segmentation task, learn the training dataset for the first task and the distance map and Gaussian map based on user clicks, and update the interactive image segmentation model for the first task with the first click loss and the general click loss. S4. In the next interactive image segmentation task, the interactive image segmentation model learned in the previous task is regarded as the old model, and the interactive image segmentation model of the current task is initialized using the old model. The training dataset of the current task and the distance map and Gaussian map based on user clicks are learned. The interactive image segmentation model of the current task is updated with the first click loss, general click loss and knowledge distillation loss. S5. When there is an unlearned interactive image segmentation task, repeat step S3 continuously to learn a training dataset containing new non-overlapping target categories to update the interactive image segmentation model until all training datasets for interactive image segmentation tasks have been learned. Then, use the final interactive image segmentation model to perform interactive segmentation on the image to obtain the interactive segmentation result.

2. The interactive image segmentation method based on incremental learning according to claim 1, characterized in that, In step S1, the image segmentation data stream D containing non-overlapping target categories includes the training datasets D = {D} for M interactive image segmentation tasks. 1 D 2 ,…,D M Each interactive image segmentation task contains a training dataset, and the training dataset corresponding to the t-th interactive image segmentation task is... in This represents the i-th image in the t-th training dataset. Represents the corresponding true segmentation label, |D t | Represents the training dataset D t The number of samples in the middle; the goal of interactive image segmentation based on incremental learning is to accurately segment all previously learned target categories through user interaction.

3. The interactive image segmentation method based on incremental learning according to claim 1, characterized in that, In step S2, the first click within the target category in the training dataset is 'a'. f The set of positive clicks within the target category is A. p The set of negative clicks outside the target category is A. n Let A be the set of all clicks, and G be the set of all pixels within the target category. p The set of all pixels outside the target category is G. n The set of all pixels is G; Calculate the distance from pixel p to pixel a in the image. f The distance to the first click is obtained from the map, and the distance from pixel p in the image to set A is calculated. p The minimum distance from the midpoint is used to obtain the distance map of the frontal click, and the distance from pixel p in the image to set A is calculated. n The minimum distance from the midpoint is used to obtain the distance map for negative clicks.

4. The interactive image segmentation method based on incremental learning according to claim 1, characterized in that, In step S2, a Gaussian function is applied to the distance map to soften the distance information, resulting in a Gaussian map.

5. The image interactive segmentation method based on incremental learning according to claim 1, characterized in that, In step S3, in the first interactive image segmentation task, the first click loss L f The calculation formula is: Where: N represents the total number of pixels; G represents the set of all pixels; Binary cross-entropy loss L p The calculation formula is: in: This represents the prediction value of the current interactive segmentation model for the p-th pixel of the i-th image in the first training dataset. This represents the true label of the p-th pixel in the i-th image of the first training dataset; Weighted weight of the first click The calculation formula is: Where α and β are constants; Pixel p is far from the first click a f The distance ψ(p,{a f The formula for calculating}) is: Where τ is a constant; Pixel p is far from the first click a f The minimum distance φ(p,a) between the midpoints f The formula for calculating ) is: Where: d(p,p) s ) represents the distance from point p to point p. s Euclidean distance.

6. The interactive image segmentation method based on incremental learning according to claim 1, characterized in that, In step S3, in the first interactive image segmentation task, the typical click loss L... g The calculation formula is: Where N represents the total number of pixels, and G represents the set of all pixels; L p This represents the binary cross-entropy loss; Weighted weight of general clicks The calculation formula is: Where α and β are constants; the distance set A of pixel point p p and A n The distance ψ(p,A) p ) and ψ(p,A n The calculation formulas for ) are as follows: Where τ is a constant; the distance of pixel p from set A p and A n The minimum distance φ(p,A) between the midpoints p ) and φ(p,A n The calculation formulas for ) are as follows: p s Represents the set of positive clicks, A. p Or negative click set A n Any point in the array.

7. The interactive image segmentation method based on incremental learning according to claim 1, characterized in that, In step S4, in the next interactive image segmentation task, the first click loss L f The calculation formula is: Where N represents the total number of pixels, and G represents the set of all pixels; L p This represents the binary cross-entropy loss; Binary cross-entropy loss L p The calculation formula is: in: This represents the prediction value of the current interactive segmentation model for the p-th pixel of the i-th image in the t-th training dataset. This represents the true label of the p-th pixel in the i-th image of the t-th training dataset; Weighted weight of the first click The calculation formula is: Where α and β are constants, and the distance from pixel p to the first click a is... f The distance ψ(p,{a f The formula for calculating}) is: Where τ is a constant.

8. The image interactive segmentation method based on incremental learning according to claim 7, characterized in that, In step S4, in the next interactive image segmentation task, the typical click loss L... g The calculation formula is: Weighted weight of general clicks The calculation formula is: Pixel p distance set A p and A n The distance ψ(p,A) p ) and ψ(p,A n The calculation formulas for ) are as follows: Pixel p distance set A p and A n The minimum distance φ(p,A) between the midpoints p ) and φ(p,A n The calculation formulas for ) are as follows: Where d(p,p) s ) represents the distance from point p to point p. s Euclidean distance.

9. The interactive image segmentation method based on incremental learning according to claim 8, characterized in that, In step S4, in the next interactive image segmentation task, the knowledge distillation loss L... d The calculation formula is: in: Let KL(·||·) represent the prediction value of the old model for the p-th pixel of the i-th image in the t-th training dataset, and let KL(·||·) represent the Kullback-Leibler divergence.

10. An interactive image segmentation method based on incremental learning according to any one of claims 1 to 9, characterized in that, When there are multiple training datasets for interactive image segmentation that have not been learned in step S1, continue to learn training datasets containing new non-overlapping target categories according to step S4 to update the interactive image segmentation model, until all training datasets for interactive image segmentation have been learned. Then, use the finally obtained interactive image segmentation model to perform interactive segmentation on all previously learned target categories to obtain interactive segmentation results.

Images