Balanced continuous panoramic segmentation method and system

Through the memory strategy of past class backtracking distillation and category proportionality, the problems of knowledge stability and adaptability balance and class distribution imbalance in continuous panoramic segmentation are solved, and the overall performance of the model and the old class memory ability are improved.

CN119091132AActive Publication Date: 2024-12-06SHANDONG UNIV
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
CN202410944524.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-15
Publication Date
2024-12-06
Estimated Expiration
2044-07-15

AI Technical Summary

Technical Problem

While maintaining old knowledge, the existing continuous panoramic segmentation method limits the learning ability of new knowledge, and the class distribution of the replayed sample set is unbalanced, which cannot effectively reflect the cumulative distribution in historical training data, affecting the memory of old classes.

Method used

Using past class backtracking distillation (PCBD) and class proportional memory strategy (CPM), the training samples and replay samples are processed in parallel, the features are extracted using the Transformer decoder, and the features related to past classes are traced backtracked in the distillation loss. Combining greedy algorithms, a category-based replay sample set is constructed to ensure the consistency of class distribution.

Benefits of technology

While maintaining the stability of old knowledge, it allows the model to flexibly adapt to new knowledge, improves the overall performance of the model in continuous panoramic segmentation tasks, and enhances the memory of old classes and the learning ability of new classes.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119091132A_ABST
    Figure CN119091132A_ABST
Patent Text Reader

Abstract

The invention provides a balanced continuous panorama segmentation method and system. The method comprises the following steps: obtaining a training sample and a playback sample; adopting a first model and a second model to respectively perform parallel operation processing on the acquired training sample and the replay sample; after the labels in the training samples are supplemented, the labels of the output blocks of the training samples are distributed to obtain distributed images, and the distributed images and second features output by a second Transform decoder are subjected to backtracking processing to obtain third features; and obtaining a classification output result of the image.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of continuous panoramic segmentation, and in particular relates to a balanced continuous panoramic segmentation method and system. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] In the field of computer vision, panoptic segmentation is an important basic task. It combines the concepts of semantic segmentation and instance segmentation, aiming to classify each pixel in an image into a specific semantic category and distinguish different instances.

[0004] In Continuous Panoptic Segmentation (CPS), the training process consists of multiple steps, each step focusing on different categories. In each step, only some categories are considered as foreground and annotated with the true label, while foreground categories from past or future steps are marked as background.

[0005] In addition, traditional panoptic segmentation methods are usually trained on static datasets. When the dataset is updated, the entire network needs to be retrained. This method can lead to catastrophic forgetting, that is, the model will forget previously learned information. Therefore, how to make the panoptic segmentation model learn new information without forgetting the acquired knowledge has become an important research topic, called Continual Panoptic Segmentation (CPS).

[0006] Existing continuous panoramic segmentation methods deal with catastrophic forgetting by performing knowledge distillation at the output level. However, this method may limit the learning of new knowledge while retaining old knowledge. In addition, the replay technique in the existing continuous semantic segmentation (CSS) also has the potential to be used in CPS. This type of method saves some samples from past steps and replays them in future steps to preserve memory. However, the preservation of replay samples does not consider the rationality of the category distribution, and often selects the same number of images for each category, ignoring the differences in the frequency of category occurrence in the training data.

[0007] Therefore, in the process of implementing the present invention, the inventors found that the prior art has at least the following shortcomings and deficiencies:

[0008] There is a lack of balance between knowledge stability and adaptability in existing continuous panoramic segmentation: existing methods alleviate the forgetting problem through knowledge distillation, but these methods usually distill all features or outputs, which to some extent limits the ability of image processing models to learn new knowledge. Therefore, it is necessary to consider how to effectively maintain the stability of old knowledge while ensuring the flexible adaptation of new knowledge.

[0009] The class distribution in the replay sample set in the existing continuous panoramic segmentation is unbalanced: the traditional method selects an equal number of images of each class in the replay sample set, ignoring the difference in the frequency of each class in the training data. This neglect causes the class distribution in the replay sample set to fail to reflect the cumulative distribution in the historical training data, thus affecting the memory of the old classes. Therefore, it is necessary to consider how to make the class distribution in the replay sample set better reflect the cumulative distribution in the historical training data to enhance the memory of the old classes. Summary of the invention

[0010] To overcome the above-mentioned deficiencies of the prior art, the present invention provides a balanced continuous panoramic segmentation method, which allows other features to flexibly adapt to new knowledge while maintaining the stability of old knowledge, thereby achieving a balance between stability and adaptability.

[0011] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:

[0012] In a first aspect, a balanced continuous panoptic segmentation method is disclosed, comprising:

[0013] Obtain training samples and replay samples;

[0014] The first model and the second model are used to perform parallel operation processing on the acquired training samples and the replay samples respectively;

[0015] The first model outputs the prediction results to the first pixel decoder and the first Transformer decoder respectively, performs feature extraction on the output of the first Transformer decoder to obtain the first feature, and the first feature is superimposed on the prediction result of the first pixel decoder as a pseudo label of the past category of the training sample to supplement the annotation in the training sample;

[0016] Perform feature extraction again on the first feature to obtain a fourth feature;

[0017] The second model outputs the prediction results to the second pixel decoder and the second Transformer decoder respectively, performs feature extraction on the output of the second Transformer decoder to obtain the second feature, and the second feature and the prediction result of the second pixel decoder are used as pseudo labels of the past categories of the training samples to supplement the annotations in the training samples;

[0018] After supplementing the annotations in the training sample, the labels of the output blocks of the training sample are assigned to obtain the assigned image, and the assigned image is back-processed with the second feature output by the second Transformer decoder to obtain the third feature;

[0019] A classification output result of the image is obtained based on the third feature and the fourth feature.

[0020] As a further technical solution, the training samples include multiple image-label pairs, the label of each image includes multiple true value blocks, and the replay samples are replay samples retained by a category-proportional memory strategy.

[0021] As a further technical solution, the parameters of the first model are frozen during the parallel operation process.

[0022] As a further technical solution, the label assignment of the output blocks of the training samples is implemented by using a binary matching method.

[0023] As a further technical solution, the output of each input image in the training sample includes a set of N q A block, represented by where c i represents classification prediction, m i represents mask prediction, N q is the total number of query vectors and the number of output tiles.

[0024] As a further technical solution, the distillation loss is obtained based on the third and fourth characteristics.

[0025]

[0026] here, represents the indicator function, ε past Refers to the set of output blocks that are assigned past category labels, MSE means mean square error, Represents the previous model M t-1 Transformer features.

[0027] As a further technical solution, the specific process of obtaining replay samples through the category-proportional memory strategy is as follows:

[0028] In the first training step, the number of occurrences of each true cut of the current class in the training samples is counted. These counts enable the calculation of the class distribution, which is used to construct a sample set containing the required number of samples.

[0029] As a further technical solution, a greedy algorithm is used to obtain a usable local solution in constructing a sample set containing the required number of samples; specifically, the greedy algorithm iterates the required number of times on all images in the training sample, and each time selects an image to minimize the difference between the changing category distribution and the calculated category distribution;

[0030] In each subsequent step, the number of occurrences of the current category is calculated and the cumulative category distribution is updated. Using this distribution, the replay sample set is updated.

[0031] As a further technical solution, after updating the replay sample set multiple times, a constraint is introduced to form Time Retention A certain proportion of the image.

[0032] As a further technical solution, in each incremental training step, With D t Combined training.

[0033] In a second aspect, a balanced continuous panoptic segmentation system is disclosed, comprising:

[0034] The sample acquisition module is configured to: acquire training samples and replay samples;

[0035] The parallel operation processing module is configured to: use the first model and the second model to perform parallel operation processing on the acquired training samples and the replay samples respectively;

[0036] The first model outputs the prediction results to the first pixel decoder and the first Transformer decoder respectively, performs feature extraction on the output of the first Transformer decoder to obtain the first feature, and the first feature is superimposed on the prediction result of the first pixel decoder as a pseudo label of the past category of the training sample to supplement the annotation in the training sample;

[0037] Perform feature extraction again on the first feature to obtain a fourth feature;

[0038] The second model outputs the prediction results to the second pixel decoder and the second Transformer decoder respectively, performs feature extraction on the output of the second Transformer decoder to obtain the second feature, and the second feature and the prediction result of the second pixel decoder are used as pseudo labels of the past categories of the training samples to supplement the annotations in the training samples;

[0039] The backtracking module is configured to: after supplementing the annotations in the training sample, obtain an assigned image after assigning the labels of the output slices of the training sample, and backtrack the assigned image and the second feature output by the second Transformer decoder to obtain a third feature;

[0040] The classification output module is configured to obtain a classification output result of the image based on the third feature and the fourth feature.

[0041] One or more of the above technical solutions have the following beneficial effects:

[0042] The technical solution of the present invention proposes Past-Class Backtrace Distillation (PCBD), which aims to balance the dual needs of memorizing old classes and learning new classes. At the same time, the technical solution of the present invention also proposes a class-proportional memory strategy (CPM) to ensure the reasonable frequency of each class during the playback process. Through these improvements, the present invention improves the overall performance of the model in the continuous panoramic segmentation task.

[0043] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0045] Figure 1 The overall network architecture diagram of the method according to the embodiment of the present invention;

[0046] Figure 2 Schematic diagram of past-type retrospective distillation (PCBD) according to an embodiment of the present invention;

[0047] Figure 3 Schematic diagram of the visualization result of continuous panoramic segmentation according to an embodiment of the present invention, wherein: (a) input image; (b) prediction result of BCPS; and (c) true label. DETAILED DESCRIPTION

[0048] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0049] It should be noted that the terms used herein are for describing specific embodiments only and are not intended to be limiting of exemplary embodiments according to the present invention.

[0050] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0051] Different from traditional panoramic segmentation, the key challenge facing continuous panoramic segmentation (CPS) is how to keep the memory of old categories while continuously learning new categories, ensuring that the model does not forget the learned categories in the process of mastering new categories.

[0052] The sub-technical solution of this embodiment proposes Past-Class Backtrace Distillation (PCBD), which aims to balance the dual needs of memorizing old classes and learning new classes. At the same time, a class-proportional memory strategy (CPM) is proposed to ensure the reasonable frequency of each class during playback. Through these improvements, the sub-technical solution of this embodiment improves the overall performance of the model in the continuous panoramic segmentation task.

[0053] Embodiment 1

[0054] This embodiment discloses a balanced continuous panoramic segmentation method (BCPS), which uses past class retrospective distillation (PCBD) to trace back the features related to the old class output slices after the label assignment of the network output is completed, and performs knowledge distillation on these features, thereby ensuring the recognition accuracy of the past classes and allowing other features to flexibly adapt to new knowledge. In addition, a class proportional memory strategy (CPM) is adopted to select a replay sample set according to the class distribution of the historical training data, so that the class distribution in the replay sample set is consistent with the class distribution in the historical training data, thereby ensuring that important classes occupy a larger proportion in the replay sample set and enhancing the recall efficiency of past classes.

[0055] See attached Figure 1 As shown in the figure, given regular training samples and replay samples as input, the current model and the previous model process it simultaneously. For regular training samples, the output of the previous model is used as a pseudo-label to supplement the annotation of the past category. After the label assignment, we trace back to the previous features related to the output slices labeled as past categories, and apply knowledge distillation loss between the previous model and the current model for these features. At the same time, both the replay sample and the regular training sample output slices are also guided by the classification loss and the segmentation loss.

[0056] Specifically, we first introduce that in the continuous panoptic segmentation task, the training is divided into T steps, and incremental training is performed in T steps. In each step t∈1,…,T, the training set D t Contains a series of image-label pairs. Each image label contains N gt True value blocks, represented as

[0057] in, is the true value of the category, is the true value of the binary segmentation map, H×W represents the spatial dimension of the image, H is the height of the image, and W is the width of the image. Represents the set of categories that are concerned in step t. gt is the abbreviation of ground-truth. In this embodiment, all capital Ns represent quantities. When the superscript is gt, it represents the number of ground-truth segments; when the superscript is q, it represents the number of query vectors.

[0058] It should be noted that the category sets of different steps are disjoint. After training in step t, the goal is to enable the model to accurately predict all seen categories (denoted as ). The seen categories include all the categories that the current step focuses on and all the categories that the previous steps focus on. This embodiment discloses a balanced continuous panoramic segmentation method, such as Figure 1 As shown in Figure 2, it is built on the widely used panoramic segmentation framework Mask2Former. Its input includes D t The regular training images in and the replay samples retained by the category proportional memory strategy (CPM). Given a batch, the previous model M t-1 and the current model M t Parallel operation, the "previous model" refers to the model after the previous step of training, which is used to provide knowledge, and the "current model" refers to the current state of the model, which needs to be trained. t-1 The parameters are frozen. t-1 The prediction of is used as the pseudo label of the past category of the regular image, supplemented by D t After assigning labels to the output chunks (via binary matching), this embodiment backtracks the features associated with the chunks marked as past categories in the Transformer decoder. t-1 To M t Perform knowledge distillation, that is, from M t-1 To M t Transfer knowledge. The specific operation is given by the following formula (5).

[0059] Finally, the classification loss and segmentation loss The output results of supervised replay samples and regular images are given by the following formulas:

[0060]

[0061] Among them, c i and m i Respectively represent the classification results and binary segmentation maps of the output blocks, and Respectively represent their corresponding true values. Focal represents Focal loss, and Dice represents Dice loss. For details, please refer to the literature Lin TY, Goyal P, Girshick R, et al. Focal loss for dense object detection[C] / / Proceedings of the IEEE international conference on computer vision.2017:2980-2988. and the literature Milletari F, Navab N, Ahmadi S AV-net: Fully convolutional neural networks for volumetric medical image segmentation[C] / / 2016 fourth international conference on 3D vision(3DV).Ieee,2016:565-571.

[0062] In all continuous learning scenarios, a primary challenge is to balance the stability of existing knowledge with adaptability to new information. Existing continuous segmentation methods use knowledge distillation to limit changes in the entire feature or output. Although this restriction retains existing knowledge, it may weaken the ability to absorb new information. The sub-technical scheme of this embodiment believes that by identifying specific features related to existing knowledge, an optimal balance between stability and adaptability can be achieved. Only limiting these identified features and allowing other features to develop freely can fully utilize the network's ability to integrate past and new knowledge. Based on this idea, a past category backtracking distillation strategy is proposed. For more information about past category backtracking distillation, see Appendix. Figure 2 shown.

[0063] In the network, features fuse content from past and new categories, making it challenging to identify parts related to past categories. However, at the end of the workflow, during the label assignment process, each output slice is matched to an appropriate label. In the basic model of the sub-technical solution of this embodiment, this assignment is performed through similarity-based bipartite matching. Therefore, output slices that are close to past category annotations (derived from pseudo-labels or replayed samples) are assigned to past category labels, while those that are similar to new category annotations are labeled as new categories accordingly. This process makes it possible to identify slices corresponding to past categories. By tracing back to the sources of these slices, the sub-technical solution of this embodiment can identify features related to past categories.

[0064] Specifically, the output for each input image consists of a set of N q A block, represented by where c i represents classification prediction, m i N represents mask prediction. q is the total number of query vectors (Query), and therefore the number of output tiles. According to the workflow:

[0065]

[0066] in, and denote the output features of the Transformer decoder and pixel decoder respectively. MLP denotes multi-layer perceptron, sigmoid is the Sigmoid function, and [·] denotes the channel or element index. The above formula means that if the i-th output block is assigned a past category label, then The i-th channel of is specifically related to the past category.

[0067] Since the characteristics of the Transformer decoder output By initial query and backbone feature f b The operation of each attention layer is Where s∈[1,S] represents the number of layers and Attn represents the attention operation. Given the one-to-one correspondence between the input and output channels of the attention layer, for all s∈[1,S], if the i-th output slice is marked as the past category, then the corresponding It is related to the past category.

[0068] Based on the above backtracking process, after obtaining the label assignment results, the sub-technical solution of this embodiment can identify all features related to the past categories. Subsequently, the sub-technical solution of this embodiment performs knowledge distillation from the previous model to the current model on these identified features, and obtains the distillation loss of the sub-technical solution of this embodiment

[0069]

[0070] here, represents the indicator function, ε past Refers to the set of output blocks that are assigned past category labels, MSE means mean square error, Represents the previous model M t-1 This distillation focuses on features related to past categories, achieving a balance between stability and adaptability.

[0071] About the memory of proportional categories:

[0072] Existing continuous learning research has verified that replaying some samples of past categories is effective in preventing catastrophic forgetting. In addition, by enhancing the network's ability to distinguish between past and new categories, these replay samples also help improve the performance of identifying new categories. However, due to the limited capacity of the replay sample set, the selection of samples will greatly affect the effect. This leads to the second key balance point: the balance of categories within the replay sample set. This balance should not be just an average distribution, but should reflect the cumulative category distribution of past training sets. This is because the more common categories in the training set usually show greater diversity, making it more challenging to fully understand their representation. In addition, categories that appear frequently in the dataset are usually more common in related application scenarios. Therefore, it is both important and reasonable to include more samples of these categories in the replay sample set. In addition, this consistency of category distribution helps to maintain a stable classification tendency for past categories. Based on these considerations, a category-proportional memory strategy is designed.

[0073] Specifically, in the first training step, for each current category, The number of occurrences of the real cut block is denoted by π c These counts enable the calculation of the class distribution Π 1 ,Right now Using this distribution, we constructed a r Sample set of samples Simply put, the goal is to learn from the current training set D 1 Select N r images to best approximate the desired class distribution Π 1 Since the computational complexity of finding the global optimal solution for this task is high, a greedy algorithm is used to efficiently obtain a usable local solution. Specifically, the greedy algorithm is used in D 1 Iterate N over all images in r times, each time selecting an image to minimize the difference between the changing class distribution and Π 1 The process can be concisely expressed as:

[0074]

[0075] Where Ω represents the greedy algorithm.

[0076] At each subsequent step t, calculate The number of occurrences of and update the cumulative category distribution as Using this distribution, the sub-technical solution of this embodiment updates the replay sample set. Select the image in D tUsually contains many more images than The local optimization of the greedy algorithm tends to choose fewer Therefore, after updating the replay sample set multiple times, there may be insufficient images of categories multiple steps ago. To solve this problem, a constraint is introduced. Time Retention A certain proportion images, thereby reducing the risk of over-diluting early category samples. Here, |·| represents the cardinality of the set. The update mechanism of the replay sample set is formulated as:

[0077]

[0078] in Indicates the first item The category distribution of Select λ t N r Class distribution behind the image.

[0079] In each incremental training step t, the sub-technical solution of this embodiment will With D t Combined training. Category-proportional selection of replay samples plays a vital role in maintaining past category knowledge and helping to distinguish between new and old categories. r Within the size limit of , the method of the sub-technical solution of this embodiment prioritizes categories that are important in practical applications and are more likely to show wide diversity. This maximizes the value of the replay sample set to help memory and ensures a stable classification tendency. In addition, by calculating the category distribution based on the count of the slices rather than the count of each image class, the strategy of the sub-technical solution of this embodiment better meets the instance perception requirements of CPS.

[0080] About the loss function:

[0081] To summarize, the total training loss in an incremental step is defined as:

[0082]

[0083] in, and They represent classification and segmentation losses, respectively, and their definitions are consistent with those in Mask2Former. [4] α, β, and γ are hyperparameters for balancing several losses and are set to 2, 5, and 5 respectively. In the basic step (i.e., the first step), the total loss does not include The other parts are consistent with Formula 6.

[0084] The sub-technical solution of this embodiment provides a new continuous panoramic segmentation framework BCPS, which is committed to maintaining the memory of old class knowledge during incremental learning and can effectively adapt and learn new information. Experimental results on the ADE20K dataset show that the performance of BCPS is better than the existing state-of-the-art methods.

[0085] This embodiment proposes a past class back-propagation distillation (PCBD) technique, which, after assigning labels at the network output, back-propagates features related to past classes for targeted knowledge distillation, thereby maintaining the recognition accuracy of old classes while allowing other features to flexibly adapt to new knowledge.

[0086] The sub-technical solution of this embodiment proposes a class proportional memory strategy (CPM). This strategy makes the class distribution in the replay sample set consistent with the class distribution of the historical training data, thereby achieving a true balance when recalling past classes and improving the utility of the replay sample set.

[0087] The sub-technical scheme of this embodiment aims to solve the key problems in continuous panoramic segmentation (CPS), and has significant advantages over the existing technology. The existing state-of-the-art technology mainly prevents catastrophic forgetting by distilling knowledge for the entire feature or output. However, these methods may limit the ability to learn new categories while ensuring the memory of old categories. The present invention proposes a past category backtracking distillation method, which strikes a balance between knowledge stability and adaptability by selectively distilling features related to past categories while allowing other features to freely adapt to new knowledge. In addition, the present invention introduces a class-proportional memory strategy to select playback samples based on the cumulative class distribution of historical training sets, so that the playback sample set more accurately reflects the true distribution of classes, and improves the memory ability of past knowledge and the consistency of classification tendencies. Experimental results on the ADE20K dataset show that the performance of the continuous panoramic segmentation framework BCPS proposed in the present invention is significantly better than the existing state-of-the-art technology. This not only verifies the effectiveness of each innovative mechanism, but also demonstrates the practicality and superiority of the overall framework.

[0088] The schematic diagram of the visualization result of continuous panoptic segmentation according to the embodiment of the present invention is shown in the attached Figure 3 As shown in the figure, (a) input image; (b) prediction result of BCPS; (c) true label.

[0089] The sub-technical solution of this embodiment introduces a past class back-testing distillation method (PCBD): by selectively distilling features related to old classes, while maintaining the stability of old knowledge, other features are allowed to flexibly adapt to new knowledge, thereby achieving a balance between stability and adaptability.

[0090] The sub-technical solution of this embodiment proposes a category proportional memory strategy (CPM): constructing a replay sample set based on the class distribution of historical training data to truly reflect the balance of class distribution, enhance the practicality of the replay sample set in recalling old class knowledge, and ensure the stability of classification tendency.

[0091] Embodiment 2

[0092] The purpose of this embodiment is to provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the program.

[0093] Embodiment 3

[0094] The purpose of this embodiment is to provide a computer-readable storage medium.

[0095] A computer-readable storage medium stores a computer program, which executes the steps of the above method when executed by a processor.

[0096] Embodiment 4

[0097] The purpose of this embodiment is to provide a balanced continuous panoptic segmentation system, including:

[0098] The sample acquisition module is configured to: acquire training samples and replay samples;

[0099] The parallel operation processing module is configured to: use the first model and the second model to perform parallel operation processing on the acquired training samples and the replay samples respectively;

[0100] The first model outputs the prediction results to the first pixel decoder and the first Transformer decoder respectively, performs feature extraction on the output of the first Transformer decoder to obtain the first feature, and the first feature is superimposed on the prediction result of the first pixel decoder as a pseudo label of the past category of the training sample to supplement the annotation in the training sample;

[0101] Perform feature extraction again on the first feature to obtain a fourth feature;

[0102] The second model outputs the prediction results to the second pixel decoder and the second Transformer decoder respectively, performs feature extraction on the output of the second Transformer decoder to obtain the second feature, and the second feature and the prediction result of the second pixel decoder are used as pseudo labels of the past categories of the training samples to supplement the annotations in the training samples;

[0103] The backtracking module is configured to: after supplementing the annotations in the training sample, obtain an assigned image after assigning the labels of the output slices of the training sample, and backtrack the assigned image and the second feature output by the second Transformer decoder to obtain a third feature;

[0104] The classification output module is configured to obtain a classification output result of the image based on the third feature and the fourth feature.

[0105] Embodiment 5

[0106] The purpose of this embodiment is to provide a computer program product containing instructions, which, when running on a computer, enables the computer to execute the methods and functions involved in any of the above embodiments.

[0107] The steps involved in the apparatus of the above embodiment correspond to the method embodiment 1, and the specific implementation method can refer to the relevant description part of embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood to include any medium that can store, encode or carry an instruction set for execution by a processor and enable the processor to execute any method in the present invention.

[0108] Those skilled in the art should understand that the modules or steps of the present invention described above can be implemented by a general-purpose computer device, or alternatively, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0109] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A balanced continuous panoptic segmentation method, characterized in that include: Obtain training samples and replay samples; The first model and the second model are used to perform parallel operation processing on the acquired training samples and the replay samples respectively; The first model outputs the prediction results to the first pixel decoder and the first Transformer decoder respectively, performs feature extraction on the output of the first Transformer decoder to obtain the first feature, and the first feature is superimposed on the prediction result of the first pixel decoder as a pseudo label of the past category of the training sample to supplement the annotation in the training sample; Perform feature extraction again on the first feature to obtain a fourth feature; The second model outputs the prediction results to the second pixel decoder and the second Transformer decoder respectively, performs feature extraction on the output of the second Transformer decoder to obtain the second feature, and the second feature and the prediction result of the second pixel decoder are used as pseudo labels of the past categories of the training samples to supplement the annotations in the training samples; After supplementing the annotations in the training sample, the labels of the output blocks of the training sample are assigned to obtain the assigned image, and the assigned image is back-processed with the second feature output by the second Transformer decoder to obtain the third feature; A classification output result of the image is obtained based on the third feature and the fourth feature.

2. A balanced continuous panoptic segmentation method as claimed in claim 1, characterized in that: The training samples include multiple image-label pairs, the label of each image includes multiple true value blocks, and the replay samples are replay samples retained by a category-proportional memory strategy.

3. A balanced continuous panoptic segmentation method as claimed in claim 1, characterized in that: The output for each input image in the training sample consists of a set of N q A block, represented by where c i represents classification prediction, m i represents mask prediction, N q is the total number of query vectors and the number of output tiles.

4. A balanced continuous panoptic segmentation method as claimed in claim 1, characterized in that: Obtaining distillation loss based on the third and fourth characteristics here, represents the indicator function, ε past Refers to the set of output blocks that are assigned past category labels, MSE means mean square error, Represents the previous model M t-1 Transformer features.

5. A balanced continuous panoptic segmentation method as claimed in claim 1, characterized in that: The specific process of obtaining replay samples through the category-proportional memory strategy is as follows: In the first training step, the number of occurrences of each true cut of the current class in the training samples is counted. These counts enable the calculation of the class distribution, which is used to construct a sample set containing the required number of samples.

6. A balanced continuous panoptic segmentation method as claimed in claim 1, characterized in that: To construct a sample set containing the required number of samples, a greedy algorithm is used to obtain a usable local solution; specifically, the greedy algorithm iterates the required number of times on all images in the training sample, selecting one image each time to minimize the difference between the changing category distribution and the calculated category distribution; In each subsequent step, the number of occurrences of the current category is calculated and the cumulative category distribution is updated. Using this distribution, the replay sample set is updated.

7. A balanced continuous panoptic segmentation system, characterized in that include: The sample acquisition module is configured to: acquire training samples and replay samples; The parallel operation processing module is configured to: use the first model and the second model to perform parallel operation processing on the acquired training samples and the replay samples respectively; The first model outputs the prediction results to the first pixel decoder and the first Transformer decoder respectively, performs feature extraction on the output of the first Transformer decoder to obtain the first feature, and the first feature is superimposed on the prediction result of the first pixel decoder as a pseudo label of the past category of the training sample to supplement the annotation in the training sample; Perform feature extraction again on the first feature to obtain a fourth feature; The second model outputs the prediction results to the second pixel decoder and the second Transformer decoder respectively, performs feature extraction on the output of the second Transformer decoder to obtain the second feature, and the second feature and the prediction result of the second pixel decoder are used as pseudo labels of the past categories of the training samples to supplement the annotations in the training samples; The backtracking module is configured to: after supplementing the annotations in the training sample, obtain an assigned image after assigning the labels of the output slices of the training sample, and backtrack the assigned image and the second feature output by the second Transformer decoder to obtain a third feature; The classification output module is configured to obtain a classification output result of the image based on the third feature and the fourth feature.

8. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method described in any one of claims 1 to 6 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method described in any one of claims 1 to 6 are performed.

Citation Information

Patent Citations

  • Pseudo-label model training method and device, equipment and storage medium

    CN115082676A

  • Training method and device based on image annotation and segmentation model

    CN115546605A

  • Test scene-oriented maskless supervision small sample segmentation method

    CN116824140A

  • Continuous open world target detection method based on knowledge distillation and sample playback

    CN118247488A

  • Robust training in the presence of label noise

    US20210089964A1