A Balanced Continuous Panoramic Segmentation Method and System

Abstract

The present invention provides a balanced continuous panoramic segmentation method and system, including: obtaining training samples and replay samples; respectively performing parallel operation processing on the obtained training samples and replay samples by using a first model and a second model; after supplementing the annotations in the training samples, obtaining an assigned image after label assignment of the output chunks of the training samples, and performing backtracking processing on the assigned image and the second feature output by the second Transformer decoder to obtain a third feature; obtaining a classification output result of the image.

Description

Technical Field

[0001] The present invention belongs to the technical field of continuous panoramic segmentation, and particularly relates to a balanced continuous panoramic segmentation method and system. Background Art

[0002] The statements in this part only provide background technical information related to the present invention and do not necessarily constitute prior art.

[0003] In the field of computer vision, panoptic segmentation is an important fundamental task. It combines the concepts of semantic segmentation and instance segmentation, aiming to classify each pixel in an image into a specific semantic category and distinguish different instances.

[0004] In continual panoptic segmentation (CPS), the training process consists of multiple steps, and each step focuses on different categories. In each step, only some categories are regarded as foregrounds and labeled with true labels, while the foreground categories from past or future steps are labeled as backgrounds.

[0005] In addition, traditional panoramic segmentation methods are usually trained on static datasets. When the dataset is updated, the entire network needs to be retrained. This method leads to catastrophic forgetting, that is, the model will forget the information learned previously. Therefore, how to enable the panoramic segmentation model to learn new information without forgetting the knowledge already mastered has become an important research topic, called continual panoptic segmentation (CPS).

[0006] Existing continual panoptic segmentation methods cope with catastrophic forgetting by knowledge distillation at the output level. However, while this method preserves old knowledge, it may limit the learning of new knowledge. In addition, the replay techniques in existing continual semantic segmentation (CSS) also have the potential for use in CPS. Such methods save some samples from past steps and replay them in future steps to retain memory. However, the preservation of the replay samples does not consider the rationality of the category distribution, often selecting the same number of images for each category and ignoring the differences in the frequency of category occurrences in the training data.

[0007] Therefore, in the process of implementing the present invention, the inventor found that there are at least the following disadvantages and deficiencies in the prior art:

[0008] There is a lack of balance between knowledge stability and adaptability in existing continuous panoramic segmentation: Existing methods alleviate the forgetting problem through knowledge distillation, but these methods usually distill all features or outputs, which to some extent limits the learning ability of the model for processing images to new knowledge. Therefore, it is necessary to consider how to effectively ensure the flexible adaptation of new knowledge while maintaining the stability of old knowledge.

[0009] The class distribution in the replay sample set in existing continuous panoramic segmentation is unbalanced: Traditional methods select an equal number of images for each class in the replay sample set, ignoring the differences in the frequencies of various classes in the training data. This neglect results in the class distribution in the replay sample set not reflecting the cumulative distribution in the historical training data, thus affecting the memory of old classes. Therefore, it is necessary to consider how to make the class distribution in the replay sample set better reflect the cumulative distribution in the historical training data to enhance the memory of old classes. Summary of the Invention

[0010] To overcome the deficiencies of the above-mentioned prior art, the present invention provides a balanced continuous panoramic segmentation method, which allows other features to flexibly adapt to new knowledge while maintaining the stability of old knowledge, thereby achieving a balance between stability and adaptability.

[0011] To achieve the above object, one or more embodiments of the present invention provide the following technical solutions:

[0012] In a first aspect, a balanced continuous panoramic segmentation method is disclosed, including:

[0013] Obtain training samples and replay samples;

[0014] Use a first model and a second model to perform parallel operation processing on the obtained training samples and replay samples respectively;

[0015] Among them, the first model outputs the prediction results to a first pixel decoder and a first Transformer decoder respectively, extracts features from the output of the first Transformer decoder to obtain a first feature, and superimposes the first feature and the prediction result of the first pixel decoder as the pseudo-label of the past classes of the training samples to supplement the annotations in the training samples;

[0016] Extract features from the first feature again to obtain a fourth feature;

[0017] The second model outputs the prediction results to a second pixel decoder and a second Transformer decoder respectively, extracts features from the output of the second Transformer decoder to obtain a second feature, and uses the second feature and the prediction result of the second pixel decoder as the pseudo-label of the past classes of the training samples to supplement the annotations in the training samples;

[0018] After annotating the supplementary training samples, the assigned image is obtained after the label assignment of the output chunks of the training samples. The third feature is obtained by backtracking the assigned image and the second feature output by the second Transformer decoder;

[0019] Based on the third feature and the fourth feature, the classification output result of the image is obtained.

[0020] As a further technical solution, the training samples include multiple image-label pairs, the label of each image contains multiple ground-truth chunks, and the replay samples are replay samples retained by the class-proportional memory strategy.

[0021] As a further technical solution, during the parallel operation processing of the first model, the parameters are frozen.

[0022] As a further technical solution, the label assignment of the output chunks of the training samples is implemented by using the bipartite matching method.

[0023] As a further technical solution, the output of each input image in the training samples includes a set of N q chunks, denoted as where c i represents the classification prediction, m i represents the mask prediction, N q is the total number of query vectors and also the number of output chunks.

[0024] As a further technical solution, the distillation loss is obtained based on the third feature and the fourth feature

[0025]

[0026] Here, represents the indicator function, ε past refers to the set of output chunks assigned with past class labels, MSE represents the mean squared error, represents the Transformer feature of the previous model M t-1 of.

[0027] As a further technical solution, the specific process of obtaining the replay samples by the class-proportional memory strategy is as follows:

[0028] In the first training step, calculate the occurrence times of the ground-truth chunks of each current class in the training samples. These counts enable the calculation of the class distribution. Using this distribution, construct a sample set containing the required number of samples.

[0029] As a further technical solution, a greedy algorithm is used to obtain a usable local solution in constructing a sample set containing the required number of samples; specifically, the greedy algorithm iterates the required number of times on all images in the training sample, and each time selects an image to minimize the difference between the changing category distribution and the calculated category distribution;

[0030] In each subsequent step, the number of occurrences of the current category is calculated and the cumulative category distribution is updated. Using this distribution, the replay sample set is updated.

[0031] As a further technical solution, after updating the replay sample set multiple times, a constraint is introduced to form Time Retention A certain proportion of the image.

[0032] As a further technical solution, in each incremental training step, With D t Combined training.

[0033] In a second aspect, a balanced continuous panoptic segmentation system is disclosed, comprising:

[0034] The sample acquisition module is configured to: acquire training samples and replay samples;

[0035] The parallel operation processing module is configured to: use the first model and the second model to perform parallel operation processing on the acquired training samples and the replay samples respectively;

[0036] The first model outputs the prediction results to the first pixel decoder and the first Transformer decoder respectively, performs feature extraction on the output of the first Transformer decoder to obtain the first feature, and the first feature is superimposed on the prediction result of the first pixel decoder as a pseudo label of the past category of the training sample to supplement the annotation in the training sample;

[0037] Perform feature extraction again on the first feature to obtain a fourth feature;

[0038] The second model outputs the prediction results to the second pixel decoder and the second Transformer decoder respectively, performs feature extraction on the output of the second Transformer decoder to obtain the second feature, and the second feature and the prediction result of the second pixel decoder are used as pseudo labels of the past categories of the training samples to supplement the annotations in the training samples;

[0039] The backtracking module is configured to: after supplementing the annotations in the training sample, obtain an assigned image after assigning the labels of the output slices of the training sample, and backtrack the assigned image and the second feature output by the second Transformer decoder to obtain a third feature;

[0040] The classification output module is configured to: obtain the classification output result of the image based on the third feature and the fourth feature.

[0041] The above one or more technical solutions have the following beneficial effects:

[0042] The technical solution of the present invention proposes Past-Class Backtrace Distillation (PCBD), aiming to balance the dual needs of memorizing old classes and learning new classes. At the same time, the technical solution of the present invention also proposes a Class-Proportional Memory (CPM) strategy to ensure reasonable frequencies of each class during the replay process. Through these improvements, the present invention improves the overall performance of the model in the continuous panoramic segmentation task.

[0043] Advantages of additional aspects of the present invention will be given in part in the following description, become apparent in part from the following description, or be learned through the practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The accompanying drawings constituting a part of this specification are used to provide a further understanding of the present invention. The schematic embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation to the present invention.

[0045] Figure 1 It is the overall network architecture diagram of the method of the embodiment of the present invention;

[0046] Figure 2 It is the schematic diagram of Past-Class Backtrace Distillation (PCBD) of the embodiment of the present invention;

[0047] Figure 3 It is the schematic diagram of the visualization result of continuous panoramic segmentation of the embodiment of the present invention, where (a) is the input image; (b) is the prediction result of BCPS; (c) is the ground truth label. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0048] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which the present invention belongs.

[0049] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present invention.

[0050] In the case of no conflict, the embodiments in the present invention and the features in the embodiments can be combined with each other.

[0051] Different from traditional panoptic segmentation, the key challenge faced by continuous panoptic segmentation (CPS) is how to maintain the memory of old classes while continuously learning new classes, ensuring that the model does not forget the learned classes during the process of mastering new classes.

[0052] The technical solution of this embodiment proposes Past-Class Backtrace Distillation (PCBD) to balance the dual needs of remembering old classes and learning new classes. At the same time, a Class-Proportional Memory (CPM) strategy is proposed to ensure the reasonable frequency of each class during the replay process. Through these improvements, the technical solution of this embodiment improves the overall performance of the model in the continuous panoptic segmentation task.

[0053] Example 1

[0054] This embodiment discloses a Balanced Continuous Panoptic Segmentation method (BCPS). Through Past-Class Backtrace Distillation (PCBD), after the label assignment of the network output is completed, the features related to the output chunks of old classes are traced, and knowledge distillation is performed on these features to ensure the recognition accuracy of past classes while allowing other features to flexibly adapt to new knowledge. In addition, a Class-Proportional Memory (CPM) strategy is adopted. According to the class distribution of historical training data, a replay sample set is selected to make the class distribution in the replay sample set consistent with the class distribution in the historical training data, so as to ensure that important classes account for a larger proportion in the replay sample set and enhance the recall efficiency of past classes.

[0055] See Appendix Figure 1 As shown, given regular training samples and replay samples as input, the current model and the previous model process it simultaneously. For regular training samples, the output of the previous model is used as pseudo-labels to supplement the annotations of past classes. After label assignment, we trace back to the pre-order features related to the output chunks labeled as past classes, and impose a knowledge distillation loss on these features between the previous model and the current model. At the same time, the output chunks of replay samples and regular training samples are also guided by classification loss and segmentation loss.

[0056] Specifically, first, it is introduced that in the continuous panoptic segmentation task, the training is divided into T steps, and incremental training is performed in these T steps. At each step t ∈ {1, …, T}, the training set D t contains a series of image-label pairs. The label of each image contains N gt ground-truth chunks, denoted as

[0057] where, is the class ground truth, is the true value of the binary segmentation map. H×W represents the spatial dimension of the image, where H is the height of the image and W is the width of the image. represents the set of categories of interest at step t. gt is the abbreviation of ground-truth. In this example, all uppercase Ns represent quantities. When the superscript is gt, it represents the number of ground-truth segments; when the superscript is q, it represents the number of query vectors.

[0058] It should be noted that the category sets in different steps are disjoint. After training in step t, the goal is to enable the model to accurately predict the segmentation of all seen categories (denoted as ). The seen categories include all categories of interest in the current step and all categories of interest in previous steps. This example discloses a balanced continuous panoramic segmentation method, as shown in Figure 1 , which is built on the widely used panoramic segmentation framework Mask2Former. Its input includes the regular training images in D t and the replay samples retained by the class-proportional memory strategy (CPM). Given a batch, the previous model M t-1 and the current model M t operate in parallel. The "previous model" refers to the model after being trained in the previous step and is used to provide knowledge. The "current model" refers to the current state of the model that needs to be trained. The parameters of M t-1 are frozen. Referring to PLOP, the predictions of M t-1 are used as pseudo-labels for the past categories of the regular images to supplement the annotations in D t . After the label assignment of the output segments (achieved through bipartite matching), the technical solution of this example traces back the features in the Transformer decoder associated with the segments labeled as past categories. For these features, knowledge distillation is performed from M t-1 to M t , that is, knowledge is transferred from M t-1 to M t . The specific operation is given by formula (5) later.

[0059] Finally, the classification loss and the segmentation loss supervise the output results of the replay samples and the regular images, respectively, and are given by the following formulas:

[0060]

[0061] where c i and m i represent the classification result and the binary segmentation map of the output segments, respectively, and respectively represent their corresponding true values. Focal represents the Focal loss, and Dice represents the Dice loss. For details, see the literature Lin T Y, Goyal P, Girshick R, et al. Focal loss for dense object detection[C] / / Proceedings of the IEEE international conference on computer vision. 2017:2980 - 2988. and the literature Milletari F, Navab N, Ahmadi S A. V-net: Fully convolutional neural networks for volumetric medical image segmentation[C] / / 2016 fourth international conference on 3D vision(3DV). Ieee, 2016:565 - 571.

[0062] In all continuous learning scenarios, a primary challenge is to balance the stability of existing knowledge and the adaptability to new information. Existing continuous segmentation methods use knowledge distillation to limit the changes in the entire feature or output. Although this limitation preserves existing knowledge, it may weaken the ability to absorb new information. The technical solution of this embodiment believes that by identifying specific features related to existing knowledge, the best balance between stability and adaptability can be achieved. Only restricting these identified features and allowing other features to develop freely can make full use of the network's ability to integrate past and new knowledge. Based on this idea, the past category backtracking distillation strategy is proposed. Regarding the past category backtracking distillation, see the appendix Figure 2 as shown.

[0063] In the network, the features incorporate the content of past and new categories, making it challenging to identify the parts related to past categories. However, at the end of the workflow, during the label assignment process, each output chunk is matched with an appropriate label. In the base model of the technical solution of this embodiment, this assignment is performed through similarity-based bipartite matching. Therefore, output chunks close to (derived from pseudo-labels or replay samples) past category annotations are assigned to past category labels, while those similar to new category annotations are correspondingly labeled as new categories. This process makes it possible to identify the chunks corresponding to past categories. By tracing back the sources of these chunks, the technical solution of this embodiment can identify the features related to past categories.

[0064] Specifically, the output of each input image includes a set of N qThe cut blocks, denoted as where c i represents the classification prediction, and m i represents the mask prediction. N q is the total number of query vectors (Queries), and thus also the number of output cut blocks. According to the workflow:

[0065]

[0066] Among them, and represent the output features of the Transformer decoder and the pixel decoder respectively. MLP represents the multi-layer perceptron, sigmoid is the Sigmoid function, and [·] represents the channel or element index. The above formula means that if the i-th output cut block is assigned the past class label, then the i-th channel of is specifically related to that past class.

[0067] Since the features output by the Transformer decoder are generated by applying the attention layer layer by layer to the initial query and the backbone feature f b The operation of each attention layer is where s ∈ [1, S] represents the layer number, and Attn represents the attention operation. Given the one-to-one correspondence between the input and output channels of the attention layer, for all s ∈ [1, S], if the i-th output cut block is labeled as the past class, then the corresponding is related to the past class.

[0068] Based on the above backtracking process, after obtaining the label assignment result, the technical solution of this embodiment can identify all features related to the past class. Subsequently, the technical solution of this embodiment performs knowledge distillation from the previous model to the current model on these identified features, and obtains the distillation loss of the technical solution of this embodiment

[0069]

[0070] Here, represents the Indicator Function, and ε past refers to the set of output cut blocks assigned the past class label, MSE represents the mean squared error, represents the Transformer features of the previous model M t-1 This distillation focuses on the features related to the past class, achieving a balance between stability and adaptability.

[0071] Memory proportional to the class:

[0072] Existing continuous learning research has verified that replaying some samples of past classes is effective in preventing catastrophic forgetting. Additionally, by enhancing the network's ability to distinguish between past and new classes, these replayed samples also help improve the performance of identifying new classes. However, due to the limited capacity of the replayed sample set, the selection of samples greatly affects the effect. This leads to the second key balance point: the class balance within the replayed sample set. This balance should not be just an equal distribution, but should reflect the cumulative class distribution of the past training set. This is because the more common classes in the training set usually exhibit greater diversity, making it more challenging to comprehensively understand their representations. Moreover, the classes that frequently appear in the dataset are usually more common in the relevant application scenarios. Therefore, it is both important and reasonable to include more samples of these classes in the replayed sample set. Additionally, this consistency in class distribution helps maintain a stable classification tendency for past classes. Based on these considerations, a class-proportional memory strategy is designed.

[0073] Specifically, in the first training step, the occurrence times of the true chunks of each current class are calculated and denoted as π c . These counts enable the calculation of the class distribution Π 1 , that is Using this distribution, a sample set containing N r samples is constructed Briefly speaking, the goal is to select N 1 images from the current training set D r to best approximate the desired class distribution Π 1 . Since the computational complexity of finding the global optimal solution for this task is high, a greedy algorithm is adopted to efficiently obtain a usable local solution. Specifically, the greedy algorithm iterates N 1 times over all the images in D r , each time selecting an image to minimize the difference between the changing class distribution and Π 1 . This process can be concisely expressed as:

[0074]

[0075] where Ω represents the greedy algorithm.

[0076] In each subsequent step t, the occurrence times of are calculated, and the cumulative class distribution is updated to Using this distribution, the technical solution of this embodiment updates the replayed sample set. However, since the technical solution of this embodiment selects images from , and Dt usually contains far more images than The local optimization of the greedy algorithm tends to select fewer images in. Therefore, after multiple updates of the replay sample set, there may be a shortage of class images from several steps ago. To solve this problem, a constraint is introduced to retain a certain proportion when forming of the images in, thus reducing the risk of over-diluting early class samples. Here, |·| represents the cardinality of the set. The update mechanism of the replay sample set is formulated as: of the images, thus reducing the risk of over-diluting early class samples. Here, |·| represents the cardinality of the set. The update mechanism of the replay sample set is formulated as:

[0077]

[0078] where represents the class distribution of the first term , that is, the class distribution after selecting λ images from t N r images.

[0079] In each incremental training step t, the technical solution of this embodiment combines with D t for training. Selecting replay samples in proportion to the classes plays a crucial role in retaining past class knowledge and helping to distinguish between new and old classes. Within the size limit of N r , the method of the technical solution of this embodiment gives priority to the classes that are important in practical applications and are more likely to exhibit a wide variety of diversity. This maximizes the value of the replay sample set for helping memory and ensures a stable classification tendency. In addition, by calculating the class distribution based on chunk counts rather than image counts per class, the strategy of the technical solution of this embodiment better meets the instance-aware requirements of CPS.

[0080] Regarding the loss function:

[0081] All in all, the total training loss in the incremental step is defined as:

[0082]

[0083] where and respectively represent the classification segmentation loss, defined to be consistent with Mask2Former [4] . α, β, and γ are hyperparameters for balancing several losses, set to 2, 5, and 5 respectively. In the base step (i.e., the first step), the total loss does not include The other parts are consistent with Equation 6.

[0084] The technical solution of this embodiment provides a new continuous panoramic segmentation framework BCPS, which is committed to maintaining the memory of old-class knowledge during the incremental learning process and can effectively adapt to and learn new information. Experimental results on the ADE20K dataset show that the performance of BCPS is better than that of existing state-of-the-art methods.

[0085] The technical solution of this embodiment proposes Past-Class Backward Distillation (PCBD). Through this technique, after assigning labels at the network output, relevant features related to past classes are traced back for targeted knowledge distillation, so as to maintain the recognition accuracy of old classes while allowing other features to flexibly adapt to new knowledge.

[0086] The technical solution of this embodiment proposes a Class-Proportional Memory strategy (CPM). This strategy makes the class distribution in the replay sample set consistent with the class distribution of historical training data, thus achieving a real balance when recalling past classes and enhancing the utility of the replay sample set.

[0087] The technical solution of this embodiment aims to solve the key problems in Continuous Panoramic Segmentation (CPS) and has significant advantages compared with the prior art. Existing state-of-the-art techniques mainly prevent catastrophic forgetting through knowledge distillation for the entire feature or output. However, these methods may limit the learning ability for new classes while ensuring the memory of old classes. The present invention proposes a past-class backward distillation method, which selectively distills features related to past classes while allowing other features to freely adapt to new knowledge, thereby achieving a balance between knowledge stability and adaptability. In addition, the present invention introduces a class-proportional memory strategy, which selects replay samples according to the cumulative class distribution of the historical training set, making the class distribution in the replay sample set more accurately reflect the true class distribution, enhancing the memory ability for past knowledge and the consistency of classification tendency. Experimental results on the ADE20K dataset show that the performance of the continuous panoramic segmentation framework BCPS proposed by the present invention is significantly better than that of existing state-of-the-art techniques. This not only verifies the effectiveness of each innovative mechanism but also demonstrates the practicality and superiority of the overall framework.

[0088] For the schematic diagram of the visualization results of continuous panoramic segmentation in the embodiment of the present invention, please refer to the appendix Figure 3 as shown, where (a) the input image; (b) the prediction result of BCPS; (c) the ground truth label.

[0089] The technical solution of this embodiment introduces the Past-Class Backward Distillation method (PCBD): by selectively distilling features related to old classes, while maintaining the stability of old knowledge, allowing other features to flexibly adapt to new knowledge, thereby achieving a balance between stability and adaptability.

[0090] The technical solution of this embodiment proposes a class-proportional memory strategy (CPM): constructing a replay sample set based on the class distribution of historical training data to truly reflect the balance of the class distribution, enhancing the practicality of the replay sample set in recalling old class knowledge, and ensuring the stability of the classification tendency.

[0091] Example Two

[0092] The purpose of this embodiment is to provide a computer device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the steps of the above method are implemented.

[0093] Example Three

[0094] The purpose of this embodiment is to provide a computer-readable storage medium.

[0095] A computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, the steps of the above method are executed.

[0096] Example Four

[0097] The purpose of this embodiment is to provide a balanced continuous panoramic segmentation system, including:

[0098] A sample acquisition module, configured to: acquire training samples and replay samples;

[0099] A parallel operation processing module, configured to: perform parallel operation processing on the acquired training samples and replay samples respectively using a first model and a second model;

[0100] Among them, the first model outputs the prediction results to a first pixel decoder and a first Transformer decoder respectively, extracts features from the output of the first Transformer decoder to obtain a first feature, and superimposes the first feature and the prediction result of the first pixel decoder as the pseudo-label of the past class of the training sample to supplement the annotation in the training sample;

[0101] Extract features from the first feature again to obtain a fourth feature;

[0102] The second model outputs the prediction results to a second pixel decoder and a second Transformer decoder respectively, extracts features from the output of the second Transformer decoder to obtain a second feature, and uses the second feature and the prediction result of the second pixel decoder as the pseudo-label of the past class of the training sample to supplement the annotation in the training sample;

[0103] The backtracking module is configured to: after supplementing the annotations in the training samples, obtain the assigned image after the label assignment of the output chunks of the training samples, and perform backtracking processing on the assigned image and the second feature output by the second Transformer decoder to obtain the third feature;

[0104] The classification output module is configured to: obtain the classification output result of the image based on the third feature and the fourth feature.

[0105] Embodiment 5

[0106] The purpose of this embodiment is to provide a computer program product containing instructions, which, when running on a computer, enables the computer to execute the methods and functions involved in any one of the above embodiments.

[0107] Each step involved in the device of the above embodiments corresponds to the first method embodiment. For specific implementation manners, reference may be made to the relevant description part of the first embodiment. The term "computer-readable storage medium" should be understood to include a single medium or multiple media containing one or more instruction sets; it should also be understood to include any medium that can store, encode, or carry an instruction set for execution by a processor and enable the processor to execute any method in the present invention.

[0108] Those skilled in the art should understand that the above-mentioned modules or steps of the present invention can be implemented by a general-purpose computer device. Optionally, they can be implemented by program codes executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be separately fabricated into individual integrated circuit modules, or multiple modules or steps among them can be fabricated into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0109] Although the specific implementation manners of the present invention have been described above in conjunction with the accompanying drawings, it is not a limitation to the protection scope of the present invention. Those skilled in the art should understand that based on the technical solutions of the present invention, various modifications or deformations that can be made without creative efforts by those skilled in the art are still within the protection scope of the present invention.

Claims

1. A balanced continuous panoramic segmentation method, characterized in that Including: Obtain training samples and replay samples; Use the first model and the second model to perform parallel operation processing on the obtained training samples and replay samples respectively; Among them, the first model outputs the prediction results to the first pixel decoder and the first Transformer decoder respectively, extracts features from the output of the first Transformer decoder to obtain the first feature, and the first feature is superimposed with the prediction result of the first pixel decoder as the pseudo-label of the past category of the training sample to supplement the annotation in the training sample; Extract features from the first feature again to obtain the fourth feature; The second model outputs the prediction results to the second pixel decoder and the second Transformer decoder respectively, extracts features from the output of the second Transformer decoder to obtain the second feature, and the second feature and the prediction result of the second pixel decoder are used as the pseudo-label of the past category of the training sample to supplement the annotation in the training sample; After supplementing the annotation in the training sample, after the label of the output chunk of the training sample is assigned, the assigned image is obtained, and the assigned image and the second feature output by the second Transformer decoder are backtracked to obtain the third feature; Obtain the classification output result of the image based on the third feature and the fourth feature.

2. The balanced continuous panoramic segmentation method according to claim 1, wherein The training sample includes multiple image-label pairs, and the label of each image contains multiple ground truth chunks, and the replay sample is a replay sample retained by the class-proportional memory strategy.

3. The balanced continuous panoramic segmentation method according to claim 1, characterized in that, The output for each input image in the training sample consists of a set of N q chunks, denoted as where c i represents the classification prediction, m i represents the mask prediction, and N q is the total number of query vectors and also the number of output chunks.

4. A balanced continuous panoramic segmentation method according to claim 1, characterized in that, Obtain the distillation loss based on the third feature and the fourth feature Here, denotes the indicator function, ε past refers to the set of output chunks assigned with past class labels, and MSE denotes the mean squared error, denotes the Transformer features of the previous model M t-1 c i denotes the classification prediction, m i denotes the masked prediction.

5. A balanced continuous panoramic segmentation method according to claim 2, characterized in that The specific process of obtaining the replay sample by the class-proportional memory strategy is as follows: In the first training step, calculate the number of occurrences of the true chunks of each current category in the training sample. These counts enable the calculation of the class distribution. Using this distribution, construct a sample set containing the required number of samples.

6. A balanced continuous panoramic segmentation method as claimed in claim 5, characterized in that, In constructing a sample set containing the required number of samples, a greedy algorithm needs to be used to obtain a usable local solution; specifically, the greedy algorithm iterates the required number of times over all the images in the training sample, and each time selects an image to minimize the difference between the changing class distribution and the calculated class distribution; In each subsequent step, calculate the number of occurrences of the current category and update the cumulative class distribution. Using this distribution, update the replay sample set.

7. A balanced continuous panoramic segmentation system, characterized in that, Including: A sample acquisition module, configured to: obtain training samples and replay samples; A parallel operation processing module, configured to: use the first model and the second model to perform parallel operation processing on the obtained training samples and replay samples respectively; Among them, the first model outputs the prediction results to the first pixel decoder and the first Transformer decoder respectively, extracts features from the output of the first Transformer decoder to obtain the first feature, and the first feature is superimposed with the prediction result of the first pixel decoder as the pseudo-label of the past category of the training sample to supplement the annotation in the training sample; Extract features from the first feature again to obtain the fourth feature; The second model outputs the prediction results to a second pixel decoder and a second Transformer decoder respectively, extracts features from the output of the second Transformer decoder to obtain second features, and uses the second features and the prediction results of the second pixel decoder as pseudo-labels of the past categories of the training samples to supplement the annotations in the training samples; A backtracking module, configured to: after supplementing the annotations in the training samples, obtain an allocated image after label assignment of the output chunks of the training samples, and perform backtracking processing on the allocated image and the second features output by the second Transformer decoder to obtain third features; A classification output module, configured to: obtain a classification output result of the image based on the third features and the fourth features.

8. A computer program product comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method according to any one of claims 1 to 6.

9. A computer device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method according to any one of the above claims 1-6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by a processor, it executes the steps of the method according to any one of the above claims 1-6.

Images