Continuous semantic segmentation method and system based on balanced multi-granularity fusion feature distillation

Abstract

The invention discloses a continuous semantic segmentation method and system based on balanced multi-granularity fusion feature distillation, and relates to the technical field of continuous semantic segmentation, and the method comprises the steps: obtaining a picture, inputting the picture into a feature extraction network of a pre-established new and old model, and outputting the multi-layer features of the picture; inputting the multi-layer features of the picture into a pre-established parallel self-attention module, and outputting to obtain preliminary fusion multi-granularity features; inputting the preliminary fusion multi-granularity feature into a pre-established moment-based channel attention module, and outputting to obtain a balanced multi-granularity fusion feature; calculating the consistency constraint of the balanced multi-granularity fusion features based on knowledge distillation, and calculating the total training loss based on the consistency constraint of the balanced multi-granularity fusion features; and training the new model based on the training total loss to obtain a trained new model, providing an optimal initialization decision boundary for the new task based on the auxiliary classifier through the trained new model, and improving the accuracy of the continuous semantic segmentation task.

Description

Technical Field

[0001] The present invention relates to the technical field of continuous semantic segmentation, and specifically to a continuous semantic segmentation method and system based on balanced multi-granularity fusion feature distillation. Background Art

[0002] With the rapid development of convolutional neural networks, the accuracy of semantic segmentation tasks has been greatly improved. However, most current semantic segmentation methods are based on static batch big data for training, while in practical applications, the data that various industries need to process are mostly dynamic, open, and unevenly distributed data streams. The model obtained by this batch learning training can only recognize categories that have appeared in the training set. For categories that do not appear in the training set, the model can only be retrained using a new training set containing new categories, which not only wastes a lot of time and material resources, but also causes catastrophic forgetting, that is, the performance of the model on previously learned old categories is significantly reduced. In order to solve this problem, researchers began to introduce the continuous learning paradigm into the semantic segmentation task, trying to enable the model to continuously use the newly arrived data to learn new information, learn to handle new tasks, and maintain the ability to handle old tasks.

[0003] Recently, many continuous semantic segmentation methods have tried to use knowledge distillation technology to constrain the consistency of deep features or output confidence extracted by new and old models, and randomly initialize the classifiers of new categories. However, these methods have two shortcomings: first, knowledge distillation only constrains deep semantic features of a single granularity, ignoring the migration of information contained in other granular features that is crucial to the semantic segmentation task; second, when faced with non-discriminative features between new and old categories at the beginning of a new task, randomly initialized classifiers will produce serious prediction biases. Therefore, the key to the continuous semantic segmentation task is how to retain as much knowledge as possible that plays a key role in preventing catastrophic forgetting of the model, and how to eliminate the prediction bias caused by non-discriminative features at the beginning of a new task to enhance the learning of the new task. Summary of the invention

[0004] In order to solve the deficiencies mentioned in the above background technology, the purpose of the present invention is to provide a continuous semantic segmentation method and system based on balanced multi-granularity fusion feature distillation.

[0005] In a first aspect, the purpose of the present invention can be achieved by the following technical solution: a continuous semantic segmentation method based on balanced multi-granularity fusion feature distillation, the method comprising the following steps: Get the image, input the image into the pre-established feature extraction network of the old and new models, and output the multi-layer features of the image; input the multi-layer features of the image into the pre-established parallel self-attention module, and output the preliminary fused multi-granularity features; The preliminary fused multi-granularity features are input into the pre-established moment-based channel attention module, and the balanced multi-granularity fusion features are output; the consistency constraints of the balanced multi-granularity fusion features are calculated based on knowledge distillation, and the total training loss is calculated based on the consistency constraints of the balanced multi-granularity fusion features; The new model is trained based on the total training loss to obtain a new model after training. The new model after training provides the optimal initialization decision boundary for the new task based on the auxiliary classifier, wherein the new model is the task stage Model.

[0006] In combination with the first aspect, in some implementations of the first aspect, the method further includes: the output process of the preliminary fusion of multi-granularity features: The parallel self-attention module consists of several parallel branches, each of which follows the self-attention network structure in the visual transformer; Convert the intermediate features of different layers of the model to the same size using bilinear interpolation; The features of different layers after bilinear interpolation are input into different branches of the parallel self-attention module respectively, and each branch corresponds to a layer of features; The outputs of each branch of the parallel self-attention module are spliced ​​in the channel dimension to form a preliminary fused multi-granularity feature.

[0007] In combination with the first aspect, in some implementations of the first aspect, the method further includes: an output process of the balanced multi-granularity fusion feature: Based on the preliminary fusion of multi-granular features, the moment statistics are calculated in the spatial dimension, including the first-order moment: mean, the second-order moment: variance and the third-order moment: skewness. Three feature statistical vectors are obtained corresponding to the three statistics. Input the three feature statistical vectors into the multi-layer perceptron network and output three channel attention vectors respectively; The three channel attention vectors are added together and multiplied with the preliminary fused multi-granularity features to obtain the balanced multi-granularity fusion features.

[0008] In combination with the first aspect, in some implementations of the first aspect, the method further includes: a calculation process of the total training loss: Based on the knowledge distillation technology, the distance between the balanced multi-granularity fusion features of the new and old models is calculated as the training loss, and the consistency constraint of the balanced multi-granularity fusion features is calculated. ; Calculate the standard classification loss , and the confidence distillation loss for the final output of the model ; Calculate the total loss of model training : in, and It is the balance coefficient used to balance various losses.

[0009] In combination with the first aspect, in some implementations of the first aspect, the method further includes: The calculation process is as follows: In the formula, and Represents the spatial dimension of balanced multi-granularity fusion features, Represents balanced multi-granularity fusion features In the spatial dimension Line The value of the column, Subscript Indicates that the balanced multi-granularity fusion feature is composed of the task stage Model extraction.

[0010] The standard classification loss for: in, represents the category label, and represents the size of the output mask, Indicates the task phase The model is based on the pixels in the image Corresponding category The confidence level of the output, represents taking the logarithm, Represents an indicator function that outputs 1 if its argument is true, otherwise it outputs 0. represents the segmentation confidence of the model output, is the pixel in the output mask, Represents pixels The real label, and are the sizes of the output mask in the spatial dimension, Indicates the task phase The set of new categories learned by the model, Indicates the task stage The number of all categories learned by the model; The confidence distillation loss for: .

[0011] in, Represents all the old categories that the model has learned.

[0012] In combination with the first aspect, in some implementations of the first aspect, the method further includes: the process of providing an optimal initialization decision boundary for a new task based on an auxiliary classifier using the trained new model is as follows: Classification loss variants Train an auxiliary classifier for each class: in, represents the category label, and represents the size of the output mask, Indicates the task phase The model is based on the pixels in the image Corresponding category The confidence level of the output, represents taking the logarithm, Represents an indicator function, which outputs 1 if the argument is true, and 0 otherwise. represents the segmentation confidence of the model output, is the pixel in the output mask, Represents pixels The real label. and are the sizes of the output mask in the spatial dimension, Indicates the task phase The set of new categories learned by the model, Indicates the task stage The number of all categories learned by the model; Calculate and store feature prototypes for each category based on the model at the completion of each task stage training; At the beginning of a new task, the feature prototype of each new class is calculated based on the new model, and the old class with the highest similarity between the prototype and the new class is matched, and the new class classifier is initialized with the auxiliary classifier of the old class with the highest similarity.

[0013] In combination with the first aspect, in some implementations of the first aspect, the method further includes: the process of calculating and storing the feature prototype of each category is as follows: For each category, a binary mask is generated for each image based on the annotations in the dataset, where pixels belonging to this category are marked as 1 in the binary mask, and pixels that do not belong to this category are marked as 0 in the binary mask; Based on the model trained at each task stage, features are extracted for each category of images, the extracted feature map is multiplied by the binary mask of the corresponding image, and the category-specific feature embedding of each image is obtained through global average pooling; Calculate the mean of the category-specific feature embeddings of all images in each class, obtain the feature prototype of the category, and store it.

[0014] In combination with the first aspect, in some implementations of the first aspect, the method further includes: the process of initializing the new class classifier with the old class auxiliary classifier with the highest similarity is as follows: At the beginning of each task stage, the feature extractor of the new model is initialized with the feature extractor trained in the previous task; For each new category, a binary mask is generated for each image based on the annotations in the dataset, where pixels belonging to the category are marked as 1 in the binary mask, and pixels not belonging to the category are marked as 0 in the binary mask; Based on the initialized new model feature extractor, features are extracted for each category of images. The extracted feature map is multiplied with the binary mask of the corresponding image, and the category-specific feature embedding of each image is obtained through global average pooling. Calculate the mean of the category-specific feature embeddings of all images of each new category to obtain the feature prototype of the new category, and calculate the similarity with the stored feature prototypes of all old categories; For each new class, select the old class with the highest similarity to its feature prototype, and use the auxiliary classifier of this old class to initialize the new class classifier.

[0015] Secondly, in order to achieve the above objectives, a continuous semantic segmentation system based on balanced multi-granularity fusion feature distillation includes: The feature processing module is used to obtain the image, input the image into the pre-established feature extraction network of the new and old models, and output the multi-layer features of the image; input the multi-layer features of the image into the pre-established parallel self-attention module, and output the preliminary fused multi-granularity features; A loss calculation module is used to input the preliminary fused multi-granularity features into a pre-established moment-based channel attention module, and output balanced multi-granularity fused features; calculate the consistency constraints of the balanced multi-granularity fused features based on knowledge distillation, and calculate the total training loss based on the consistency constraints of the balanced multi-granularity fused features; The model training module is used to train the new model based on the total training loss to obtain the trained new model, and provide the optimal initialization decision boundary for the new task based on the auxiliary classifier through the trained new model, wherein the new model is the task stage Model.

[0016] In another aspect of the present invention, in order to achieve the above-mentioned purpose, a terminal device is disclosed, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein the memory stores a computer program capable of running on the processor, and when the processor loads and executes the computer program, the continuous semantic segmentation method based on balanced multi-granularity fusion feature distillation as described above is adopted.

[0017] Beneficial effects of the present invention: The present invention can well solve the common problems in existing continuous semantic segmentation methods, that is, only constraining the deep semantic features of a single granularity, ignoring the migration of information contained in other granular features that is crucial to the semantic segmentation task, and the randomly initialized classifier will produce serious prediction bias. It can also reduce the material and time costs required to retrain the semantic segmentation model when facing a continuous incoming data stream. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative work. Figure 1 It is a schematic flow chart of the method of the present invention; Figure 2 It is a structural schematic diagram of the continuous semantic segmentation method of the present invention; Figure 3 It is a schematic diagram of a continuous semantic segmentation verification case of the present invention; Figure 4 It is a schematic diagram of the system structure of the present invention. DETAILED DESCRIPTION

[0019] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] Embodiment 1: like Figure 1 As shown, the continuous semantic segmentation method based on balanced multi-granularity fusion feature distillation includes the following steps: S101: Obtain an image, input the image into the pre-established feature extraction network of the new and old models, and output the multi-layer features of the image; input the multi-layer features of the image into the pre-established parallel self-attention module, and output the preliminary fused multi-granularity features; In the task phase , given a training image and an old model trained in the previous task phase, the new model is initialized from the old model, and the old model remains frozen during the training process; During the training process, the intermediate features from each layer of the new and old model feature extraction network (such as ResNet50 or ResNet101) are selected. feature maps from a specific layer, represented as and ,in , Indicates that among all the selected intermediate features, and The network layers from shallow to deep are ranked Bit, , and Respectively represent characteristics Figure 3 The size of the dimension.

[0021] Shallow feature maps contain fine-grained local detail information, such as color, geometry, etc., while deep feature maps contain high-level semantic information. This information is important for pixel-level classification tasks such as semantic segmentation. Features of various granularities are extracted to balance and fuse multi-granular features in the future, and knowledge distillation is proposed to preserve them.

[0022] The output process of preliminary fusion of multi-granular features: The parallel self-attention module consists of several parallel branches, each of which follows the self-attention network structure in the visual transformer; Convert the intermediate features of different layers of the model to the same size using bilinear interpolation; For all the extracted intermediate features, bilinear interpolation is uniformly performed in the spatial dimension, and its spatial size becomes , so that the features of each layer are concatenated in the channel dimension after outputting the parallel self-attention module; The features of different layers after bilinear interpolation are input into different branches of the parallel self-attention module respectively, and each branch corresponds to a layer of features; The outputs of each branch of the parallel self-attention module are spliced ​​in the channel dimension to form a preliminary fused multi-granularity feature.

[0023] The new model initially integrates multi-granular features For example, its size is Similarly, the old model produces preliminary fusion multi-granular features in the same way .

[0024] S102: inputting the preliminary fused multi-granularity features into a pre-established moment-based channel attention module, and outputting balanced multi-granularity fused features; calculating the consistency constraints of the balanced multi-granularity fused features based on knowledge distillation, and calculating the total training loss based on the consistency constraints of the balanced multi-granularity fused features; The output process of balanced multi-granularity fusion features: Based on the preliminary fusion of multi-granular features, the moment statistics are calculated in the spatial dimension, including the first-order moment: mean, the second-order moment: variance and the third-order moment: skewness. Three feature statistical vectors are obtained corresponding to the three statistics. To initially integrate multi-granular features Take it as an example, and perform moment operation on each channel. on the channel Moment : in, . and Indicates the initial fusion of multi-granularity features The spatial dimensions of Indicates the spatial coordinates of the pixel. Moment, preliminary fusion of multi-granularity features Get a moment vector . Corresponding to the first-order moment, the second-order moment and the third-order moment, a total of three moment vectors are obtained.

[0025] Input the three feature statistical vectors into the multi-layer perceptron network and output three channel attention vectors respectively; Add the three channel attention vectors and multiply them with the preliminary fused multi-granularity feature to obtain the balanced multi-granularity fusion feature and .

[0026] The calculation process of the total training loss is: Based on the knowledge distillation technology, the distance between the balanced multi-granularity fusion features of the new and old models is calculated as the training loss, and the consistency constraint of the balanced multi-granularity fusion features is calculated. ; Calculate the standard classification loss , and the confidence distillation loss for the final output of the model ; Calculate the total loss of model training : in, and It is the balance coefficient used to balance various losses.

[0027] The balanced multi-granularity fusion feature consistency constraint The calculation process is as follows: In the formula, and Represents the spatial dimension of balanced multi-granularity fusion features, Represents balanced multi-granularity fusion features In the spatial dimension Line The value of the column, Subscript Indicates that the balanced multi-granularity fusion feature is composed of the task stage Model extraction.

[0028] The standard classification loss for: in, represents the category label, and represents the size of the output mask, Indicates the task phase The model is based on the pixels in the image Corresponding category The confidence level of the output, represents taking the logarithm, Represents an indicator function that outputs 1 if its argument is true, otherwise it outputs 0. represents the segmentation confidence of the model output, is the pixel in the output mask, Represents pixels The real label, and are the sizes of the output mask in the spatial dimension, Indicates the task phase The set of new categories learned by the model, Indicates the task stage The number of all categories learned by the model; The confidence distillation loss for: .

[0029] in, Represents all the old categories that the model has learned.

[0030] S103: Train the new model based on the total training loss to obtain a trained new model, and use the trained new model to provide an optimal initialization decision boundary for the new task based on the auxiliary classifier, wherein the new model is the task stage Model.

[0031] The process of providing the optimal initialization decision boundary for the new task based on the auxiliary classifier by the trained new model is as follows: Classification loss variants Train an auxiliary classifier for each class: in, represents the category label, and represents the size of the output mask, Indicates the task phase The model is based on the pixels in the image Corresponding category The confidence level of the output, represents taking the logarithm, Represents an indicator function, which outputs 1 if the argument is true, and 0 otherwise. represents the segmentation confidence of the model output, is the pixel in the output mask, Represents pixels The real label. and are the sizes of the output mask in the spatial dimension, Indicates the task phase The set of new categories learned by the model, Indicates the task stage The number of all categories learned by the model; Calculate and store feature prototypes for each category based on the model at the completion of each task stage training; At the beginning of a new task, the feature prototype of each new class is calculated based on the new model, and the old class with the highest similarity between the prototype and the new class is matched, and the new class classifier is initialized with the auxiliary classifier of the old class with the highest similarity.

[0032] The process of calculating and storing the feature prototype of each category is as follows: For each category, a binary mask is generated for each image based on the annotations in the dataset, where pixels belonging to this category are marked as 1 in the binary mask, and pixels that do not belong to this category are marked as 0 in the binary mask; Based on the model trained at each task stage, features are extracted for each category of images, the extracted feature map is multiplied by the binary mask of the corresponding image, and the category-specific feature embedding of each image is obtained through global average pooling; Calculate the mean of the category-specific feature embeddings of all images in each class, obtain the feature prototype of the category, and store it.

[0033] The process of initializing the new class classifier with the old class auxiliary classifier with the highest similarity is as follows: At the beginning of each task stage, the feature extractor of the new model is initialized with the feature extractor trained in the previous task; For each new category, a binary mask is generated for each image based on the annotations in the dataset, where pixels belonging to the category are marked as 1 in the binary mask, and pixels not belonging to the category are marked as 0 in the binary mask; Based on the initialized new model feature extractor, features are extracted for each category of images. The extracted feature map is multiplied with the binary mask of the corresponding image, and the category-specific feature embedding of each image is obtained through global average pooling. Calculate the mean of the category-specific feature embeddings of all images of each new category to obtain the feature prototype of the new category, and calculate the similarity with the stored feature prototypes of all old categories; For each new class, select the old class with the highest similarity to its feature prototype, and use the auxiliary classifier of this old class to initialize the new class classifier.

[0034] Figure 2 The block diagram of the continuous semantic segmentation method and system is shown. In the task t stage, the new model and the old model of the task t-1 stage and the training pictures are given. The training pictures are respectively input into the feature extractors of the new and old models to obtain a set of multi-granular features. Then, the multi-granular features of the new and old models are respectively input into the corresponding parallel self-attention modules according to the number of layers they are in. Through multiple sets of self-attention layers, the task-related areas are mined for the features of different granularities of the new and old models, and then they are spliced ​​in the channel dimension to obtain the preliminary fused multi-granular features of the new and old models. Then, the preliminary fused multi-granular features of the new and old models are respectively input into the moment-based channel attention module, and the moment statistics of the preliminary fused multi-granular features are calculated in the spatial dimension to obtain the moment vector of the preliminary fused multi-granular features. Then, the corresponding moment-based channel attention vector is obtained through an MLP network. The vector is summed and multiplied with the preliminary fused multi-granular features to obtain the balanced multi-granular fusion features of the new and old models. Finally, the balanced multi-granularity fusion features of the new and old models are input into the classifiers of the new and old models respectively, and the confidence corresponding to each category is output. The classification loss is calculated using the binary cross entropy loss based on the confidence and the true label of the training data. , calculate the feature distillation loss based on the balanced multi-granularity fusion features of the new and old models . Calculate the confidence distillation loss based on the output confidence of the old model and the output confidence of the new model . Based on the above three losses, the new model is optimized using the gradient descent algorithm. Based on a classification loss variant The auxiliary classifier is optimized using the gradient descent algorithm and used to initialize the new class classifier at the beginning of the next task stage.

[0035] Figure 3 This is a continuous semantic segmentation verification case for this method and system. The figure shows the segmentation effect of this method and system in various common scenarios and the comparison with other methods in the same field. The Image column represents the input image, the GT column represents the ideal segmentation result of the input image, and the D 2 UP is the segmentation effect of this method and system, PLOP and EDL are the segmentation effects of other two methods and systems in this field. In the figure, "person", "bottle" and "chair" are old categories learned before, and "sofa", "potted plant" and "train" are new categories. Figure 3 The results show that no matter it is an old class or a new class, the effect of this method and system is better than other methods.

[0036] Embodiment 2: In the second aspect, as Figure 4 As shown, in order to achieve the above purpose, a continuous semantic segmentation system based on balanced multi-granularity fusion feature distillation includes: The feature processing module 11 is used to obtain a picture, input the picture into the pre-established feature extraction network of the new and old models, and output the multi-layer features of the picture; input the multi-layer features of the picture into the pre-established parallel self-attention module, and output the preliminary fused multi-granularity features; The loss calculation module 12 is used to input the preliminary fused multi-granularity features into the pre-established moment-based channel attention module, and output the balanced multi-granularity fused features; calculate the consistency constraints of the balanced multi-granularity fused features based on knowledge distillation, and calculate the total training loss based on the consistency constraints of the balanced multi-granularity fused features; The model training module 13 is used to train the new model based on the total training loss to obtain a new model after training, and provide an optimal initialization decision boundary for the new task based on the auxiliary classifier through the new model after training, wherein the new model is the task stage Model.

[0037] Based on the same inventive concept, the present invention also provides a computer device, which includes: one or more processors, and a memory for storing one or more computer programs; the program includes program instructions, and the processor is used to execute the program instructions stored in the memory. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is used to implement one or more instructions, specifically for loading and executing one or more instructions in a computer storage medium to implement the above method.

[0038] It should be further explained that, based on the same inventive concept, the present invention also provides a computer storage medium, on which a computer program is stored, and the computer program is executed by the processor to execute the above method. The storage medium can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electrical, magnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0039] In the description of this specification, the description with reference to the terms "one embodiment", "example", "specific example", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0040] The above shows and describes the basic principles, main features and advantages of the present disclosure. Those skilled in the art should understand that the present disclosure is not limited by the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present disclosure. Without departing from the spirit and scope of the present disclosure, the present disclosure may have various changes and improvements, and these changes and improvements fall within the scope of the present disclosure to be protected.

Claims

1. A continuous semantic segmentation method based on balanced multi-granularity fusion feature distillation, characterized by: The method comprises the following steps: Get the image, input the image into the pre-established feature extraction network of the new and old models, and output the multi-layer features of the image; The multi-layer features of the image are input into the pre-established parallel self-attention module, and the output is the preliminary fusion of multi-granular features; The preliminary fused multi-granularity features are input into the pre-established moment-based channel attention module, and the balanced multi-granularity fused features are output; Based on knowledge distillation, the consistency constraints of balanced multi-granularity fusion features are calculated, and the total training loss is calculated based on the consistency constraints of balanced multi-granularity fusion features; The new model is trained based on the total training loss to obtain a new model after training. The new model after training provides the optimal initialization decision boundary for the new task based on the auxiliary classifier, wherein the new model is the task stage Model.

2. The continuous semantic segmentation method based on balanced multi-granularity fusion feature distillation according to claim 1 is characterized in that: The output process of the preliminary fusion of multi-granularity features: The parallel self-attention module consists of several parallel branches, each of which follows the self-attention network structure in the visual transformer; Convert the intermediate features of different layers of the model to the same size using bilinear interpolation; The features of different layers after bilinear interpolation are input into different branches of the parallel self-attention module, and each branch corresponds to a layer of features; The outputs of each branch of the parallel self-attention module are spliced ​​in the channel dimension to form a preliminary fused multi-granularity feature.

3. The continuous semantic segmentation method based on balanced multi-granularity fusion feature distillation according to claim 1 is characterized in that: The output process of the balanced multi-granularity fusion feature: Based on the preliminary fusion of multi-granular features, the moment statistics are calculated in the spatial dimension, including the first-order moment: mean, the second-order moment: variance and the third-order moment: skewness. Three feature statistical vectors are obtained corresponding to the three statistics. Input the three feature statistical vectors into the multi-layer perceptron network and output three channel attention vectors respectively; The three channel attention vectors are added together and multiplied with the preliminary fused multi-granularity features to obtain the balanced multi-granularity fusion features.

4. The continuous semantic segmentation method based on balanced multi-granularity fusion feature distillation according to claim 1 is characterized in that: The calculation process of the total training loss is: Based on the knowledge distillation technology, the distance between the balanced multi-granularity fusion features of the new model and the old model is calculated as the training loss, and the consistency constraint of the balanced multi-granularity fusion feature is calculated. ; Calculate the standard classification loss , and for the task phase Confidence distillation loss of the final output of the model ; Computational task phase The total loss of model training : in, and It is the balance coefficient used to balance various losses.

5. The continuous semantic segmentation method based on balanced multi-granularity fusion feature distillation according to claim 4 is characterized in that: The balanced multi-granularity fusion feature consistency constraint The calculation process is as follows: In the formula, and Represents the spatial dimension of balanced multi-granularity fusion features, Represents balanced multi-granularity fusion features In the spatial dimension Line The value of the column, Subscript Indicates that the balanced multi-granularity fusion feature is composed of the task stage Model extraction; The standard classification loss for: in, represents the category label, and represents the size of the output mask, Indicates the task phase The model is based on the pixels in the image Corresponding category The confidence level of the output, represents taking the logarithm, Represents an indicator function that outputs 1 if its argument is true, otherwise it outputs 0. represents the segmentation confidence of the model output, is the pixel in the output mask, Represents pixels The real label, and are the sizes of the output mask in the spatial dimension, Indicates the task phase The set of new categories learned by the model, Indicates the task stage The number of all categories learned by the model; The confidence distillation loss for: in, Represents all the old categories that the model has learned.

6. The continuous semantic segmentation method based on balanced multi-granularity fusion feature distillation according to claim 1, characterized in that: The process of providing the optimal initialization decision boundary for the new task based on the auxiliary classifier by the trained new model is as follows: Classification loss variants Train an auxiliary classifier for each class: in, represents the category label, and represents the size of the output mask, Indicates the task phase The model is based on the pixels in the image Corresponding category The confidence level of the output, represents taking the logarithm, Represents an indicator function, which outputs 1 if the argument is true, and 0 otherwise. represents the segmentation confidence of the model output, is the pixel in the output mask, Represents pixels The true label of and are the sizes of the output mask in the spatial dimension, Indicates the task phase The set of new categories learned by the model, Indicates the task stage The number of all categories learned by the model; Calculate and store feature prototypes for each category based on the model at the completion of each task stage training; At the beginning of a new task, the feature prototype of each new class is calculated based on the new model, and the old class with the highest similarity between the prototype and the new class is matched, and the new class classifier is initialized with the auxiliary classifier of the old class with the highest similarity.

7. The method for continuous semantic segmentation based on balanced multi-granularity fusion feature distillation according to claim 6, characterized in that: The process of calculating and storing the feature prototype of each category is as follows: For each category, a binary mask is generated for each image based on the annotations in the dataset, where pixels belonging to this category are marked as 1 in the binary mask, and pixels that do not belong to this category are marked as 0 in the binary mask; Based on the model trained at each task stage, features are extracted for each category of images, the extracted feature map is multiplied by the binary mask of the corresponding image, and the category-specific feature embedding of each image is obtained through global average pooling; Calculate the mean of the category-specific feature embeddings of all images in each class, obtain the feature prototype of the category, and store it.

8. The method for continuous semantic segmentation based on balanced multi-granularity fusion feature distillation according to claim 7, characterized in that: The process of initializing the new class classifier with the old class auxiliary classifier with the highest similarity is as follows: At the beginning of each task stage, the feature extractor of the new model is initialized with the feature extractor trained in the previous task; For each new category, a binary mask is generated for each image based on the annotations in the dataset, where pixels belonging to the category are marked as 1 in the binary mask, and pixels not belonging to the category are marked as 0 in the binary mask; Based on the initialized new model feature extractor, features are extracted for each category of images. The extracted feature map is multiplied with the binary mask of the corresponding image, and the category-specific feature embedding of each image is obtained through global average pooling. Calculate the mean of the category-specific feature embeddings of all images of each new category to obtain the feature prototype of the new category, and calculate the similarity with the stored feature prototypes of all old categories; For each new class, select the old class with the highest similarity to its feature prototype, and use the auxiliary classifier of this old class to initialize the new class classifier.

9. A continuous semantic segmentation system based on balanced multi-granularity fusion feature distillation, which adopts the continuous semantic segmentation method based on balanced multi-granularity fusion feature distillation according to any one of claims 1 to 8, characterized in that: include: The feature processing module is used to obtain images, input the images into the pre-established feature extraction network of the new and old models, and output the multi-layer features of the images; The multi-layer features of the image are input into the pre-established parallel self-attention module, and the output is the preliminary fusion of multi-granular features; A loss calculation module is used to input the preliminary fused multi-granularity features into a pre-established moment-based channel attention module, and output a balanced multi-granularity fused feature; Based on knowledge distillation, the consistency constraints of balanced multi-granularity fusion features are calculated, and the total training loss is calculated based on the consistency constraints of balanced multi-granularity fusion features; The model training module is used to train the new model based on the total training loss to obtain the trained new model, and provide the optimal initialization decision boundary for the new task based on the auxiliary classifier through the trained new model, wherein the new model is the task stage Model.

10. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that: The memory stores a computer program that can be run on the processor. When the processor loads and executes the computer program, the continuous semantic segmentation method based on balanced multi-granularity fusion feature distillation described in any one of claims 1 to 8 is adopted.

Images