Method and system for limited data continual semantic segmentation based on decoupled knowledge distillation

Abstract

The application discloses a limited data continuous semantic segmentation method based on decoupling knowledge distillation, comprising the following steps: constructing an initial semantic segmentation model, training to obtain a first semantic segmentation model and a second semantic segmentation model; constructing a decoupling knowledge distillation loss function based on the feature similarity obtained by the first semantic segmentation model and the second semantic segmentation model; constructing a semantic consistency loss function based on the first semantic segmentation model and the second semantic segmentation model; constructing a cross-entropy loss function based on the prediction result of the first semantic segmentation model and the prediction result of the second semantic segmentation model; supervising and training the prediction result of the second semantic segmentation model and the fusion label input cross-entropy loss function, jointly inputting the prediction results of the first and second semantic segmentation models into the decoupling knowledge distillation loss function and the semantic consistency loss function for joint training to obtain an optimized semantic segmentation model with updated parameters; and inputting an image to be segmented into the optimal semantic segmentation model to obtain a semantic segmentation result.

Description

Technical Field

[0001] This invention relates to the fields of machine learning and computer vision technology, and more specifically to a method and system for continuous semantic segmentation of finite data based on decoupled knowledge distillation. Background Technology

[0002] Semantic segmentation is a task that assigns a label to each pixel in an image, with each label belonging to a set of predefined object categories, such as people, cars, trees, and sky, to achieve pixel-level classification of image content. Thanks to its dense predictive capabilities, semantic segmentation has applications in areas such as autonomous driving systems using digital images and remote sensing image processing. However, current deep learning-based semantic segmentation models require large-scale datasets for training. In application environments such as spaceborne platforms and embedded systems, storage space is limited, and data is constantly being added and updated. Furthermore, traditional deep learning training methods require the training and test sets to have consistent boundaries, meaning they contain the same categories. When a new category is added, all data from both the old and new categories need to be retrained, placing high demands on storage space and computing resources.

[0003] Continuous learning, also known as incremental learning or lifelong learning, is a typical task in the field of deep learning. The problems of existing continuous semantic segmentation algorithms can be mainly divided into three aspects: 1) catastrophic forgetting of old knowledge during continuous learning; 2) semantic confusion between new and old knowledge; 3) poor model generalization. Among them, catastrophic forgetting refers to the phenomenon that when there is no old category data to participate in model training, the model almost completely forgets the previously trained content after learning new knowledge.

[0004] Therefore, how to solve the problems of catastrophic forgetting of old knowledge, semantic confusion between new and old knowledge, and poor model generalization in continuous semantic segmentation tasks is an urgent problem that needs to be solved by those skilled in the art. Summary of the Invention

[0005] In view of this, the present invention provides a method and system for sustained semantic segmentation of limited data based on decoupled knowledge distillation. Based on human cognitive patterns, the high-dimensional features of an image are decomposed and coupled into semantically invariant features and sample relevance features. A targeted knowledge transfer method is designed to enhance the model's performance on sustained semantic segmentation tasks. This addresses the problems of catastrophic forgetting of old knowledge, semantic confusion between new and old knowledge, and poor model generalization in existing technologies.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] Limited data persistent semantic segmentation methods based on decoupled knowledge distillation include:

[0008] Construct an initial semantic segmentation model, and obtain the first semantic segmentation model through incremental learning training at step t-1. Obtain the second semantic segmentation model through incremental learning training at step t.

[0009] Based on the first semantic segmentation model and the second semantic segmentation model, feature similarity is obtained, and a decoupled knowledge distillation loss function is constructed based on the feature similarity.

[0010] Construct a semantic consistency loss function based on the first semantic segmentation model and the second semantic segmentation model;

[0011] After constructing fused annotations based on pseudo-labels generated from the prediction results of the first semantic segmentation model and the current step supervision information, a cross-entropy loss function is constructed with the prediction results of the second semantic segmentation model.

[0012] The prediction results of the second semantic segmentation model and the fusion annotation are input into the cross-entropy loss function for supervised training. The prediction results of the second semantic segmentation model and the prediction results of the first semantic segmentation model are jointly input into the decoupled knowledge distillation loss function and the semantic consistency loss function for joint training. The parameters of the second semantic segmentation model are updated together to obtain the optimized semantic segmentation model with updated parameters.

[0013] The image to be segmented is input into the optimal semantic segmentation model to obtain the semantic segmentation result.

[0014] Preferably, feature similarity is obtained, specifically including:

[0015] The training set images are input into the first semantic segmentation model M. t-1 and the second semantic segmentation model M t The first feature e is obtained accordingly. t-1 Second feature e t ;

[0016] The feature similarity is calculated based on the first feature and the second feature, and the feature similarity calculation formula is as follows:

[0017]

[0018] in, and They respectively represent the following: and The result after transformation into a one-dimensional tensor The α-th channel represents the second feature. Let S(·) represent the α-th channel of the first feature, S(·) represent the similarity calculation operation, and d(·) represent the distance metric function, where α∈{1,2,…,N} c}, N cIndicates the number of feature channels.

[0019] Preferably, a decoupling knowledge distillation loss function is constructed based on the feature similarity, specifically including:

[0020] The features are sorted according to their similarity scores to obtain the feature similarity ranking result R. ε :

[0021]

[0022] in, Represents the α-th channel feature and The similarity between them, Φ represents the similarity ranking operation based on the feature similarity calculation formula, S represents the operation of channel-dimensional feature correlation calculation, and t represents the t-th incremental training step;

[0023] Based on the feature similarity ranking results, the first feature e is respectively... t-1 Second feature e t Decoupling is performed, and features with similarity greater than or equal to a threshold are classified as semantically invariant features e. SI Features with similarity less than a threshold are classified as sample relevance features e. SS ;

[0024] Based on e SI and e SS Establish corresponding prototype alignment loss functions L SPM The loss function L for preserving features related to samples SFP ;

[0025] According to L SPM and L SFP Constructing a decoupled knowledge distillation loss function L DD :

[0026]

[0027] Where, N l This represents the total number of layers in the neural network, and l represents the current layer of the neural network.

[0028] Preferably, the prototype alignment loss function L SPM Specifically:

[0029]

[0030]

[0031]

[0032] in, This represents the loss function for aligning similar prototypes. C represents the heterogeneous prototype separation loss function. 0:t-1 C represents the number of categories learned in step t-1. 0:t This represents the number of categories learned in step t, where k represents the number of categories. Represents the feature prototype. Let KL represent the new class prototype calculated in the t-th learning step, and KL represent the KL divergence. This represents the feature vector output by the model in the current step.

[0033] Preferably, the new category prototype calculated in the t-th learning step Specifically:

[0034]

[0035]

[0036] Where, |N k | represents the number of images belonging to category k in each iteration step, p bg M represents the mean of all category prototypes. x This represents the segmentation label of image x. Let Ω represent the semantically invariant feature embedding of the output with respect to image x, and let Ω represent the indicator function.

[0037] Preferably, the sample correlation feature preservation loss function L SFP Specifically:

[0038]

[0039]

[0040] Among them, D CL (·) represents the contrastive learning loss of triplet. and These respectively represent anchored class embedding, positive sample class embedding, and negative sample class embedding. Indicates that it comes from M t-1 Regarding the anchored category embedding of the k-th class, and They represent from M t Regarding the positive and negative sample class embeddings of the k-th class, m represents the normality coefficient.

[0041] Preferably, a semantic consistency loss function is constructed, specifically including:

[0042] Calculate the correlation score The result is propagated from layer l to layer (l-1):

[0043]

[0044] Where j and k represent two neurons in adjacent layers of the neural network, This represents the relevance score of neuron j in the (l-1)th layer of the neural network. This represents the relevance score of neuron k in the l-th layer of the neural network. This represents the activation response between neuron j and neuron k in the l-th layer of a neural network;

[0045] The neuron correlation g(x,c) is obtained by summing the results of propagating the correlation scores of all neurons layer by layer:

[0046]

[0047] Among them, R (l) This represents the correlation score of all neurons in the l-th layer of the neural network. Indicates a dimension of N c The feature space;

[0048] Based on the neuron correlation calculation formula g(x,c), the neurons from M are respectively obtained. t-1 Neuron-related g t-1 (x,c) and from M t Neuronal correlation g t (x,c);

[0049] Based on g t-1 (x,c) and g t The semantic consistency loss function L is calculated from (x,c). NSC :

[0050]

[0051] Preferably, the cross-entropy loss function is constructed, specifically including:

[0052] Based on the first semantic segmentation model M t-1 The prediction results generate uncertain pseudo-labels for the old categories, which are then fused with the real labels for the new categories to obtain fused labels.

[0053] According to the second semantic segmentation model M t Based on the prediction results and the fused annotations, the cross-entropy loss function L is obtained. CE :

[0054]

[0055]

[0056] in, M represents the second semantic segmentation model t During training, the fused annotation corresponding to pixel i M represents the second semantic segmentation model t The output is the prediction result for pixel i, F θ Describing a feature extraction network, x i Represents pixel i in the original image.

[0057] The preferred method for generating uncertain pseudo-labels is as follows:

[0058]

[0059]

[0060]

[0061]

[0062] in M represents the first semantic segmentation model t-1 The prediction result of the output pixel i, c u c represents an unknown category. b Let Γ represent the background category, ζ represent the threshold for controlling the confidence of pseudo-labels, and u represent the threshold for measuring the stability of the model's prediction results. i This represents the uncertainty of pixel i. This represents the highest value predicted by the model for the category. Δ represents the second-highest value for the class predicted by the model. i This represents the range of uncertainty for pixel i. This represents the lowest value of the category predicted by the model.

[0063] A finite data continuous semantic segmentation system based on decoupled knowledge distillation includes: a model acquisition module, a first loss function construction module, a second loss function construction module, a third loss function construction module, a model training module, and a result output module;

[0064] The model acquisition module is used to construct an initial semantic segmentation model, obtain a first semantic segmentation model through incremental learning training at step t-1, and obtain a second semantic segmentation model through incremental learning training at step t.

[0065] The first loss function construction module is used to obtain feature similarity based on the first semantic segmentation model and the second semantic segmentation model, and to construct a decoupled knowledge distillation loss function based on the feature similarity.

[0066] The second loss function construction module is used to construct a semantic consistency loss function based on the first semantic segmentation model and the second semantic segmentation model;

[0067] The third loss function construction module is used to construct a fused label based on the pseudo label generated by the prediction result of the first semantic segmentation model and the supervision information of the current step, and then construct a cross-entropy loss function with the prediction result of the second semantic segmentation model.

[0068] The model training module is used to supervise the training of the cross-entropy loss function by inputting the prediction results of the second semantic segmentation model and the fused annotations, and to jointly train the second semantic segmentation model by inputting the prediction results of the first semantic segmentation model and the decoupled knowledge distillation loss function and the semantic consistency loss function together, thereby updating the parameters of the second semantic segmentation model and obtaining an optimized semantic segmentation model with updated parameters.

[0069] The result output module is used to input the image to be segmented into the optimal semantic segmentation model to obtain the semantic segmentation result.

[0070] As can be seen from the above technical solution, compared with the prior art, this invention discloses a method and system for continuous semantic segmentation of limited data based on decoupled knowledge distillation. Based on human cognitive patterns and combined with the superior learning ability of artificial neural networks, it establishes multiple constraints in the high-dimensional feature latent space extracted by the model, including neuron correlation semantic consistency constraints, decoupled knowledge distillation constraints, and semantic segmentation constraints, achieving the inheritance of old knowledge without the need for old data to participate in training. It has the following beneficial effects:

[0071] 1. It does not require storing old data and can achieve incremental learning capabilities for new categories while retaining the classification of known categories. It can achieve more robust continuous learning capabilities under limited incremental data conditions.

[0072] 2. This invention proposes a feature decoupling mechanism based on human cognitive patterns, which decouples the high-dimensional features of an image into semantically invariant features and sample correlation features, and the two satisfy the channel dimension additive relationship, thus solving the problem of catastrophic forgetting of old knowledge during continuous learning.

[0073] 3. This invention proposes a decoupled knowledge distillation method. By decoupling semantic invariant features and sample relevance features to establish knowledge distillation constraints, it can effectively alleviate the knowledge forgetting phenomenon in semantic segmentation models. At the same time, it can alleviate the semantic drift problem during the continuous learning process of the model under limited new data conditions. It can be applied to scenarios such as remote sensing earth observation and autonomous driving systems. For continuously increasing semantic categories and scenarios, it can achieve continuous model updates without significantly increasing storage space and computing resources.

[0074] 4. This invention proposes a semantic consistency constraint based on neuron correlation, which constrains the features output by the old and new models based on Euclidean distance, enabling effective knowledge transfer between the old and new models, enhancing the generalization and robustness of the continuous semantic segmentation model, and solving the problem of poor generalization of continuous learning models under cross-class and cross-modal incremental conditions; embedding neuron correlation constraints into the continuous learning model improves the model's continuous learning performance while enhancing its interpretability.

[0075] 5. This invention proposes a pseudo-label generation method based on uncertainty, which improves the ability of the continuously incremental semantic segmentation model to inherit old knowledge under the condition of limited incremental data, and provides reliable supervision information for the training of new models.

[0076] 6. The continuous learning method proposed in this invention can dynamically adjust the parameters of the model during the learning process to adapt to the input of new data, while retaining previously learned knowledge. It can continuously learn new tasks or categories without forgetting previously learned knowledge, thus achieving continuous learning. By using the continuous learning method, we can better understand and imitate the learning methods of the human brain and the structural mechanism of biological neural networks at the system level, providing a technical foundation for developing new computational models and effective learning algorithms.

[0077] 7. As a theoretical and technical exploration in the field of continuous semantic segmentation, this invention provides a theoretical and technical foundation for the development of strong artificial intelligence theory and the application of artificial intelligence in embedded platforms. At the same time, it provides theoretical support and technical verification for research in the academic fields of artificial intelligence and computer vision and the implementation of related products in the industry. Attached Figure Description

[0078] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0079] Figure 1 The flowchart of the continuous semantic segmentation method provided by the present invention.

[0080] Figure 2 The flowchart of the continuous semantic segmentation method based on decoupled knowledge distillation provided by the present invention is shown.

[0081] Figure 3 This is a schematic diagram of the input image and pixel-level annotations for the training set provided by the present invention.

[0082] Figure 4 A flowchart illustrating the construction of the semantic consistency loss function provided by this invention.

[0083] Figure 5 This is a schematic diagram illustrating the effect of incremental semantic segmentation of categories in a remote sensing Earth observation scenario provided by the present invention.

[0084] Figure 6 This is a schematic diagram of the qualitative comparison results provided in Embodiment 5 of the present invention.

[0085] Figure 7 A schematic diagram of the continuous semantic segmentation system provided by the present invention. Detailed Implementation

[0086] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0087] Example 1

[0088] like Figure 1 As shown, this embodiment of the invention discloses a finite data persistent semantic segmentation method based on decoupled knowledge distillation, including:

[0089] Construct an initial semantic segmentation model, and obtain the first semantic segmentation model through incremental learning training at step t-1. Obtain the second semantic segmentation model through incremental learning training at step t.

[0090] Feature similarity is obtained based on the first semantic segmentation model and the second semantic segmentation model, and a decoupled knowledge distillation loss function is constructed based on feature similarity.

[0091] Construct a semantic consistency loss function based on the first semantic segmentation model and the second semantic segmentation model;

[0092] After constructing fused annotations based on pseudo-labels generated from the prediction results of the first semantic segmentation model and the current step supervision information, a cross-entropy loss function is constructed with the prediction results of the second semantic segmentation model.

[0093] The prediction results of the second semantic segmentation model are trained in a supervised manner with the cross-entropy loss function of the fused labeled input. The prediction results of the second semantic segmentation model and the first semantic segmentation model are jointly trained by inputting the decoupled knowledge distillation loss function and the semantic consistency loss function. The parameters of the second semantic segmentation model are updated together to obtain the optimized semantic segmentation model with updated parameters.

[0094] The image to be segmented is input into the optimal semantic segmentation model to obtain the semantic segmentation result.

[0095] Example 2

[0096] like Figure 2 As shown, this embodiment of the invention discloses a finite data persistent semantic segmentation method based on decoupled knowledge distillation, including:

[0097] An initial semantic segmentation model is constructed. The first semantic segmentation model is obtained by incremental learning and training at step t-1. The second semantic segmentation model is obtained by incremental learning and training at step t.

[0098] Preferably, the training set is set as D = {(x i ,y i )},in Indicates the input image. Indicates the input image x i The corresponding pixel-level annotations are as follows: C represents the number of image channels, H and W represent the height and width of the image in pixels, respectively, and D... t Let D represent the training set for the t-th incremental learning step. In the category incremental semantic segmentation task, when D... t When performing initial model training, {D} cannot be obtained. 0 D 1 ,···,D t-1 The existing category's labeled data, and only the current category D. t The corresponding label category.

[0099] Preferred, such as Figure 3 As shown, the training set of this invention may include autonomous driving street view images and their corresponding pixel-level annotations, and remote sensing images and their corresponding pixel-level annotations.

[0100] Preferably, the training steps for incremental semantic segmentation include: {Step 0 Step 1 Step t-1 Step t Step T}, construct an initial semantic segmentation model and train it using the training set D, in training step Step t-1 The first semantic segmentation model M is obtained at that time. t-1 In the training step Step t The second semantic segmentation model M is obtained at that time. t .

[0101] Feature similarity is obtained based on the first semantic segmentation model and the second semantic segmentation model, and a decoupled knowledge distillation loss function is constructed based on feature similarity.

[0102] Preferably, the training set D is input into the first semantic segmentation model M. t-1Second semantic segmentation model M t M t-1 Parameter freezing, meaning no parameter updates are performed during training, M t The parameters are trained to obtain the first high-dimensional feature e. t-1 Second high-dimensional feature e t ;

[0103] Based on the first high-dimensional feature e t-1 Second high-dimensional feature e t The feature similarity is calculated using the following formula:

[0104]

[0105] in, and They respectively represent the following: and The result after transformation into a one-dimensional tensor The α-th channel represents the second feature. Let S(·) represent the α-th channel of the first feature, S(·) represent the similarity calculation operation, and d(·) represent the distance metric function, where α∈{1,2,…,N} c}, N c Indicates the number of feature channels.

[0106] Preferably, the features are sorted according to their similarity, resulting in a feature similarity ranking result R. ε :

[0107]

[0108] in, Represents the α-th channel feature and The similarity between them, Φ represents the similarity ranking operation based on the feature similarity calculation formula, S represents the operation of channel-dimensional feature correlation calculation, and t represents the t-th incremental training step;

[0109] Based on the feature similarity ranking results, the first high-dimensional feature e is... t-1 Second high-dimensional feature e t Decoupling is performed, and features with similarity greater than or equal to a threshold are classified as semantically invariant features e. SI Features with similarity less than a threshold are classified as sample relevance features e. SS ;

[0110] Based on e SI and e SS Establish corresponding prototype alignment loss functions L SPM The loss function L for preserving features related to samplesSFP ;

[0111] According to L SPM and L SFP Constructing a decoupled knowledge distillation loss function L DD :

[0112]

[0113] Where, N l This represents the total number of layers in the neural network, and l represents the current layer of the neural network.

[0114] Preferably, the present invention will M t-1 and M t The extracted high-dimensional features are embedded in the semantic features e in the latent space and decoupled into semantically invariant features e. SI and sample correlation features e SS And satisfy in Indicates channel-dimensional fusion, e SI Represents the core semantic content related to the category, e SS It enhances the model's generalization ability, maintains consistency between the output of the old model and the current model, and avoids the problem of model classifier degradation caused by semantic confusion.

[0115] Preferably, the prototype alignment loss function L SPM Specifically:

[0116]

[0117]

[0118]

[0119]

[0120]

[0121] in, This represents the loss function for aligning similar prototypes. C represents the heterogeneous prototype separation loss function. 0:t-1 C represents the number of categories learned in step t-1. 0:t This represents the number of categories learned in step t, where k represents the number of categories. Represents the feature prototype. Let KL represent the new class prototype calculated in the t-th learning step, and KL represent the KL divergence. p represents the feature vector output by the model in the current step. bg The background prototype is defined as the mean of all category prototypes, M.x This represents the segmentation label of image x. Let Ω represent the semantically invariant feature embedding of the output with respect to image x, and let Ω represent the indicator function.

[0122] Preferably, p bg Defined as the mean of all category prototypes, it can reduce the interference of background semantic changes on category prototypes. Background prototype p bg The calculation and update are performed after each incremental training step is completed.

[0123] Preferably, the sample correlation feature preservation loss function L SFP Specifically:

[0124]

[0125]

[0126] Wherein d(e t ,e t-1 ) represents a global consistency constraint. D represents the class consistency constraint. CL (·) represents the contrastive learning loss of triplet. and These respectively represent anchored class embedding, positive sample class embedding, and negative sample class embedding. Indicates that it comes from M t-1 Regarding the anchored category embedding of the k-th class, and They represent from M t Regarding the positive and negative sample class embeddings of the k-th class, m represents the normality coefficient.

[0127] Preferably, the sample correlation feature e obtained by decoupling high-dimensional features is... SS Since the number of categories increases continuously in the incremental learning steps, this invention proposes an asymmetric distillation mechanism based on contrastive learning. The asymmetry of this distillation mechanism is manifested in two aspects: first, the inconsistency between the number of categories in the current learning step and the number of categories already learned; typically, the number of learned categories gradually increases with the accumulation of learning steps; second, the inconsistency in the pairing mechanism of feature embedding, where the anchored category embedding changes from M... t-1 The output is obtained from M, including the positive sample class embedding and the negative sample class embedding. t The output is obtained.

[0128] Preferably, the positive sample class embedding represents the class that is consistent with the anchor class embedding, and the negative sample class embedding represents the class that is inconsistent with the anchor class embedding.

[0129] like Figure 4As shown, a semantic consistency loss function is constructed based on the first semantic segmentation model and the second semantic segmentation model; Figure 3 In this context, RS (Relevance Score) represents the relevance score for a category, with a value range of [0,1].

[0130] Preferably, a semantic consistency loss function L is established. NSC Specifically, it includes:

[0131] The correlation score is calculated using the Layer-wise Relevance Propagation (LRP) method. The result is propagated from layer l to layer (l-1):

[0132]

[0133] Where j and k represent two neurons in adjacent layers of the neural network, This represents the relevance score of neuron j in the (l-1)th layer of the neural network. This represents the relevance score of neuron k in the l-th layer of the neural network. This represents the activation response between neuron j and neuron k in the l-th layer of the neural network; by propagating layer by layer from the deepest to the shallowest layers of the neural network, the contribution of each pixel in the image to the final prediction result can be obtained.

[0134] The neuron correlation g(x,c) is obtained by summing the correlation scores propagated layer by layer:

[0135]

[0136] Among them, R (l) This represents the correlation score of all neurons in the l-th layer of the neural network. Indicates a dimension of N c The feature space of , where c represents the category;

[0137] Based on the neuron correlation calculation formula g(x,c), the neurons from M are respectively obtained. t-1 Neuron-related g t-1 (x,c) and from M t Neuronal correlation g t (x,c);

[0138] Based on g t-1 (x,c) and g t The semantic consistency loss function L is calculated from (x,c). NSC :

[0139]

[0140] After constructing fused annotations based on the pseudo-labels generated by the prediction results of the first semantic segmentation model and the supervision information of the current step, a cross-entropy loss function is constructed with the prediction results of the second semantic segmentation model.

[0141] Preferably, the cross-entropy loss function is constructed, specifically including:

[0142] Based on the first semantic segmentation model M t-1 The prediction results generate uncertain pseudo-labels for the old categories, which are then fused with the real labels for the new categories to obtain fused labels.

[0143] According to the second semantic segmentation model M t The prediction results and fused annotations are used to obtain the cross-entropy loss function L. CE :

[0144]

[0145]

[0146] in, M represents the second semantic segmentation model t During training, the fused annotation corresponding to pixel i M represents the second semantic segmentation model t The output is the prediction result for pixel i, F θ Describing a feature extraction network, x i Represents pixel i in the original image.

[0147] The preferred method for generating uncertain pseudo-labels is as follows:

[0148]

[0149]

[0150]

[0151]

[0152] in M represents the first semantic segmentation model t-1 The prediction result of the output pixel i, c u c represents an unknown category. b Let Γ represent the background category, ζ represent the threshold for controlling the confidence of pseudo-labels, and u represent the threshold for measuring the stability of the model's prediction results. i This represents the uncertainty of pixel i. This represents the highest value predicted by the model for the category. Δ represents the second-highest value for the class predicted by the model. iThis represents the range of uncertainty for pixel i. This represents the lowest value of the category predicted by the model.

[0153] Preferably, in order to achieve effective updates to the semantic segmentation model, this invention uses a pseudo-label generation method to obtain supervision information of old categories, without the need to store old data.

[0154] Preferably, the actual annotations for the newly added categories are generated manually.

[0155] Preferably, when u i When the value is less than the threshold, it indicates that the prediction result has high confidence. i When the value exceeds the threshold, it indicates that the confidence level of the prediction result is low.

[0156] Preferably, the present invention addresses the unknown category c. u Explicit modeling was performed to avoid semantic confusion caused by the context class's semantics simultaneously including the real context class, known classes, and unknown classes during the continuous learning process.

[0157] The prediction results of the second semantic segmentation model are trained under supervision with the cross-entropy loss function of the fused labeled input. The prediction results of the second semantic segmentation model and the first semantic segmentation model are jointly trained by inputting the decoupled knowledge distillation loss function and the semantic consistency loss function. The parameters of the second semantic segmentation model are updated together to obtain the optimized semantic segmentation model with updated parameters.

[0158] The image to be segmented is input into the optimal semantic segmentation model to obtain the corresponding semantic segmentation result.

[0159] Preferably, the semantic segmentation result is a single-channel image of the same size as the original image, with different gray values ​​representing different categories.

[0160] Example 3

[0161] Semantic segmentation is a typical computer vision problem that assigns a label to each pixel in an image so that pixels with the same label are associated with a single semantic entity in the image. Each label belongs to a set of predefined object categories, such as people, cars, trees, and sky, to achieve pixel-level classification of image content. The continuous semantic segmentation method of this invention has applications covering at least autonomous driving systems and remote sensing image processing that use digital images as carriers.

[0162] like Figure 5 As shown, the continuous semantic segmentation method of the present invention achieves the following effect after performing category-increment continuous semantic segmentation in remote sensing Earth observation scenarios:

[0163] In the diagram, Step 0, Step 1, etc., represent continuous learning steps, with different colors representing different semantic categories. Step 0 learns two categories: impermeable ground and buildings; after training, the model can predict these two categories. Step 1 adds three new categories: low vegetation, trees, and vehicles. The method of this invention allows the model to continuously learn new categories based on known categories, while maintaining its semantic segmentation ability for known categories. The model trained on the new data can include both the two known categories and the three new categories added in the current step in its prediction results. The ground truth labeled images contain all manually labeled semantic categories and are used to evaluate the learning performance of the continuous semantic segmentation model.

[0164] Example 4

[0165] The superiority of the proposed method was verified using three general semantic segmentation datasets:

[0166] The general semantic segmentation datasets are Pascal VOC 2012, ADE20K, and ISPRS; Pascal VOC2012 contains 20 target classes, ADE20K contains 150 target classes, and ISPRS contains 5 target classes.

[0167] For the aforementioned general semantic segmentation datasets, the incremental semantic segmentation classes are set as VOC 15-1, ADE100-10, and ISPRS2-1, respectively. For VOC 15-1, the first step learns 15 classes, and then each incremental learning step learns 1 class, for a total of 6 incremental learning steps. For ADE 100-10, the first step learns 100 classes, and then each incremental learning step learns 10 classes, for a total of 6 incremental learning steps. For ISPRS2-1, the first step learns 4 classes, and then each incremental learning step learns 1 class, for a total of 2 incremental learning steps.

[0168] Ablation experiments were conducted on the proposed method using the three semantic segmentation datasets described above. The average intersection-union ratio (IUU) was used as the evaluation metric, and the results are shown in Table 1.

[0169] Table 1 Ablation Experiment Results of Persistent Semantic Segmentation Algorithm

[0170]

[0171] In Table 1, "fine-tuning" indicates an isolated learning approach, where training is performed using only the newly added category labels; "SPM" indicates training supervised by semantically invariant prototype alignment constraints; "SFP" indicates training supervised by sample relevance feature preservation constraints; "SPM&SFP" indicates training supervised by both semantically invariant prototype alignment constraints and sample relevance feature preservation constraints; and "NSC" indicates training supervised by neuron relevance semantic consistency constraints. "all" represents the mean Intersection over Union (mIoU) for all categories, and a higher mIoU value indicates better model performance.

[0172] Experimental results show that, compared with the traditional isolated learning method, the decoupled knowledge distillation method proposed in this invention can effectively improve the performance of category incremental semantic segmentation tasks.

[0173] Example 5

[0174] Performance tests of the semantic segmentation model were conducted on the general semantic segmentation dataset Pascal VOC 2012 under limited incremental data conditions.

[0175] The category incremental semantic segmentation is set to VOC 15-1. For this incremental learning task, the first step learns 15 categories, and then each incremental learning step learns 1 category, for a total of 6 incremental learning steps.

[0176] The experimental tests used the mean Intersection over Union (mIoU) as the performance evaluation index. A higher mIoU value indicates better model performance. 'all' represents the mean IoU for all classes. The results are shown in Table 2.

[0177] Table 2 Performance test results of the continuous learning semantic segmentation model under limited incremental data conditions

[0178]

[0179] Compared with the existing method PLOP, the model of the present invention showed an overall performance gain of 8.72% when the incremental data ratio decreased from 100% to 10%, which is far less than the 26.17% of the PLOP model, proving the effectiveness of the present invention.

[0180] The qualitative comparison results of the model of this invention on the VOC 15-1 task are as follows: Figure 6 As shown, compared with the existing incremental semantic segmentation method PLOP, the model proposed in this invention has significant advantages in balancing the learning of new and old categories and reducing semantic confusion.

[0181] Example 6

[0182] like Figure 7 As shown, this embodiment of the invention discloses a finite data persistent semantic segmentation system based on decoupled knowledge distillation, including: a model acquisition module, a first loss function construction module, a second loss function construction module, a third loss function construction module, a model training module, and a result output module;

[0183] The model acquisition module is used to construct an initial semantic segmentation model. The first semantic segmentation model is obtained by incremental learning and training at step t-1, and the second semantic segmentation model is obtained by incremental learning and training at step t.

[0184] The first loss function construction module is used to obtain feature similarity based on the first semantic segmentation model and the second semantic segmentation model, and to construct a decoupled knowledge distillation loss function based on feature similarity.

[0185] The second loss function construction module is used to construct a semantic consistency loss function based on the first semantic segmentation model and the second semantic segmentation model.

[0186] The third loss function construction module is used to construct a fused label based on the pseudo-labels generated by the prediction results of the first semantic segmentation model and the supervision information of the current step, and then construct a cross-entropy loss function with the prediction results of the second semantic segmentation model.

[0187] The model training module is used to supervise the training of the prediction results of the second semantic segmentation model and the cross-entropy loss function of the fused labeled input. It also combines the prediction results of the second semantic segmentation model and the prediction results of the first semantic segmentation model with the decoupled knowledge distillation loss function and the semantic consistency loss function for joint training. Together, they update the parameters of the second semantic segmentation model to obtain the optimized semantic segmentation model with updated parameters.

[0188] The result output module is used to input the image to be segmented into the optimal semantic segmentation model to obtain the semantic segmentation result.

[0189] As can be seen from the above technical solution, compared with the prior art, this invention discloses a method and system for continuous semantic segmentation of limited data based on decoupled knowledge distillation. Based on human cognitive patterns and combined with the superior learning ability of artificial neural networks, it establishes multiple constraints in the high-dimensional feature latent space extracted by the model, including neuron correlation semantic consistency constraints, decoupled knowledge distillation constraints, and semantic segmentation constraints, achieving the inheritance of old knowledge without the need for old data to participate in training. It has the following beneficial effects:

[0190] 1. It does not require storing old data and can achieve incremental learning capabilities for new categories while retaining the classification of known categories. It can achieve more robust continuous learning capabilities under limited incremental data conditions.

[0191] 2. This invention proposes a feature decoupling mechanism based on human cognitive patterns, which decouples the high-dimensional features of an image into semantically invariant features and sample correlation features, and the two satisfy the channel dimension additive relationship, thus solving the problem of catastrophic forgetting of old knowledge during continuous learning.

[0192] 3. This invention proposes a decoupled knowledge distillation method. By decoupling semantic invariant features and sample relevance features to establish knowledge distillation constraints, it can effectively alleviate the knowledge forgetting phenomenon in semantic segmentation models. At the same time, it can alleviate the semantic drift problem during the continuous learning process of the model under limited new data conditions. It can be applied to scenarios such as remote sensing earth observation and autonomous driving systems. For continuously increasing semantic categories and scenarios, it can achieve continuous model updates without significantly increasing storage space and computing resources.

[0193] 4. This invention proposes a semantic consistency constraint based on neuron correlation, which constrains the features output by the old and new models based on Euclidean distance, enabling effective knowledge transfer between the old and new models, enhancing the generalization and robustness of the continuous semantic segmentation model, and solving the problem of poor generalization of continuous learning models under cross-class and cross-modal incremental conditions; embedding neuron correlation constraints into the continuous learning model improves the model's continuous learning performance while enhancing its interpretability.

[0194] 5. This invention proposes a pseudo-label generation method based on uncertainty, which improves the ability of the continuously incremental semantic segmentation model to inherit old knowledge under the condition of limited incremental data, and provides reliable supervision information for the training of new models.

[0195] 6. The continuous learning method proposed in this invention can dynamically adjust the parameters of the model during the learning process to adapt to the input of new data, while retaining previously learned knowledge. It can continuously learn new tasks or categories without forgetting previously learned knowledge, thus achieving continuous learning. By using the continuous learning method, we can better understand and imitate the learning methods of the human brain and the structural mechanism of biological neural networks at the system level, providing a technical foundation for developing new computational models and effective learning algorithms.

[0196] 7. As a theoretical and technical exploration in the field of continuous semantic segmentation, this invention provides a theoretical and technical foundation for the development of strong artificial intelligence theory and the application of artificial intelligence in embedded platforms. At the same time, it provides theoretical support and technical verification for research in the academic fields of artificial intelligence and computer vision and the implementation of related products in the industry.

[0197] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0198] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A limited data continual semantic segmentation method based on decoupled knowledge distillation, characterized in that, The method comprises the following steps: An initial semantic segmentation model is constructed, a first semantic segmentation model is obtained through t-1 step incremental learning training, and a second semantic segmentation model is obtained through t step incremental learning training; A feature similarity is obtained based on the first semantic segmentation model and the second semantic segmentation model, and a decoupling knowledge distillation loss function is constructed based on the feature similarity; The feature similarity is obtained, specifically comprising: The training set images are input into the first semantic segmentation model. and the second semantic segmentation model The first feature is obtained accordingly. Second feature ; The feature similarity is calculated based on the first feature and the second feature, and the feature similarity calculation formula is as follows: ； wherein, and denote the result of transforming and to one-dimensional tensors, denotes the alpha-th channel of the second feature, denotes the alpha-th channel of the first feature, denotes a similarity computation operation, denotes a distance metric function, , denotes the number of feature channels; The feature similarity degree is ranked according to the feature similarity degree size, and a feature similarity degree ranking result is obtained : ； wherein, denotes a similarity between the features of the channel dimension and , denotes a similarity ranking operation based on a feature similarity computation formula, S denotes an operation of channel dimension feature correlation computation, and t denotes a t-th step incremental training step. According to the feature similarity ranking result, the first feature and the second feature are decoupled, features with similarity greater than or equal to a threshold are classified as semantic invariance features , and features with similarity less than the threshold are classified as sample correlation features . Based on and corresponding to the prototype alignment loss function and the sample correlation feature retention loss function respectively According to and constructing decoupled knowledge distillation loss function : ； wherein, represents the total number of layers of the neural network, represents the current layer of the neural network; A semantic consistency loss function is constructed based on the first semantic segmentation model and the second semantic segmentation model; A cross-entropy loss function is constructed based on the pseudo-label generated by the prediction result of the first semantic segmentation model and the supervision information of the current step, and the fusion label is constructed. The prediction result of the second semantic segmentation model is input into the cross-entropy loss function for supervised training, and the prediction result of the second semantic segmentation model and the prediction result of the first semantic segmentation model are jointly input into the decoupling knowledge distillation loss function and the semantic consistency loss function for joint training, so that the parameters of the second semantic segmentation model are updated, and an optimized semantic segmentation model is obtained after the parameters are updated. An image to be segmented is input into the optimal semantic segmentation model to obtain a semantic segmentation result.

2. The method of claim 1, wherein, Prototype alignment loss function In particular: ； ； ； wherein, represents a same-class prototype alignment loss function, represents a different-class prototype separation loss function, represents the number of classes that have been learned in the t-1th step, represents the number of classes that have been learned in the tth step, k represents a class, represents a feature prototype, represents a new class prototype calculated in the tth learning step, KL represents a KL divergence, represents a feature vector output by a model in a current step.

3. The method of claim 2, wherein, the new class prototype calculated in the t-th step of learning , in particular: ； ； wherein, denotes the number of images belonging to class k in each iteration step, denotes the mean of all class prototypes, denotes the segmentation annotation of image x, denotes the output semantic invariance feature embedding for image x, denotes the indicator function.

4. The method of claim 3, wherein, Sample correlation feature preserving loss function In particular, the application relates to ； ； wherein, denotes a triple contrastive learning loss, , and respectively correspond to denote an anchor class embedding, a positive sample class embedding and a negative sample class embedding, denotes a positive sample class embedding, an anchor class embedding for the k-th class, and respectively denote a positive sample class embedding and a negative sample class embedding, a negative sample class embedding for the k-th class, m denotes a normalizing coefficient.

5. The method of claim 4, wherein, The semantic consistency loss function is constructed, specifically comprising: Computing a correlation score From Layer to the Layer propagates results: ； Where j and k represent two neurons in adjacent layers of the neural network, Indicates the neural network's first... The correlation score of neuron j in the layer. Indicates the neural network's first... The correlation score of neuron k in the layer. Indicates the neural network's first... The activation response between neuron j and neuron k in the layer; computing a sum of all neuron correlation scores propagating results layer by layer, obtaining neuron correlation : ； wherein, represents the correlation score of a neuron in the neural network layer represents the correlation score of all neurons in the neural network layer represents a feature space of dimension ​ According to the neuron correlation calculation formula respectively obtain the neuron correlation from The neuron correlation and the neuron correlation from ;​ Based on and The semantic consistency loss function is calculated : 。 6. The method of claim 5, wherein, The cross-entropy loss function is constructed, specifically comprising: generating, based on a prediction result of the first semantic segmentation model an uncertainty pseudo-label of the old class, and fusing the uncertainty pseudo-label with a real label of the new class to obtain a fusion label; According to the prediction result of the second semantic segmentation model and the fusion label, a cross-entropy loss function is obtained. ； ； wherein, denotes a second semantic segmentation model a fusion annotation corresponding to pixel i in the training process, denotes a second semantic segmentation model a predicted result for pixel i outputted by the second semantic segmentation model, denotes a feature extraction network, denotes pixel i in the original image.

7. The method of claim 6, wherein, The uncertainty pseudo-label generation method comprises the following steps: ； ； ； ； wherein denotes a first semantic segmentation model denotes a prediction result of a pixel i, denotes an unknown class, denotes a background class, denotes a threshold for controlling pseudo-label confidence, denotes a threshold for measuring stability of model prediction results, denotes an uncertainty of the pixel i, denotes a highest value of a model predicted class, denotes a second highest value of a model predicted class, denotes an uncertainty range of the pixel i, denotes a lowest value of a model predicted class.

8. A limited data continual semantic segmentation system based on decoupled knowledge distillation, configured to perform the method of limited data continual semantic segmentation based on decoupled knowledge distillation according to any one of claims 1-7, characterized in that, The method comprises the following steps: The model acquisition module is used to construct an initial semantic segmentation model, a first semantic segmentation model is obtained through t-1 step incremental learning training, and a second semantic segmentation model is obtained through t step incremental learning training; The first loss function construction module is used to obtain a feature similarity based on the first semantic segmentation model and the second semantic segmentation model, and a decoupling knowledge distillation loss function is constructed based on the feature similarity; The second loss function construction module is used to construct a semantic consistency loss function based on the first semantic segmentation model and the second semantic segmentation model; The third loss function construction module is used to construct a cross-entropy loss function based on the pseudo-label generated by the prediction result of the first semantic segmentation model and the fusion label constructed after the supervision information of the current step, and the prediction result of the second semantic segmentation model; The model training module is used to input the prediction result of the second semantic segmentation model and the fusion label into the cross-entropy loss function for supervised training, and jointly input the prediction result of the second semantic segmentation model and the prediction result of the first semantic segmentation model into the decoupling knowledge distillation loss function and the semantic consistency loss function for joint training, so that the parameters of the second semantic segmentation model are updated, and an optimized semantic segmentation model is obtained after the parameters are updated. The result output module is used to input an image to be segmented into the optimal semantic segmentation model to obtain a semantic segmentation result. ​

Images