Incremental semantic segmentation method and automatic driving image semantic segmentation method

Abstract

The invention discloses an incremental semantic segmentation method and an automatic driving image semantic segmentation method. The method comprises the steps that a target picture data set is acquired and preprocessed to construct a training data set; the method comprises the following steps: constructing an incremental semantic segmentation primary model comprising an encoder, a decoder and a distiller on the basis of a global attention mechanism, a Transform mechanism and a self-adaptive multi-scale distillation mechanism; in combination with embedding loss, distillation loss, background loss and cross loss, training the incremental semantic segmentation primary model to obtain an incremental semantic segmentation model; and carrying out semantic segmentation on an actual image by adopting the obtained incremental semantic segmentation model. According to the method, the incremental semantic segmentation model including the global attention mechanism and the self-adaptive multi-scale distillation mechanism is constructed, and the model is trained by adopting the loss function including the embedding loss, the distillation loss and the background loss, so that the semantic segmentation of the image is realized, the reliability is higher, and the accuracy is better.

Description

Technical Field

[0001] The present invention belongs to the field of image processing, and in particular relates to an incremental semantic segmentation method and an automatic driving image semantic segmentation method. Background Art

[0002] The incremental semantic segmentation solution enables the model to gradually learn information about new categories without forgetting the already learned categories, thereby adapting to the dynamically changing actual environment. This solution combines the advantages of incremental learning and semantic segmentation to solve the limitations of traditional methods in open scenarios and provide a more flexible and efficient solution for practical applications. The incremental semantic segmentation solution can effectively cope with the challenges of dynamic category changes and open scenarios, greatly improving the robustness and flexibility of intelligent systems.

[0003] Existing incremental semantic segmentation solutions address key challenges in incremental learning (such as catastrophic forgetting) through pseudo-labeling strategies and knowledge distillation, and have achieved certain results. However, existing incremental semantic segmentation solutions ignore the contextual information between each task, resulting in a lack of integrity in the learning process, so the reliability and accuracy of the final model are relatively low. In addition, existing incremental semantic segmentation solutions will directly distill all the knowledge of the old model into the new model, but some of the knowledge of the old model may have a negative effect on the new model; this situation will also reduce the reliability and accuracy of the final model.

[0004] Similarly, in the process of autonomous driving, an incremental semantic segmentation solution is needed to perform semantic segmentation of autonomous driving images, and the segmentation results will directly provide data support for the autonomous driving process. However, the low reliability and low accuracy of the existing incremental semantic segmentation solution will greatly affect the semantic segmentation results of autonomous driving images, and thus affect the reliability of autonomous driving. Summary of the invention

[0005] One of the objectives of the present invention is to provide an incremental semantic segmentation method with high reliability and good accuracy.

[0006] A second object of the present invention is to provide an autonomous driving image semantic segmentation method including the incremental semantic segmentation method.

[0007] The incremental semantic segmentation method provided by the present invention comprises the following steps: S1. Obtain the target image dataset; S2. Preprocessing the target image dataset obtained in step S1 to construct a training dataset; S3. Based on the global attention mechanism, Transformer mechanism and adaptive multi-scale distillation mechanism, a primary model of incremental semantic segmentation including encoder, decoder and distiller is constructed; The encoder is used to encode the input image data to extract the feature information of the input image; by replacing the attention module in the Transformer mechanism with a global attention mechanism module, the decoder decodes the feature information extracted by the encoder to obtain global information; the distiller is used to perform feature distillation and fusion operations on the output of the decoder to ultimately achieve semantic segmentation; S4. Using the training dataset constructed in step S2, combined with embedding loss, distillation loss, background loss and cross loss, the incremental semantic segmentation primary model constructed in step S3 is trained to obtain an incremental semantic segmentation model; S5. Use the incremental semantic segmentation model obtained in step S4 to perform semantic segmentation of the actual image.

[0008] The step S3 comprises the following steps: Build an encoder based on the Transformer mechanism to extract feature information of the input image; In the decoder, the background class label, the learned new label and the global label are used as features and input into the Transformer decoder for decoding to obtain global information; wherein, the attention module in the Transformer decoder is replaced by a global attention mechanism module; A distiller is constructed based on an adaptive multi-scale distillation mechanism to perform feature distillation and fusion operations on the output of the decoder to ultimately achieve semantic segmentation.

[0009] The encoder specifically includes the following contents: The constructed encoder includes an embedding layer, a linear projection layer, and a Transformer encoder layer; The embedding layer is used to divide the input image into several visual blocks and input the visual blocks into the linear projection layer; The linear projection layer is used to linearly project the input visual block to map the input visual block into a high-dimensional representation vector, and input the obtained data information into the Transformer encoder layer; The Transformer encoder layer is used to extract the feature information and dependencies of the input image based on the input data information to obtain the corresponding encoder result.

[0010] The decoder specifically includes the following contents: The constructed decoder includes a feature construction layer and a Transformer decoder layer; The feature construction layer is used to construct the feature information required by the Transformer decoder layer; the features constructed by the feature construction layer include the target class labels to be learned and the global label to be learned , and the constructed , Together with the encoder result, it is input into the Transformer decoder layer; among them, the target class label Expressed as ,in is the background class marker, is the label of the new category learned in the current t-th round of training; global label Used to aggregate and represent the global knowledge of the current task. Before the first training, the global mark Initialize to the set value , during the current t-th round of training, the global mark The value of is the global label value obtained from the t-1th round of training ; The Transformer decoder layer is used to and , decode the encoder result to obtain global information; in the Transformer decoder layer, replace the attention module in the Transformer decoder with a global attention mechanism module to obtain global information; The output of the decoder includes decoder features and background class marker features, where the decoder features are the features obtained after the encoder results are processed by the decoder, and the background class marker features are Features obtained after being processed by the decoder.

[0011] The global attention mechanism module specifically includes the following contents: In the current t round of training, the encoder result The label of the new category learned in the current t round of training Perform stitching and obtain the first stitching feature ;Will Perform normalization to obtain the normalized first splicing feature , and As a query The value of The value of the global tag of the current t-th round of training Perform splicing and obtain the first splicing mark , and as a key Value and value The value of The processing of the global attention mechanism module is expressed as: In the formula is the output of the global attention mechanism module; is the weight matrix of the output layer; is the processing function of the multi-layer perceptron; It is the processing function of two mask normalization layers connected in series; is the bias term of the output layer; is an intermediate feature, and , is the softmax function, d is the embedding size, and h is the number of attention heads.

[0012] The distiller specifically comprises the following contents: The distiller includes a reshaping layer, a similarity calculation layer, an upsampling layer, and a normalization layer; In the current t round of training, the decoder features After reshaping through the reshape layer, the reshape feature is obtained ;Will With background class marker features The cosine similarity is calculated by the similarity calculation layer; the cosine similarity is upsampled by the upsampling layer and then processed by the normalization layer to obtain the final segmentation prediction result; Among them, the normalization layer uses the softmax function for normalization.

[0013] The training described in step S4 includes the following steps: In the current t-th round of training, the embedding loss is calculated based on the cosine similarity between the current background class marker feature and the background class marker feature of the previous round; In the current t-th round of training, the distillation loss is calculated based on the cosine similarity between the feature segmentation results of the current reshaped features at different scales and the feature segmentation results of the previous round of reshaped features at different scales; In the current t-th round of training, the background loss is calculated based on the KL divergence between the current background output and the background output of the previous round, and the difference between the current boundary area and the boundary area of ​​the previous round; In the current t-th round of training, the cross loss is calculated based on the current segmentation prediction result, the current true value and the segmentation prediction result of the previous round; Finally, the obtained embedding loss, distillation loss, background loss and cross loss are weighted summed as the total loss function of the current t-th round of training; The obtained total loss function is used for training.

[0014] The training described in step S4 specifically includes the following steps: Embedding loss: In the current t-th round of training, the embedding loss is calculated using the following formula : In the formula is the total number of old categories in the current t-th round of training; Mark the features of the current i-th background class; Mark the features of the i-th background class in the previous round; Calculate the cosine similarity function; Distillation loss: Reshape features Segmentation is performed at the set scale s; then at each segmentation scale, the segmented features are averaged pooled in the height and width directions, expressed as: In the formula is the height-direction pooling embedding result at scale s; H is the reshaped feature Height; for The value at height h, width w and channel c; is the width-wise pooling embedding result at scale s; W is the reshaped feature Width; At each scale, the obtained pooling embedding results in the height direction and the pooling embedding results in the width direction are concatenated to obtain the pooling embedding results at the corresponding scale; Concatenate the pooling embedding results at each scale to obtain a high-dimensional feature representation ; In the current t-th round of training, the distillation loss is calculated using the following formula : In the formula is the reliability mapping result, and its value is the cosine similarity value between the current pseudo label and the segmentation prediction result of the previous round; The value of the current high-dimensional feature representation at a height of h and a width of w; The high-dimensional feature representation of the previous round has a value at a height of h and a width of w; is the Euclidean distance; Background loss: In the current t-th round of training, the background loss is calculated using the following formula : In the formula is the first weight value; is the second weight value; is the background confidence loss, and , N is the total number of background pixels, bg is the background pixel, is the background confidence weight, is the output probability distribution of the model at pixel i during the current t-th round of training, is the output probability distribution of the model at pixel i during the t-1th round of training, is the KL divergence calculation, and , C is the total number of categories in the current task, is the maximum value of the new model’s predicted probability for category c at pixel i, is the maximum value of the predicted probability of category c at pixel i by the old model; is the boundary area loss, and , M is the total number of pixels in the boundary area, T is the number of pixels in the boundary area, is the reshaped feature of the pixel ii in the boundary area during the current t-th round of training, is the reshaped feature of pixel ii in the boundary area during the t-1th round of training; Cross Loss: In the current t-th round of training, the cross loss is calculated using the following formula : In the formula is a pseudo label, and its value is the fusion value of the segmentation prediction result of the iii-th pixel with classification category c in the t-1th round of training and the true value of the iii-th pixel in the current t-th round of training; is the segmentation prediction result of the iii-th pixel in the current t-th round of training with classification category c; The total loss function L is calculated using the following formula: In the formula is the embedding loss weight value; is the distillation loss weight value; is the background loss weight value; is the cross loss weight value; Finally, the total loss function L is used for training.

[0015] The present invention also provides an automatic driving image semantic segmentation method including the incremental semantic segmentation method, comprising the following steps: A. Obtain the autonomous driving image to be segmented; B. Using the incremental semantic segmentation method, semantic segmentation is performed on the autonomous driving image obtained in step A; C. Based on the semantic segmentation results obtained in step B, complete the semantic segmentation of the autonomous driving image.

[0016] The incremental semantic segmentation method and autonomous driving image semantic segmentation method provided by the present invention not only realize the semantic segmentation of the image, but also have higher reliability and better accuracy by constructing an incremental semantic segmentation model including a global attention mechanism and an adaptive multi-scale distillation mechanism, and training the model using loss functions including embedding loss, distillation loss, and background loss. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 Schematic diagram of the method flow of the incremental semantic segmentation method of the present invention.

[0018] Figure 2 Schematic diagram of comparison effect of an embodiment of the incremental semantic segmentation method of the present invention.

[0019] Figure 3 Schematic diagram of the method flow of the autonomous driving image semantic segmentation method of the present invention. DETAILED DESCRIPTION

[0020] like Figure 1 The figure shows a schematic diagram of the method flow of the incremental semantic segmentation method of the present invention: The incremental semantic segmentation method disclosed in the present invention comprises the following steps: S1. Obtain a target image dataset; for example, if you want to perform semantic segmentation of remote sensing images, obtain an existing image dataset for semantic segmentation of remote sensing images; or if you want to perform semantic segmentation of fundus images, obtain an existing image dataset for semantic segmentation of fundus images.

[0021] S2. Preprocess the target image dataset obtained in step S1 to construct a training dataset.

[0022] S3. Based on the global attention mechanism, Transformer mechanism and adaptive multi-scale distillation mechanism, a primary model of incremental semantic segmentation including encoder, decoder and distiller is constructed; Among them, the encoder is used to encode the input image data to extract the feature information of the input image; by replacing the attention module in the Transformer mechanism with the global attention mechanism module, the decoder decodes the feature information extracted by the encoder to obtain global information; the distiller is used to perform feature distillation and fusion operations on the output of the decoder to ultimately achieve semantic segmentation.

[0023] When implementing: Build an encoder based on the Transformer mechanism to extract feature information of the input image; In the decoder, the background class label, the learned new label and the global label are used as features and input into the Transformer decoder for decoding to obtain global information; wherein, the attention module in the Transformer decoder is replaced by a global attention mechanism module; A distiller is constructed based on an adaptive multi-scale distillation mechanism to perform feature distillation and fusion operations on the output of the decoder to ultimately achieve semantic segmentation.

[0024] The constructed encoder includes an embedding layer, a linear projection layer, and a Transformer encoder layer; The embedding layer is used to divide the input image into several visual blocks and input the visual blocks into the linear projection layer; The linear projection layer is used to linearly project the input visual block to map the input visual block into a high-dimensional representation vector, and input the obtained data information into the Transformer encoder layer; The Transformer encoder layer is used to extract the feature information and dependencies of the input image based on the input data information to obtain the corresponding encoder result.

[0025] The constructed decoder includes a feature construction layer and a Transformer decoder layer; The feature construction layer is used to construct the feature information required by the Transformer decoder layer; the features constructed by the feature construction layer include the target class labels to be learned and the global label to be learned , and the constructed , Together with the encoder result, it is input into the Transformer decoder layer; among them, the target class label Expressed as ,in is the background class marker, is the label of the new category learned in the current t-th round of training; global label Used to aggregate and represent the global knowledge of the current task. Before the first training, the global mark Initialize to the set value , during the current t-th round of training, the global mark The value of is the global label value obtained from the t-1th round of training ; Global tags are used to aggregate and represent the global knowledge of the current task. When entering a new task, the global tags are updated so that they can incorporate the key information of the new task and dynamically adjust to automatically adapt to the needs of the new task. At the same time, global tags can effectively improve the accuracy of semantic segmentation by interacting with local features, especially showing significant advantages when dealing with complex scenes and fine-grained features. The Transformer decoder layer is used to and , decode the encoder result to obtain global information; in the Transformer decoder layer, replace the attention module in the Transformer decoder with a global attention mechanism module to obtain global information; The output of the decoder includes decoder features and background class marker features, where the decoder features are the features obtained after the encoder results are processed by the decoder, and the background class marker features are Features obtained after being processed by the decoder.

[0026] The global attention mechanism module in the decoder includes the following: The global attention mechanism module dynamically adjusts the model's attention to different regions when processing input data by introducing global information and contextual relationships. The core idea of ​​this module is to continuously update and enhance the use of global knowledge as time goes by and task requirements change, ensuring that the model can obtain more comprehensive and accurate information in complex tasks. In the current t round of training, the encoder result The label of the new category learned in the current t round of training Perform stitching and obtain the first stitching feature ;Will Perform normalization to obtain the normalized first splicing feature , and As a query The value of The value of the global tag of the current t-th round of training Perform splicing and obtain the first splicing mark , and as a key Value and value The value of The processing of the global attention mechanism module is expressed as: In the formula is the output of the global attention mechanism module; is the weight matrix of the output layer; is the processing function of the multi-layer perceptron; It is the processing function of two mask normalization layers connected in series; is the bias term of the output layer; is an intermediate feature, and , is the softmax function, d is the embedding size, and h is the number of attention heads. The addition of mask normalization layer and multi-layer perceptron enhances the expressiveness and stability of the global attention block, and also improves the robustness and learning efficiency of the model.

[0027] The distiller includes a reshaping layer, a similarity calculation layer, an upsampling layer, and a normalization layer; In the current t round of training, the decoder features After reshaping through the reshape layer, the reshape feature is obtained ;Will With background class marker features The cosine similarity is calculated through the similarity calculation layer; the cosine similarity is upsampled through the upsampling layer and then normalized through the normalization layer to obtain the final segmentation prediction result; the distiller ensures that the model can focus on the most beneficial feature representation by dynamically adjusting the features of different scales, effectively improving the expressiveness and adaptability of the model, and can also effectively fuse features from different scales, further alleviating the information loss problem of the traditional incremental semantic segmentation model when processing multi-scale features; Among them, the normalization layer uses the softmax function for normalization.

[0028] S4. Using the training dataset constructed in step S2, combined with embedding loss, distillation loss, background loss and cross loss, the incremental semantic segmentation primary model constructed in step S3 is trained to obtain an incremental semantic segmentation model; In the current t-th round of training, the embedding loss is calculated based on the cosine similarity between the current background class marker feature and the background class marker feature of the previous round; In the current t-th round of training, the distillation loss is calculated based on the cosine similarity between the feature segmentation results of the current reshaped features at different scales and the feature segmentation results of the previous round of reshaped features at different scales; In the current t-th round of training, the background loss is calculated based on the KL divergence between the current background output and the background output of the previous round, and the difference between the current boundary area and the boundary area of ​​the previous round; In the current t-th round of training, the cross loss is calculated based on the current segmentation prediction result, the current true value and the segmentation prediction result of the previous round; Finally, the obtained embedding loss, distillation loss, background loss and cross loss are weighted summed as the total loss function of the current t-th round of training; The obtained total loss function is used for training.

[0029] In specific implementation, the training process includes the following steps: Embedding loss: In the current t-th round of training, the embedding loss is calculated using the following formula : In the formula is the total number of old categories in the current t-th round of training; Mark the features of the current i-th background class; Mark the features of the i-th background class in the previous round; Calculate the cosine similarity function; By calculating the embedding loss and the similarity, the model prediction results can be limited from shifting toward the old categories, thereby reducing the impact of background shift on the prediction results. At the same time, the embedding loss can not only retain the global knowledge in the previous task and effectively avoid the problem of forgetting the old category knowledge, but also adjust the relationship between category embeddings, reduce the impact of background shift on the model, and enhance the robustness of the model in dealing with knowledge transfer between new and old categories.

[0030] Distillation loss: The redundant information in the old knowledge often interferes with the current model's learning of new classes, making the prediction results of new classes biased towards old classes or background classes. Therefore, distillation loss is used to correct this. Reshape features Segmentation is performed at the set scale s; then at each segmentation scale, the segmented features are averaged pooled in the height and width directions, expressed as: In the formula is the height-direction pooling embedding result at scale s; H is the reshaped feature Height; for The value at height h, width w and channel c; is the width-wise pooling embedding result at scale s; W is the reshaped feature Width; Through the above multi-scale segmentation and pooling operations, the model can adaptively extract reliable multi-scale features. This method can more comprehensively capture the useful semantic knowledge in the old model and dynamically adjust the distillation process, which significantly alleviates the catastrophic forgetting and background shift problems in the incremental learning process, further improving the model's ability to retain old category knowledge and its adaptability to learning new categories. At each scale, the obtained pooling embedding results in the height direction and the pooling embedding results in the width direction are concatenated to obtain the pooling embedding results at the corresponding scale; Concatenate the pooling embedding results at each scale to obtain a high-dimensional feature representation ; Through multi-scale feature pooling, global context information can be captured while retaining fine-grained local detail information, forming a multi-scale feature expression that takes both global and local considerations into account; In the current t-th round of training, the distillation loss is calculated using the following formula : In the formula is the reliability mapping result, and its value is the cosine similarity value between the current pseudo label and the segmentation prediction result of the previous round; The value of the current high-dimensional feature representation at a height of h and a width of w; The high-dimensional feature representation of the previous round has a value at a height of h and a width of w; is the Euclidean distance; The calculation of distillation loss focuses on key knowledge areas by dynamically adjusting the weights between different features, thereby enhancing the model's ability to learn important features. At the same time, through reliability mapping, a confidence score is assigned to each feature block, so that the model can adaptively adjust the learning weights according to the reliability of each feature block, thereby focusing more accurately on high-confidence areas.

[0031] Background loss: Another challenging problem in category-incremental semantic segmentation is the background shift problem, which is caused by the fact that the background pixels predicted by the old model may include the new categories in the current step, resulting in confusion and inaccuracy in the segmentation results. This background shift phenomenon will interfere with the learning process of the new category, especially when dealing with complex or dynamic scenes, the boundary between the background and the target becomes more blurred. Therefore, the method of the present invention adopts a background loss calculation process to solve this problem. In the current t-th round of training, the background loss is calculated using the following formula : In the formula is the first weight value; is the second weight value; is the background confidence loss (which enables dynamic adjustment of the consistency of background outputs between the new and old models during training), and , N is the total number of background pixels, bg is the background pixel, is the background confidence weight (when the old model has a low confidence that pixel i belongs to the background Increases when the old model has a high confidence that pixel i belongs to the background decreases), is the output probability distribution of the model at pixel i during the current t-th round of training, is the output probability distribution of the model at pixel i during the t-1th round of training, is the KL divergence calculation, and , C is the total number of categories in the current task, is the maximum value of the new model’s predicted probability for category c at pixel i, is the maximum value of the predicted probability of category c at pixel i by the old model; is the boundary region loss (aims to enhance the learning of the boundary region between the background and the new category), and , M is the total number of pixels in the boundary area, T is the number of pixels in the boundary area, is the reshaped feature of the pixel ii in the boundary area during the current t-th round of training, is the reshaped feature of pixel ii in the boundary area during the t-1th round of training; Calculate the feature differences between the new and old models in the boundary areas to quantify the performance differences between the new and old models in these key areas; Background loss can enhance the model's learning ability at the target boundary, reduce confusion between background and target, and ensure that the model can effectively distinguish subtle differences between target and background; at the same time, background loss can effectively reduce interference to boundary areas during new category learning, thereby significantly improving the segmentation accuracy of the model; in addition, it can also reduce the offset of background areas, ensuring that the model's performance in boundary areas is more stable and accurate, which helps to optimize the overall segmentation effect and enhance the model's adaptability to complex scenes; background loss can help the model identify background pixels that do not belong to old categories in incremental learning, thereby appropriately compensating them and incorporating them into new category learning. In this way, background loss not only effectively alleviates the background offset problem, but also enhances the model's flexibility and accuracy in processing new categories.

[0032] Cross Loss: In the current t-th round of training, the cross loss is calculated using the following formula : In the formula is a pseudo label, and its value is the fusion value of the segmentation prediction result of the iii-th pixel with classification category c in the t-1th round of training and the true value of the iii-th pixel in the current t-th round of training; is the segmentation prediction result of the iii-th pixel in the current t-th round of training with classification category c.

[0033] Total loss: The total loss function L is calculated using the following formula: In the formula is the embedding loss weight value; is the distillation loss weight value; is the background loss weight value; is the cross loss weight value; Finally, the total loss function L is used for training.

[0034] S5. Use the incremental semantic segmentation model obtained in step S4 to perform semantic segmentation of the actual image.

[0035] The following ablation experiment is used to illustrate the effectiveness of the incremental semantic segmentation method of the present invention.

[0036] Ablation experiments of each component: In order to verify the effectiveness of the proposed method, a series of ablation experiments are first conducted to evaluate the contribution of each module, including the global attention mechanism module, distiller and background loss. ,The experiment is conducted on two overlapping settings of Pascal VOC 15-1 and ADE20k100-50. The experimental results are shown in Table 1: Table 1 Schematic diagram of component ablation test results

[0037] The ablation experiment results of each component are shown in Table 1. Row (a) shows the performance of the baseline model used; row (b) introduces the global attention mechanism module into the baseline framework, and the performance of the initial category is improved in both settings, with the performance improved by 0.4%p compared with the baseline. This shows that the global attention mechanism module of the present invention can effectively capture global information, thereby enhancing the model's ability to retain knowledge of old categories and improving the recognition accuracy of old categories when new categories are introduced; row (c) introduces the distiller into the baseline framework, and the performance of the model is improved. Compared with the initial category, the performance improvement of the model for the new category is significantly greater, with the performance in steps 16-20 improved by 1.9%p, and the performance in steps 100-150 improved by 2.5%p. This proves that the distiller of the present invention can effectively filter out redundant information in the old model and only retain knowledge that is beneficial to the current task, thereby reducing the model's learning of new categories during the distillation process. interference, ensuring that the learning of new categories is more focused while avoiding the negative impact of old categories, thereby improving the overall performance of the model; (d) introduces background loss into the baseline framework, which also improves the overall performance of the model to a certain extent. The performance is improved by 0.6%p in the 15-1 setting and 0.5%p in the 100-50 setting. This shows that background loss effectively helps the model to better process background information, thereby improving the performance of the model in different settings; finally, the results of (h) show the effect of integrating all modules, which show significant improvement in all settings compared with the baseline model; among them, in the new category scenario of 15-1, the improvement is the largest, with a performance improvement of 4.3%p (16-20); the experimental results not only verify the effectiveness of each module, but also prove that they are highly compatible and can produce significant synergistic effects when used in combination, thereby improving the overall performance of the model.

[0038] Ablation experiment of background loss: A separate ablation experiment was conducted for background loss. The experiment was conducted on the 15-1 overlap setting of Pascal VOC. The experimental results are shown in Table 2: Table 2 Schematic diagram of background loss ablation experimental results

[0039] A quantitative analysis is conducted on different background compensation strategies to prove the effectiveness of the proposed background compensation strategy. Specifically, there are four background compensation strategies. Without means that background loss is not used, and only the global attention mechanism module and distiller are added to the baseline framework as a reference for this experiment. "Background confidence loss" and "boundary area loss" respectively represent the use of only background confidence loss and boundary area loss as background compensation strategies, and "background confidence loss + boundary area loss" represents the background loss proposed by the present invention. According to the experimental results shown in Table 2, when only background confidence loss and boundary area loss are used as background loss, the performance of the model is slightly improved. When background confidence loss is used as background loss, the model performance is improved by 0. 4%p, when the boundary area loss is used as the background loss, the model performance is improved by 0.7%p, and after combining the two, the overall performance of the model is greatly improved, with an overall improvement of 1.6%p; this is because simply weighting the background confidence or aligning the features cannot effectively separate the complex relationship between the background and the target class, and the interference between the background information and the target class is difficult to eliminate by a single strategy, resulting in limited performance improvement; Table 2 proves the effectiveness and superiority of the background loss proposed in the present invention, which combines the advantages of the background confidence loss and the boundary area loss, and through the collaborative work of multiple strategies, while reducing background interference, it more accurately captures the characteristics of the target class, thereby achieving more significant performance improvement in various settings.

[0040] Performance comparison on the Pascal VOC dataset: The experimental results on the Pascal VOC dataset under different incremental learning settings are shown in Tables 3 to 5: Table 3 Pascal VOC dataset experimental results Table 1

[0041] Table 4 Experimental results of Pascal VOC dataset Table 2

[0042] Table 5 Pascal VOC dataset experimental results Table 3

[0043] Among them, the EWC scheme was proposed by Kirkpatrick J in his 2017 paper "Overcoming catastrophicforgetting in neural networks"; the LIT scheme was proposed by Michieli U in his 2019 paper "Incremental learning techniques for semantic segmentation"; the MiB scheme was proposed by Cermelli F in his 2020 paper "Modeling the background for incremental learningin semantic segmentation"; the SDR scheme was proposed by Michieli U in his 2021 paper "Continual semantic segmentation via repulsion-attraction of sparse anddisentangled latent representations"; the PLOP scheme was proposed by Douillard A in his 2021 paper "Plop: Learning without forgetting for continual semanticsegmentation"; the RECALL scheme was proposed by Maracani A in his 2021 paper "Recall: Replay-based continual learning in semantic segmentation"; the REMIND scheme was proposed by Phan The scheme proposed by MH in the paper "Class similarity weighted knowledge distillation for continual semantic segmentation" in 2022; the RCLL scheme is the scheme proposed by Zhang CB in the paper "Representation compensation networks for continual semantic segmentation" in 2022; the SPPA scheme is the scheme proposed by Lin Z in the paper "Continual semantic segmentation via structure preserving and projected feature alignment" in 2022;The RBC scheme is proposed by Zhao H in the paper "Rbc: Rectifying the biased context incontinual semantic segmentation" in 2022; the INC scheme is proposed by Shang C in the paper "Incrementer: Transformer for class-incremental semantic segmentation with knowledge distillation focusing on old class" in 2023; the MBS scheme is proposed by Park G in the paper "Mitigating Background Shift in Class-Incremental Semantic Segmentation" in 2025; Joint means that all classes are trained simultaneously; In the experiment, the proposed method and the existing scheme were tested in three different experimental settings: VOC-19-1, VOC-15-5 and VOC-15-1. As can be seen from the table, the proposed method is significantly better than the best method in the existing schemes in all incremental scenarios. In particular, the performance of the model has been significantly improved in the multi-step scenario. In the multi-step scenario, the introduced global information better helps the model to maintain the understanding of global knowledge in each incremental step. Compared with the recent MBS method, the proposed method exceeds the best model in the previous existing scheme by 2.0% p (poi nts) (disjoint) and 0.9%p (overlap). In addition, in the Disjoint setting of 15-1, the effect is greatly improved compared with the best method in the existing scheme, which is 5.3%p (16-20); in the experimental setting of 15-5, the mIoU of the method of the present invention is improved by 1.5%p (disjoint) and 0.6%p (overlap) compared with the best method in the existing scheme; in the experimental setting of 19-1, the overall performance of the method of the present invention is improved by 0.5%p (overlap) compared with the best method in the existing scheme, which is close to the experimental results in the Joint (indicating that all classes are trained simultaneously).

[0044] Performance comparison on the ADE20k dataset: The experimental results on the ADE20k dataset under different incremental learning settings are shown in Tables 6 and 7: Table 6 Experimental results of ADE20k dataset Table 1

[0045] Table 7 Experimental results of ADE20k dataset Table 2

[0046] From Tables 6 and 7, we can see that the proposed method is tested under four different experimental settings: ADE-100-50, ADE-50-50, ADE-100-10 and ADE-100-5. The results show that the proposed method improves the average performance by 0.95%p over the best method in the existing schemes in all experimental settings. In the 100-50 scenario, it improves the performance by 0.8%p over the most advanced method. In particular, in the incremental cluster scenario, the performance of the model is significantly improved, surpassing the best model in the previous existing schemes. 2.0%p (101-150); in the 50-50 scenario, it is 1.3%p higher than the most advanced method; from the quantitative results, it can be seen that with the increase of task steps, the method of the present invention shows the advantage of alleviating catastrophic forgetting and background shift problems when dealing with multi-step incremental learning; In general, the experimental results of the two datasets verify the stability and plasticity of the method of the present invention in incremental semantic segmentation. Through tests on different datasets, the model of the present invention shows strong generalization ability and adaptability, and can effectively cope with the challenges brought by category incremental semantic segmentation.

[0047] Performance comparison of multi-step scenarios in the Pascal VOC dataset: The experimental results of multi-step scenarios 5-3 (6 steps) and 10-1 (11 steps) in the Pascal VOC dataset are shown in Table 8: Table 8 Pascal VOC dataset experimental results

[0048] It can be seen from Table 8 that compared with the best method in the existing scheme, the overall performance of the method of the present invention under the 10-1 setting is improved by 1.62%p. The performance on the 5-3 setting was improved by 1.3%p, The performance on the new class is improved by 1.4%p(6-20); it can be seen from the data in the table that compared with the retention of old class knowledge, the improvement of the method of the present invention in new class learning is more significant, and this improvement is mainly due to the introduction of global labeling; global labeling can effectively transmit cross-step context information in the new class learning process, significantly reduce the degree to which the new task is affected by the previous task, and avoid the problem of knowledge forgetting due to information interference; at the same time, the adaptive multi-scale distillation technology and background compensation strategy accurately reduce the interference of old class knowledge and background noise on new class learning through dynamic adjustment, and effectively improve the accuracy and stability of new class learning; this shows that the method of the present invention not only promotes the accurate learning of new class knowledge, but also improves the overall robustness of the model to a certain extent.

[0049] Visual comparison of experimental results: In order to better understand the performance of the method of the present invention under different settings, Figure 2 The prediction results of the baseline and the proposed method in the Pascal VOC 15-1 overlapping setting are visualized in the figure. The qualitative visualization results can intuitively show the performance differences of the models when learning new classes, clearly present the accuracy and stability of the models during the learning process, and reveal their adaptability in handling different tasks.

[0050] according to Figure 2 From the qualitative results shown, it can be clearly seen that compared with the results of the baseline, the method of the present invention shows a better effect in preserving the old classes; even after multiple learning steps, the present invention still maintains a high accuracy in the segmentation of the old classes and will not be disturbed by the learning of new classes, especially in the details, such as the edges and protrusions of objects, the segmentation results are still accurate; in contrast, after multiple steps, the segmentation accuracy of the old classes of the baseline method is significantly reduced, and classification errors or blurred boundaries are more likely to occur at the boundaries of objects and complex areas, resulting in a significant reduction in the recognition effect of the old classes; not only that, compared with the baseline method, the method of the present invention has achieved a more accurate effect in distinguishing background classes, especially after learning multiple new classes, the details and edges of foreground objects are better preserved; traditional segmentation methods often cause confusion between background classes and foreground classes when facing background offsets, and even produce a large number of mis-segmentations, while the present invention successfully alleviates this problem through a background compensation strategy, thereby ensuring the recognition accuracy of the foreground class.

[0051] The above qualitative results show that the method of the present invention has obvious advantages over the traditional method in the segmentation of background classes. No matter in segmentation accuracy, detail presentation or overall effect, the method of the present invention can provide more accurate and detailed segmentation results; especially in the segmentation results of the final step, it can handle the boundary between the new class and the old class well to ensure that the segmentation effects of the two are balanced and accurate; through comparative analysis, it can be seen that the present invention not only effectively avoids the problem of forgetting the old class, but also retains the accuracy of the old class while learning the new class, which fully reflects its stability and robustness in the long-term learning process; these qualitative results further prove the superiority of the present invention in the category incremental semantic segmentation task, and can provide more accurate segmentation performance when facing multi-step tasks and complex backgrounds.

[0052] like Figure 3 The figure shows a method flow chart of the method for semantic segmentation of autonomous driving images of the present invention: The method for semantic segmentation of autonomous driving images including the incremental semantic segmentation method disclosed in the present invention comprises the following steps: A. Obtain the autonomous driving image to be segmented; B. Using the incremental semantic segmentation method, semantic segmentation is performed on the autonomous driving image obtained in step A; C. Complete the semantic segmentation of the autonomous driving image based on the semantic segmentation result obtained in step B. Based on the semantic segmentation result, data support can be provided for the control process of the autonomous driving vehicle; In addition, the incremental semantic segmentation method provided by the present invention is also suitable for application scenarios such as medical image processing, remote sensing image processing, robot vision, video surveillance, industrial inspection and automation, intelligent transportation systems, and environmental monitoring.

Claims

1. An incremental semantic segmentation method, characterized in that The steps include: S1. Obtain the target image dataset; S2. Preprocessing the target image dataset obtained in step S1 to construct a training dataset; S3. Based on the global attention mechanism, Transformer mechanism and adaptive multi-scale distillation mechanism, a primary model of incremental semantic segmentation including encoder, decoder and distiller is constructed; The encoder is used to encode the input image data to extract the feature information of the input image; by replacing the attention module in the Transformer mechanism with a global attention mechanism module, the decoder decodes the feature information extracted by the encoder to obtain global information; the distiller is used to perform feature distillation and fusion operations on the output of the decoder to ultimately achieve semantic segmentation; S4. Using the training dataset constructed in step S2, combined with embedding loss, distillation loss, background loss and cross loss, the incremental semantic segmentation primary model constructed in step S3 is trained to obtain an incremental semantic segmentation model; S5. Use the incremental semantic segmentation model obtained in step S4 to perform semantic segmentation of the actual image.

2. The incremental semantic segmentation method according to claim 1, characterized in that The step S3 comprises the following steps: Build an encoder based on the Transformer mechanism to extract feature information of the input image; In the decoder, the background class label, the learned new label and the global label are used as features and input into the Transformer decoder for decoding to obtain global information; wherein, the attention module in the Transformer decoder is replaced by a global attention mechanism module; A distiller is constructed based on an adaptive multi-scale distillation mechanism to perform feature distillation and fusion operations on the output of the decoder to ultimately achieve semantic segmentation.

3. The incremental semantic segmentation method according to claim 2, characterized in that The encoder specifically includes the following contents: The constructed encoder includes an embedding layer, a linear projection layer, and a Transformer encoder layer; The embedding layer is used to divide the input image into several visual blocks and input the visual blocks into the linear projection layer; The linear projection layer is used to linearly project the input visual block to map the input visual block into a high-dimensional representation vector, and input the obtained data information into the Transformer encoder layer; The Transformer encoder layer is used to extract the feature information and dependencies of the input image based on the input data information to obtain the corresponding encoder result.

4. The incremental semantic segmentation method according to claim 3, characterized in that The decoder specifically includes the following contents: The constructed decoder includes a feature construction layer and a Transformer decoder layer; The feature construction layer is used to construct the feature information required by the Transformer decoder layer; The features constructed in the feature construction layer include the target class label to be learned and the global label to be learned , and the constructed , Together with the encoder result, it is input into the Transformer decoder layer; among them, the target class label Expressed as ,in is the background class marker, is the label of the new category learned in the current t-th round of training; global label Used to aggregate and represent the global knowledge of the current task. Before the first training, the global mark Initialize to the set value , during the current t-th round of training, the global mark The value of is the global label value obtained from the t-1th round of training ; The Transformer decoder layer is used to and , decode the encoder result to obtain global information; in the Transformer decoder layer, replace the attention module in the Transformer decoder with a global attention mechanism module to obtain global information; The output of the decoder includes decoder features and background class marker features, where the decoder features are the features obtained after the encoder results are processed by the decoder, and the background class marker features are Features obtained after being processed by the decoder.

5. The incremental semantic segmentation method according to claim 4, characterized in that The global attention mechanism module specifically includes the following contents: In the current t round of training, the encoder result The label of the new category learned in the current t round of training Perform stitching and obtain the first stitching feature ;Will Perform normalization to obtain the normalized first splicing feature , and As a query The value of The value of the global tag of the current t-th round of training Perform splicing and obtain the first splicing mark , and as a key Value and value The value of The processing of the global attention mechanism module is expressed as: In the formula is the output of the global attention mechanism module; is the weight matrix of the output layer; is the processing function of the multi-layer perceptron; It is the processing function of two mask normalization layers connected in series; is the bias term of the output layer; is an intermediate feature, and , is the softmax function, d is the embedding size, and h is the number of attention heads.

6. The incremental semantic segmentation method according to claim 5, characterized in that The distiller specifically comprises the following contents: The distiller includes a reshaping layer, a similarity calculation layer, an upsampling layer, and a normalization layer; In the current t round of training, the decoder features After reshaping through the reshape layer, the reshape feature is obtained ;Will Background class marker features The cosine similarity is calculated through the similarity calculation layer; The obtained cosine similarity is upsampled through the upsampling layer and then processed through the normalization layer to obtain the final segmentation prediction result; Among them, the normalization layer uses the softmax function for normalization.

7. The incremental semantic segmentation method according to claim 6, characterized in that The training described in step S4 includes the following steps: In the current t-th round of training, the embedding loss is calculated based on the cosine similarity between the current background class marker feature and the background class marker feature of the previous round; In the current t-th round of training, the distillation loss is calculated based on the cosine similarity between the feature segmentation results of the current reshaped features at different scales and the feature segmentation results of the previous round of reshaped features at different scales; In the current t-th round of training, the background loss is calculated based on the KL divergence between the current background output and the background output of the previous round, and the difference between the current boundary area and the boundary area of ​​the previous round; In the current t-th round of training, the cross loss is calculated based on the current segmentation prediction result, the current true value and the segmentation prediction result of the previous round; Finally, the obtained embedding loss, distillation loss, background loss and cross loss are weighted summed as the total loss function of the current t-th round of training; The obtained total loss function is used for training.

8. The incremental semantic segmentation method according to claim 7, characterized in that The training described in step S4 specifically includes the following steps: Embedding loss: In the current t-th round of training, the embedding loss is calculated using the following formula : In the formula is the total number of old categories in the current t-th round of training; Mark the features of the current i-th background class; Mark the features of the i-th background class in the previous round; Calculate the cosine similarity function; Distillation loss: Reshape features Segmentation is performed at the set scale s; then at each segmentation scale, the segmented features are averaged pooled in the height and width directions, expressed as: In the formula is the height-direction pooling embedding result at scale s; H is the reshaped feature Height; for The value at height h, width w and channel c; is the width-wise pooling embedding result at scale s; W is the reshaped feature Width; At each scale, the obtained pooling embedding results in the height direction and the pooling embedding results in the width direction are concatenated to obtain the pooling embedding results at the corresponding scale; Concatenate the pooling embedding results at each scale to obtain a high-dimensional feature representation ; In the current t-th round of training, the distillation loss is calculated using the following formula : In the formula is the reliability mapping result, and its value is the cosine similarity value between the current pseudo label and the segmentation prediction result of the previous round; The value of the current high-dimensional feature representation at a height of h and a width of w; The high-dimensional feature representation of the previous round has a value at a height of h and a width of w; is the Euclidean distance; Background loss: In the current t-th round of training, the background loss is calculated using the following formula : In the formula is the first weight value; is the second weight value; is the background confidence loss, and , N is the total number of background pixels, bg is the background pixel, is the background confidence weight, is the output probability distribution of the model at pixel i during the current t-th round of training, is the output probability distribution of the model at pixel i during the t-1th round of training, is the KL divergence calculation, and , C is the total number of categories in the current task, is the maximum value of the new model’s predicted probability for category c at pixel i, is the maximum value of the predicted probability of category c at pixel i by the old model; is the boundary area loss, and , M is the total number of pixels in the boundary area, T is the number of pixels in the boundary area, is the reshaped feature of the pixel ii in the boundary area during the current t-th round of training, is the reshaped feature of pixel ii in the boundary area during the t-1th round of training; Cross Loss: In the current t-th round of training, the cross loss is calculated using the following formula : In the formula is a pseudo label, and its value is the fusion value of the segmentation prediction result of the iii-th pixel with classification category c in the t-1th round of training and the true value of the iii-th pixel in the current t-th round of training; is the segmentation prediction result of the iii-th pixel in the current t-th round of training with classification category c; The total loss function L is calculated using the following formula: In the formula is the embedding loss weight value; is the distillation loss weight value; is the background loss weight value; is the cross loss weight value; Finally, the total loss function L is used for training.

9. A method for semantic segmentation of autonomous driving images comprising the incremental semantic segmentation method according to any one of claims 1 to 8, characterized in that The steps include: A. Obtain the autonomous driving image to be segmented; B. Using the incremental semantic segmentation method, semantic segmentation is performed on the autonomous driving image obtained in step A; C. Based on the semantic segmentation results obtained in step B, complete the semantic segmentation of the autonomous driving image.

Images