Semantic segmentation model training method and system and electronic equipment

Through the collaborative training of the dual-branch network and the activation modulation module, the semantic fuzzy problem of the semantic segmentation model when the color of the front background image is similar is solved, the segmentation performance and pseudo-label quality are improved, and the generated segmentation diagram is more complete and accurate.

CN118941915APending Publication Date: 2024-11-12GUIZHOU MINZU UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410912456.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-09
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

When the existing semantic segmentation model has similar colors in the previous background image, the class activation map is only partially activated, resulting in semantic blurring, making it difficult to effectively perform semantic segmentation.

Method used

Design a dual-branch network, combining Resnet50 and Vit as feature extraction networks, generate target class activation maps through collaborative training, and use activation modulation module and semantic consistent regularization loss to export pseudo-labels for iterative training to improve the performance of semantic segmentation model .

Benefits of technology

It effectively alleviates the semantic fuzziness between the front background caused by partial activation of the class activation graph, improves the segmentation performance and pseudo-label quality of the semantic segmentation model, and the generated segmentation graph is more complete and accurate.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118941915A_ABST
    Figure CN118941915A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of weak supervision semantic segmentation of images, and discloses a semantic segmentation model training method and system and electronic equipment, and the method comprises the steps: obtaining a training data set; inputting the training data set into a semantic segmentation model; performing feature extraction on the training data set by the double-branch network, and determining a first-class activation graph and a second-class activation graph; performing cooperative training on the double-branch network through the first type of activation graph and the second type of activation graph, and determining a target double-branch network; and outputting a target class activation graph through the target double-branch network, exporting a pseudo tag from the target class activation graph, inputting the pseudo tag into a segmentation decoder, carrying out iterative training on the semantic segmentation model, and determining a target semantic segmentation model. According to the method, the problem of partial activation of the class activation graph is relieved through the multi-branch network by designing the double-branch network of different network architectures, and the problem of semantic fuzziness between the foreground and the background caused by partial activation of the class activation graph is solved.
Need to check novelty before this filing date? Find Prior Art

Description

Background Art

[0002] Semantic segmentation is an important task in computer vision, which aims to classify each pixel in an image. It has been widely used in application scenarios such as autonomous driving, medical imaging analysis, and remote sensing image interpretation.

[0003] However, the pixel-level annotation required for semantic segmentation is labor-intensive and time-consuming. In order to alleviate the expensive cost of annotating pixel-level labels for fully supervised semantic segmentation, researchers began to study the use of four relatively weak labels: image-level, point, scribble, and bounding box to complete the semantic segmentation task. Among them, image-level annotation is the weakest supervision, which only indicates whether the object of interest exists in the image. Image-level labels are widely present in large-scale datasets such as ImageNet. Due to their accessibility, more and more scholars are focusing on studying semantic segmentation based on image-level labels.

[0004] The introduction of Class Activation Map (CAM) has accelerated the research progress of weakly supervised semantic segmentation. CAM is generated by a classification network and usually only responds to the most discriminative features. If the colors of the foreground and background images in an image are similar, the CAM will be only partially activated, resulting in semantic ambiguity between the foreground and background. Summary of the invention

[0005] In order to overcome the problem that CAM is only partially activated when the colors of the foreground and background images are similar in existing images, resulting in semantic ambiguity between the foreground and background, the present invention provides a training method, system and electronic device for a semantic segmentation model.

[0006] In a first aspect, in order to solve the above technical problems, the present invention provides a method for training a semantic segmentation model, comprising:

[0007] Obtain a training dataset, wherein the training dataset includes at least one image and at least one image-level label;

[0008] Inputting the training data set into a semantic segmentation model, the semantic segmentation model comprising a dual-branch network and a segmentation decoder; wherein the dual-branch network comprises a first feature extraction network with Resnet50 as a branch and a second feature extraction network with Vit as a branch;

[0009] The first feature extraction network extracts features from the image and the image-level label to determine a first type of activation map, and the second feature extraction network extracts features from the image and the image-level label to determine a second type of activation map;

[0010] Co-training the dual-branch network by using the first type of activation map and the second type of activation map to determine a target dual-branch network;

[0011] The target dual-branch network outputs a target class activation map, and a pseudo label is derived from the target class activation map and input into a segmentation decoder to iteratively train the semantic segmentation model to determine a target semantic segmentation model.

[0012] In a second aspect, the present invention provides a training system for a semantic segmentation model, comprising:

[0013] A training data set acquisition module, used to acquire a training data set, wherein the training data set includes at least one image and at least one image-level sticky note;

[0014] A semantic segmentation model module, used for inputting the training data set into a semantic segmentation model, wherein the semantic segmentation model comprises a dual-branch network and a segmentation decoder; wherein the dual-branch network comprises a first feature extraction network with Resnet50 as a branch and a second feature extraction network with Vit as a branch;

[0015] a class activation map determination module, configured for the first feature extraction network to perform feature extraction on the image and the image-level label to determine a first class activation map, and for the second feature extraction network to perform feature extraction on the image and the image-level label to determine a second class activation map;

[0016] a target dual-branch network determination module, configured to perform collaborative training on the dual-branch network through the first type of activation map and the second type of activation map to determine a target dual-branch network;

[0017] The target semantic segmentation model determination module is used to output a target class activation map through the target dual-branch network, and derive a pseudo label from the target class activation map to input into a segmentation decoder to iteratively train the semantic segmentation model to determine the target semantic segmentation model.

[0018] In a third aspect, the present invention provides a computing device comprising a memory, a processor, and a program stored in the memory and running on the processor, wherein when the processor executes the program, the steps of a method for training a semantic segmentation model as described above are implemented.

[0019] The beneficial effect of the present invention is that by designing a dual-branch network with different network architectures and co-training the first feature extraction network and the second feature extraction network, the problem of partial activation of the class activation map is alleviated through a multi-branch network, and the semantic ambiguity between the foreground and the background caused by the partial activation of the class activation map is solved. Finally, the target dual-branch network obtained by the collaborative training outputs the target class activation map, and the pseudo-label is exported to the segmentation decoder for iterative training of the semantic segmentation model, the target semantic segmentation model is determined, and the training of the semantic segmentation model is completed. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0021] Figure 1 A flowchart of a method for training a semantic segmentation model according to an embodiment of the present invention;

[0022] Figure 2 It is a structural diagram of a channel modulation module;

[0023] Figure 3 It is a structural schematic diagram of a spatial modulation module;

[0024] Figure 4 It is a structural diagram of a double-branch network;

[0025] Figure 5 To analyze the limitations of each component on the PASCAL VOC 2012 dataset validation set;

[0026] Figure 6 Comparison results between the baseline and the dual-branch architecture baseline;

[0027] Figure 7 Analyze results for different semantic regularization methods;

[0028] Figure 8 Set the highest background threshold for the cosine decay strategy;

[0029] Fig. 9 For model complexity analysis;

[0030] Fig.10 Analyze the results of different combinations of activation modulation modules;

[0031] Fig.11 Input and ground-truth images for visual comparison of class activation maps;

[0032] Fig.12 CAM generated for the ToCo method;

[0033] Fig.13 CAM generated for a two-branch network;

[0034] Fig.14 Pseudo-label performance comparison results for the PASCAL VOC 2012 dataset;

[0035] Fig.15 Comparison results of segmentation performance on the PASCAL VOC 2012 dataset;

[0036] Fig.16 Input map for visual comparison of PASCAL VOC 2012;

[0037] Fig.17 To visualize and compare the actual segmentation results of PASCAL VOC 2012;

[0038] Fig.18 is the AFA segmentation result;

[0039] Fig.19 is the segmentation result of ToCo;

[0040] Fig. 20 is the segmentation result of the target semantic segmentation model;

[0041] Fig.21 Comparison results of segmentation performance on the MS COCO2014 dataset;

[0042] Fig. 22 The input map of the MS COCO 2014 dataset;

[0043] Fig.23 The actual segmentation result of the input map of the MS COCO 2014 dataset;

[0044] Fig.24 Segmentation results generated for ToCo;

[0045] Fig.25 is the segmentation result of the target semantic segmentation model;

[0046] Fig.26 A structural diagram of a semantic segmentation model training system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0047] The following examples are provided to further explain and supplement the present invention and do not constitute any limitation to the present invention.

[0048] The following describes a method, system, and electronic device for training a semantic segmentation model according to an embodiment of the present invention in conjunction with the accompanying drawings.

[0049] like Figure 1 As shown, the present invention provides a training method for a semantic segmentation model, comprising:

[0050] S1. Obtain a training data set, where the training data set includes at least one image and at least one image-level label;

[0051] S2. Input the training data set into a semantic segmentation model, wherein the semantic segmentation model includes a dual-branch network and a segmentation decoder; wherein the dual-branch network includes a first feature extraction network with Resnet50 as a branch and a second feature extraction network with Vit as a branch;

[0052] S3, the first feature extraction network extracts features from the image and the image-level label to determine a first type of activation map, and the second feature extraction network extracts features from the image and the image-level label to determine a second type of activation map;

[0053] S4, collaboratively training the dual-branch network through the first type of activation map and the second type of activation map to determine a target dual-branch network;

[0054] S5. Output a target class activation map through the target dual-branch network, and derive a pseudo label from the target class activation map and input it into a segmentation decoder to iteratively train the semantic segmentation model to determine a target semantic segmentation model.

[0055] In this embodiment, a dual-branch network with different network architectures is designed, and the first feature extraction network and the second feature extraction network are trained collaboratively, and then the problem of partial activation of the class activation map is alleviated through a multi-branch network, solving the semantic ambiguity between the foreground and the background caused by partial activation of the class activation map. Finally, the target dual-branch network obtained through collaborative training outputs the target class activation map, and derives the pseudo-label input into the segmentation decoder to iteratively train the semantic segmentation model, determine the target semantic segmentation model, and complete the training of the semantic segmentation model.

[0056] Optionally, the first feature extraction network includes four convolution branches and three activation modulation modules; wherein an activation modulation module is inserted between two adjacent convolution branches;

[0057] The first feature extraction network extracts features from the image and the image-level label to determine a first type of activation map, including:

[0058] S11, starting from the first stage of the convolution branch until the third stage of the convolution branch, repeatedly execute S111-S112 to determine the output target adjustment feature map;

[0059] S111, inputting the image and the image-level label into the convolution branch for feature extraction to determine an initial feature map;

[0060] S112, inputting the initial feature map into an activation modulation module, adjusting the activation distribution of the initial feature map from the channel space dimension using a modulation function, and determining a target modulation feature map;

[0061] S12, input the output target modulation feature map into the fourth stage of the convolution branch for feature extraction, and determine the first type of activation map.

[0062] In this embodiment, Resnet50 is used as the convolution branch, and the activation modulation module is embedded in the 1st stage, 2nd stage, and 3rd stage of the convolution branch to adjust the activation distribution of the feature map. The traditional class activation map is generated by training the classification network with classification loss. Different features are frequently activated during the training process, causing the model to find the most discriminative features of the target object to complete the classification task. Specifically, an activation modulation module is designed to be embedded in the convolution branch. The module adjusts the activation distribution of the feature map from the channel and spatial dimensions respectively, forcing the model to focus on activating pixels with more intermediate scores and suppressing the activation of the most sensitive and least sensitive pixels.

[0063] Optionally, the activation modulation module includes a channel modulation module and a space modulation module;

[0064] The initial feature map is input into the activation modulation module, and the activation distribution of the initial feature map is adjusted from the channel space dimension using a modulation function to determine a target modulation feature map, including:

[0065] Inputting the initial feature map into a channel modulation module, adjusting the feature distribution of the initial feature map by a Gaussian modulation function, and determining a first modulation feature map;

[0066] The first modulation feature map is input into a spatial modulation module, the activation distribution of the first modulation feature map is adjusted by a Gaussian modulation function, a second modulation feature map is determined, and the first modulation feature map and the second modulation feature map are multiplied pixel by pixel to fuse features to determine a target modulation feature map.

[0067] In this embodiment, a Gaussian modulation function is applied in the activation modulation module to redistribute the activation distribution of the feature map, so that the model focuses on features with intermediate activation values ​​and suppresses the activation of the most sensitive and least sensitive pixels, where the most sensitive pixels correspond to the most discriminative features in the CAM and the least sensitive pixels are considered to be the background. The formula of the Gaussian modulation function is as follows:

[0068]

[0069] in, Represents the activation value of the feature map, V A represents the activation value of the modulated feature map, g(.) represents the Gaussian modulation function, which obeys the standard normal distribution. The mean μ and variance σ of the Gaussian distribution are respectively calculate:

[0070]

[0071] Among them, M represents all activation values ​​in the feature map, and the distribution of activation values ​​is remapped according to the settings of μ and σ.

[0072] Alternatively, if Figure 2 As shown, the initial feature map is input into a channel modulation module, and the feature distribution of the initial feature map is adjusted by a Gaussian modulation function to determine a first modulation feature map, including:

[0073] S21. Input the initial feature map into the channel modulation module, perform global average pooling on the initial feature map, and determine a global vector.

[0074] Among them, the channel activation modulation module is modeled through the pooling layer and the convolution layer, and the initial feature map of the residual block is Perform global average pooling to obtain the global vector

[0075] S22, calculating the mean and variance of the global vector, and inputting the mean and variance into a Gaussian modulation function to adjust the characteristic distribution to determine a third modulation characteristic graph after modulation.

[0076] S23, extending the characteristic distribution of the third modulation characteristic graph to the same shape and size as the initial characteristic graph, and determining a fourth modulation characteristic graph.

[0077] Among them, the operation of S21-S23 can be expressed as the following formula:

[0078]

[0079] Among them, R(.) represents the feature distribution expansion, g(.) represents the Gaussian modulation function, conv(.) represents the convolution operation, and avg(.) represents the global average pooling layer.

[0080] S24, performing pixel-by-pixel multiplication on the fourth modulation characteristic map and the initial characteristic map to determine a first modulation characteristic map.

[0081] Among them, the operation of S24 can be expressed as the following formula:

[0082]

[0083] Among them, A i represents the first modulation characteristic diagram, represents the fourth modulation characteristic diagram, F i represents the initial feature map, Represents pixel-by-pixel multiplication.

[0084] Optionally, the spatial activation modulation module and the channel activation modulation module perform a cascade operation. In other words, the output of the channel activation modulation module is used as the input of the spatial activation modulation module. Based on this, Figure 3As shown, the first modulation feature map is input into a spatial modulation module, and the activation distribution of the first modulation feature map is adjusted by a Gaussian modulation function to determine a second modulation feature map, including:

[0085] S31. Input the first modulation feature map into the spatial modulation module, perform max operation and mean operation on the first modulation feature map in the channel dimension, and splice along the channel dimension to determine the splicing tensor.

[0086] S32. Input the concatenated tensor into a convolution layer with a convolution kernel size of 7×7 to determine a feature tensor.

[0087] S33, inputting the feature tensor into a Gaussian modulation function, adjusting the activation distribution of the feature tensor, and determining a second modulation feature map.

[0088] Among them, the operation of S31-S33 can be expressed as the following formula:

[0089]

[0090] in, Represents the second modulation feature map, conv 7×7 It represents a convolution layer with a convolution kernel size of 7X7, cat(.) represents concatenation along the channel dimension, max(.) represents the max operation, i.e. taking the maximum value, and mean(.) represents the mean operation, i.e. finding the average value.

[0091] Optionally, VisionTransformer (Vit), as the first Transformer model designed specifically for computer vision tasks, has recently achieved performance breakthroughs in multiple visual tasks. Vit has achieved certain research results in the weakly supervised semantic segmentation neighborhood by virtue of its powerful remote context modeling capability of the multi-head attention mechanism. Although Vit has powerful remote modeling capabilities, it has an over-smoothing problem. Therefore, following the ToCo method, an auxiliary classification layer is added to the tenth Transformer module, an auxiliary CAM is generated from the auxiliary classification layer, and the final CAM of the Vit branch is exported for pseudo-label supervised training. Based on this, the second feature extraction network includes a classifier and an auxiliary classifier added to the tenth Transformer module in Vit; the second feature extraction network extracts features from the image and image-level labels, and determines the second type of activation map, including:

[0092] Extracting features from the image and the image-level label using a classifier to determine a third type of activation map;

[0093] Extracting features from the image and the image-level label using an auxiliary classifier to determine a fourth type of activation map;

[0094] The second type of activation map includes the third type of activation map and the fourth type of activation map.

[0095] Optionally, in order to make the two branches generate CAMs with consistent activation, a semantic consistency regularization loss is designed. Based on this, the two-branch network is collaboratively trained through the first type of activation map and the second type of activation map to determine a target two-branch network, including:

[0096] The semantic consistency regularization loss is calculated according to the first type of activation map, the third type of activation map, and the fourth type of activation map to determine a first loss value, wherein the formula for calculating the semantic consistency regularization loss is as follows:

[0097]

[0098] in, represents the first loss value, M CNN represents the first type of activation map, M Vit represents the third type of activation map, M Aux represents the fourth type of activation map, λ1 and λ2 represent the preset weights, and ||·||1 represents regularization;

[0099] The two-branch network is collaboratively trained using the semantic regularization loss to determine a target two-branch network.

[0100] In this embodiment, Figure 4 FIG. 1 is a schematic diagram of the structure of a dual-branch network. Another embodiment is used to illustrate the training process of the dual-branch network, as follows:

[0101] First, the image and image-level label are input into the dual-branch network consisting of the resnet50 branch and the vit branch. The resnet50 branch outputs the convolution branch CAM, the auxiliary classifier outputs the auxiliary CAM, and the classifier outputs the final CAM. The classification loss of the dual-branch network is a linear combination of the convolution branch classification loss, the Vit branch auxiliary classification layer loss, and the final classification loss of the Vit branch:

[0102]

[0103] in, represents the classification loss of the two-branch network, z CNN represents the convolution branch CAM, z Aux Indicates auxiliary CAM, Z Vit Represents the final CAM, y c Represents image-level labels.

[0104] In addition, in order to prevent the over-smoothing problem in Vit from weakening the segmentation performance, following the loss function design of ToCo, patchtoken contrast loss and classtoken contrast loss are introduced respectively, as follows:

[0105]

[0106] in, represents the patch token comparison loss, F i Represents the global features of the image, F j Represents the local features of the image, cos sim(.) represents the calculation of cosine similarity, N + represents positive samples, N - represents negative samples, y i represents the i-th training data set, y j represents the jth image in the i-th training dataset.

[0107]

[0108] in, represents the classtoken contrast loss, τ represents the temperature factor, ε is a small constant, and q + ∈Q + ,q - ∈Q - , Q + Represents the set of class tokens of the local image cropped from the uncertain area, Q - A collection of class tokens representing a partial image cropped from the background area.

[0109] The consistency loss between the convolution branch CAM and the auxiliary CAM, as well as the consistency loss between the convolution branch CAM and the final CAM are calculated, and the semantic consistency regularization loss is obtained by linear weighted summation. The two-branch network is collaboratively trained through the semantic consistency regularization loss.

[0110] Optionally, deriving a pseudo label from the target class activation map and inputting it into a segmentation decoder to iteratively train the semantic segmentation model includes:

[0111] Each time the semantic segmentation model is iteratively trained, the background threshold is updated by a cosine annealing decay dynamic threshold adjustment strategy; wherein the formula of the cosine annealing decay dynamic threshold adjustment strategy is as follows:

[0112]

[0113] Where t represents the current iteration, T represents the total number of iteration training, th(t) represents the background threshold corresponding to the current iteration, and th(0) represents the set initial threshold;

[0114] Derive pseudo labels from the target class activation map according to the background threshold corresponding to the current iteration;

[0115] The pseudo label corresponding to the current iteration and the target class activation map are input into the segmentation decoder for current iteration training.

[0116] In this embodiment, in order to derive reliable pseudo-label supervision training semantic segmentation network from CAM, two background thresholds (β h and β l ) The pixels in the CAM are divided into foreground, background and uncertain area. In order to ensure the absolute confidence of the foreground, a very high threshold β needs to be set h This undoubtedly loses some confident foreground pixels with low scores. In other words, these pixels are not learned in the segmentation training, resulting in semantic ambiguity between the foreground and background or different objects. The mis-segmentation problem in scenes with similar colors of foreground and background images is attributed to the semantic object ambiguity problem, which is essentially caused by the lack of learning of the pixel in the model during the training process. In order to solve this problem, based on the assumption that the highest background threshold follows cosine annealing decay, more confident foreground pixels can be used as pseudo-labels to supervise the training segmentation head during the training process, thereby producing better segmentation results.

[0117] In this embodiment, during the model training process, β is adaptively determined. h , allowing more confident foreground pixels to participate in subsequent segmentation training and generate dense segmentation maps. Inspired by the cosine annealing learning rate decay, a dynamic threshold adjustment strategy of cosine annealing decay is designed. This strategy follows the cosine function image to adjust the highest threshold during the training process. A higher threshold is set at the beginning of model training, which helps the model to jump out of the local optimal solution. This strategy is used to adjust the threshold in each iterative training process to find the global optimal threshold.

[0118] In addition, the total loss of the entire semantic segmentation model is defined as:

[0119]

[0120] in, represents the total loss, w1, w2, w3 represent the three preset hyperparameters used to readjust the weights of the loss terms, represents the cross entropy loss of the segmentation decoder.

[0121] In this embodiment, a test experiment is designed, which is as follows:

[0122] This embodiment is experimented on two weakly supervised semantic segmentation standard datasets (PASCAL VOC 2012 and MS COCO 2014). The PASCAL VOC 2012 dataset consists of 20 semantic categories and 1 background category, wherein the training set, validation set, and test set contain 1464, 1449, and 1456 images, respectively. Following the practice of most works, the model is trained using 10582 images of the enhanced training set. Since the PASCAL VOC 2012 dataset does not disclose the true value of the test set, this paper also submits the test set results to the official evaluation server to evaluate its performance. The MS COCO 2014 dataset consists of 80 semantic categories and 1 background category. Following the practice, images without target categories are removed from the dataset, and the remaining 82081 images are used as the training set and 40137 images are used as the validation set.

[0123] In order to evaluate and compare the performance of various methods, the mean intersection over union (MIou) is used as the evaluation index, which is expressed as follows:

[0124]

[0125] Among them, mIou represents the mean intersection-over-union ratio, C represents the number of categories, C+1 represents the background category, i represents the true value, j represents the predicted value, and P ij Indicates the number of pixels that predict the i-th class pixel as the j-th class pixel, P ii Indicates the number of pixels that predict the i-th type of pixels as the i-th type of pixels.

[0126] In the training phase, the Vit branch uses the AdamW optimizer, preheats for 1500 iterations, and the preheating rate and weight decay rate are set to 1e-6 and 0.9 respectively. The learning rate of the subsequent iterative training is 6e-5, and the polynomial scheduler is used for decay. The convolution branch uses the SGD optimizer, the learning rate and weight decay are set to 0.001 and 1e-4 respectively, and the momentum size is set to 0.9. For the experiments on the PASCALVOC2012 dataset, the batch size and the number of iterations are set to 4 and 20K respectively. Multi-crop data enhancement is used to crop the global image and the local image, whose image sizes are 448x448 and 96x96 respectively. The background threshold (th(l), th(0), th(T)) is set to (0.25, 0.7, 0.55), and the λ1 and λ2 in the semantic consistency regularization loss formula are set to 0.1. The class The temperature factor τ in the token contrast loss follows the ToCo method and is set to 0.5. The w1, w2, and w3 in the total loss follow the same settings as ToCo, which are 0.2, 0.5, and 0.1, respectively. For experiments on the MSCOCO2014 dataset, the batch size is set to 8, the iterations are 80K, the background thresholds (th(l), th(0), th(T)) are set to (0.25, 0.65, 0.55), and the rest of the settings are the same as those of PASCAL VOC2012. The segmentation decoder consists of two convolutional layers with a convolution kernel size of 3x3 and one convolutional layer with a convolution kernel size of 1×1. A simple MLP layer is used for prediction. In the inference stage, following the common practice of semantic segmentation

[19] , multi-scale testing and conditional random fields (CRF) are used for post-processing. This paper adopts the Pytorch deep learning framework, and all experiments are deployed on two A100 graphics cards with a memory size of 40G.

[0127] A series of experiments are conducted on the validation set of the PASCAL VOC 2012 dataset. Figure 5 The experimental results relative to the baseline are shown. All ablation experiments did not use conditional random field CRF for refinement, and the best performance is indicated by shaded fonts.

[0128] Depend on Figure 5 It can be seen that only using the dual-branch network architecture has a 3.1% performance gain compared to the baseline model, which significantly improves the model segmentation performance. The experimental results show that designing a dual-branch network with different architectures is beneficial to expanding the CAM activation area. When the designed activation modulation module (channel modulation module + activation modulation module) is embedded in the convolution branch, the segmentation result reaches 72.2%, which has a performance gain of 4.4% compared to the baseline. The experimental results show that the activation modulation module can promote the model to activate pixels with intermediate scores, thereby forcing the model to activate more target pixels. Figure 5The subset of activated modulation modules is also shown in Figure 1. When only the channel modulation module is added, the segmentation performance is 69.6%, which is a 1.8% performance gain compared to the baseline. When only the spatial modulation module is added, the segmentation performance can be 70.4%, which is a 2.6% performance gain. The subset experiment of activated modulation modules shows that whether adding channel modulation modules or spatial modulation modules can help improve the segmentation performance of the baseline. In addition, Figure 5 The effectiveness of using the dynamic threshold adjustment strategy alone is also shown. When only the dynamic threshold adjustment strategy based on cosine annealing attenuation is added to the dual-branch network, the model has a segmentation performance of 70.8%, and the performance gain over the baseline is 3%. Experimental results show that the dynamic threshold adjustment strategy can enable more confident foreground pixels to participate in segmentation training, alleviating the problem of foreground and background semantic ambiguity. When the complete module and strategy are used for the model, the segmentation performance is further improved, and the best segmentation performance of 73.3% is obtained, which has a performance gain of 5.5% compared with the baseline model. Experimental results show that a complete segmentation map can be generated using the target semantic segmentation model of this application.

[0129] In order to verify the effectiveness of the dual-branch architecture designed in this embodiment, the over-activation rate OR is used as the evaluation index, where the over-activation rate is defined as: OR = FP / (TP+FP), FP represents the number of false positive pixels of the pseudo-label, and TP represents the number of true positive pixels of the pseudo-label. The higher the OR rate, the more pixels of non-target categories are activated by the model. Low-quality CAM leads to poor pseudo-label performance, which ultimately leads to a decline in the quality of the segmentation map. Figure 6 It can be seen that the baseline has a high over-activation rate in some categories, resulting in lower segmentation performance. On the contrary, the dual-branch architecture designed by the method in this embodiment reduces the over-activation rate of multiple categories of the baseline by more than 15%, and even the over-activation rate of the "cow" and "person" categories is only below 5%. The experimental results show that the dual-branch architecture network with different architectures effectively suppresses the over-activation rate of the baseline model and enables the CAM to activate more target pixels.

[0130] In the dual-branch architecture design, in order to make the convolution branch and the Vit branch generate CAMs with consistent activation, the method in this embodiment designs a semantically consistent regularization loss. Figure 7The segmentation performance brought by different regularization methods is shown. When L1 regularization is used as the semantically consistent regularization loss function, the segmentation performance is 73.3%. The main reason for this is the sparse characteristics of L1 regularization, which makes CAM only activate pixels of a specific category and suppress background noise. When L2 regularization is used as the loss function, the segmentation performance is 72.6%. The experimental results are slightly inferior to L1 regularization. The main reason for this is the smoothness characteristics of L2 regularization, which inevitably activates background pixels, thereby relatively reducing the segmentation performance. For this reason, the method of this embodiment uses L1 regularization as the semantically consistent regularization loss for training the dual-branch network architecture.

[0131] In the activation modulation dual-branch weakly supervised semantic segmentation network, th(t) is a dynamic background threshold, which is gradually reduced to th(T) during the training process. The dynamic threshold adjustment strategy aims to allow more confident foreground pixels to participate in the training of the segmentation head and generate accurate segmentation maps with smooth boundaries. Figure 8 The figure shows the segmentation results under different th(t) threshold settings. It can be seen from the table that when the background threshold is set to 0.55, the model achieves the best performance of 73.3%. When the background threshold is reduced to 0.5, the model performance begins to decline. This is because setting a lower background threshold makes the noise pixels also regarded as confident foreground pixels to participate in the segmentation training, which reduces the quality of the segmentation map.

[0132] Fig. 9 The figure shows the computational complexity (Flops) and parameter quantity (Params) of the baseline model and the model in this embodiment. The dual-branch network architecture designed in this embodiment is 4.1G and 5.3M lower than the baseline model in computational complexity and parameter quantity, respectively, but the segmentation performance reaches 73.3%. The experimental results well verify the effectiveness of the dual-branch network architecture based on activation modulation designed in this embodiment for segmentation tasks.

[0133] The activation modulation module designed by the method of this embodiment is a plug-and-play module and can be embedded in any convolution classification network. In order to analyze the effectiveness of the combination method of the channel modulation module and the spatial modulation module, the serial and parallel combination methods are respectively used for the channel spatial modulation model. Fig.10 The segmentation performance of different combination methods is shown. When a parallel structure is used, the segmentation performance obtained is only 71.4%. However, when the channel spatial modulation module is serially combined, the segmentation performance is 73.3%, which has a performance gain of 1.9% compared with the parallel structure. The experimental results show that the serial combination method is conducive to the model to extract rich feature information and make the model pay more attention to pixels with intermediate scores. For this reason, the spatial modulation module is cascaded after the channel modulation module.

[0134] Figure 11-13The class activation map generated by the visualization of the target semantic segmentation model of the present application is shown in Figure 1. Compared with ToCo using the same network architecture, the CAM generated by the method of this embodiment is more complete and accurate, and solves the over-activation problem of ToCo. For example, Figure 11-13 In the 1st, 2nd and 6th columns, it is worth noting that in the 4th column image, due to the extreme similarity between the person's "hair" and the background, ToCo treats "hair" as the background class when activating the "person" object, resulting in mis-segmentation, which ultimately leads to incomplete segmentation of the segmentation map. On the contrary, the target semantic segmentation model successfully responds to the entire object area. Compared with ToCo, it fully activates the target object and restores the semantic object lost by ToCo, such as the 5th column. In addition, Fig.14 The performance of pseudo-labels generated by existing weakly supervised semantic segmentation methods is quantitatively compared. In the one-stage method, pseudo-labels are directly generated by CAM, while the two-stage method requires the initial CAM to be refined to generate the final pseudo-labels. Fig.14 It can be seen that the method of this embodiment achieved the best performance both in the validation set and the test set of the PASCAL VOC 2012 dataset, reaching a pseudo-label performance of 73.5% and 74.9% on the validation set and the test set, respectively. The experimental results well demonstrate that the design of the activation modulation module and the dynamic threshold adjustment strategy can expand the activation area of ​​CAM and alleviate the semantic ambiguity problem of foreground and background pixels.

[0135] Fig.15 The segmentation performance of the two-stage weakly supervised semantic segmentation method and the single-stage weakly supervised semantic segmentation method on the PASCAL VOC 2012 dataset is shown by Fig.15 It can be seen that the segmentation results of the method in this embodiment achieve advanced performance, even better than the two-stage method using saliency maps as additional supervision. The detailed segmentation results of the test set are reported. Compared with ToCo using the same backbone network, the validation set and test set achieve performance gains of 3.1% and 1.8% respectively. The method in this embodiment achieves advanced segmentation performance thanks to the designed dual-branch network architecture, which fully utilizes the advantages of convolution and Transformer models to generate a complete segmentation map. Figure 16-Figure 20 Qualitatively comparing the segmentation results of AFA, ToCo method and the method of this embodiment, it can be seen from the visualization results that AFA has an over-segmentation problem, such as Fig.18 In the first column of , it is worth noting that the mis-segmentation problem caused by the similarity of foreground and background colors affects the final segmentation performance, for example Figure 18-19 The experimental results in the first and third columns show that the method of this embodiment can solve the problem of mis-segmentation when the foreground and background colors are similar, and generate a more complete segmentation map.

[0136] Fig.21It shows the segmentation performance on the validation set of the MSCOCO2014 dataset. The MSCOCO2014 scene is more complex and more challenging for weakly supervised semantic segmentation. Fig.21 It can be seen that the method of this embodiment achieves the best segmentation performance of 45.9% on this dataset, with a performance gain of 0.6% compared with the two-stage weakly supervised semantic segmentation method LPCAM, and a performance gain of 3.6% compared with the same backbone model ToCo. Figure 22-Figure 25 The segmentation results of the ToCo method and the method of this embodiment on the MSCOCO2014 validation set are qualitatively compared. The experimental results show that even in scenes with similar foreground and background colors, the segmentation results of the method of this embodiment are still continuous, such as Figure 24-25 In the first column, compared with ToCo, the segmentation results obtained are more complete, such as Fig.24 In column 5, the target object lost by ToCo is restored.

[0137] like Fig.26 As shown, the present invention provides a training system for a semantic segmentation model, comprising:

[0138] A training data set acquisition module, used to acquire a training data set, wherein the training data set includes at least one image and at least one image-level sticky note;

[0139] A semantic segmentation model module, used for inputting the training data set into a semantic segmentation model, wherein the semantic segmentation model comprises a dual-branch network and a segmentation decoder; wherein the dual-branch network comprises a first feature extraction network with Resnet50 as a branch and a second feature extraction network with Vit as a branch;

[0140] a class activation map determination module, configured for the first feature extraction network to perform feature extraction on the image and the image-level label to determine a first class activation map, and for the second feature extraction network to perform feature extraction on the image and the image-level label to determine a second class activation map;

[0141] a target dual-branch network determination module, configured to perform collaborative training on the dual-branch network through the first type of activation map and the second type of activation map to determine a target dual-branch network;

[0142] The target semantic segmentation model determination module is used to output a target class activation map through the target dual-branch network, and derive a pseudo label from the target class activation map to input into a segmentation decoder to iteratively train the semantic segmentation model to determine the target semantic segmentation model.

[0143] Optionally, the first feature extraction network includes four convolution branches and three activation modulation modules; wherein an activation modulation module is inserted between two adjacent convolution branches;

[0144] The class activation diagram determines the module, including:

[0145] An adjustment feature map determination module is used to repeatedly execute the corresponding functions of the initial feature map determination module and the target modulation feature map determination module from the first stage of the convolution branch to the third stage of the convolution branch to determine the output adjustment feature map;

[0146] An initial feature map determination module, used for inputting the image and the image-level label into the convolution branch for feature extraction to determine an initial feature map;

[0147] A target modulation feature map determination module is used to input the initial feature map into an activation modulation module, adjust the activation distribution of the initial feature map from the channel space dimension using a modulation function, and determine a target modulation feature map;

[0148] The first type activation map determination module is used to input the output target modulation feature map into the fourth stage of the convolution branch for feature extraction to determine the first type activation map.

[0149] Optionally, the activation modulation module includes a channel modulation module and a space modulation module;

[0150] The target modulation feature map determination module is specifically used for:

[0151] Inputting the initial feature map into a channel modulation module, adjusting the feature distribution of the initial feature map by a Gaussian modulation function, and determining a first modulation feature map;

[0152] The first modulation feature map is input into a spatial modulation module, the activation distribution of the first modulation feature map is adjusted by a Gaussian modulation function, a second modulation feature map is determined, and the first modulation feature map and the second modulation feature map are multiplied pixel by pixel to fuse features to determine a target modulation feature map.

[0153] Optionally, the target modulation characteristic map determination module is specifically used to:

[0154] Inputting the initial feature map into the channel modulation module, performing global average pooling on the initial feature map, and determining a global vector;

[0155] Calculating the mean and variance of the global vector, and inputting the mean and variance into a Gaussian modulation function to adjust the characteristic distribution to determine a third modulation characteristic graph after modulation;

[0156] Expanding the characteristic distribution of the third modulation characteristic graph to the same shape and size as the initial characteristic graph to determine a fourth modulation characteristic graph;

[0157] The fourth modulation characteristic map and the initial characteristic map are multiplied pixel by pixel to determine the first modulation characteristic map.

[0158] Optionally, the target modulation characteristic map determination module is specifically used to:

[0159] Input the first modulation feature map into the spatial modulation module, perform max operation and mean operation on the first modulation feature map in the channel dimension, and splice along the channel dimension to determine a splicing tensor;

[0160] Input the concatenated tensor into a convolution layer with a convolution kernel size of 7×7 to determine a feature tensor;

[0161] The feature tensor is input into a Gaussian modulation function, the activation distribution of the feature tensor is adjusted, and a second modulation feature map is determined.

[0162] Optionally, the second feature extraction network includes a classifier and an auxiliary classifier added to the tenth Transformer module in Vit;

[0163] The class activation map determines the module, which is specifically used for:

[0164] Extracting features from the image and the image-level label using a classifier to determine a third type of activation map;

[0165] Extracting features from the image and the image-level label using an auxiliary classifier to determine a fourth type of activation map;

[0166] The second type of activation map includes the third type of activation map and the fourth type of activation map.

[0167] Optionally, the target dual-branch network determination module is specifically used to:

[0168] The semantic consistency regularization loss is calculated according to the first type of activation map, the third type of activation map, and the fourth type of activation map to determine a first loss value, wherein the formula for calculating the semantic consistency regularization loss is as follows:

[0169]

[0170] in, represents the first loss value, M CNN represents the first type of activation map, M Vit represents the third type of activation map, M Aux represents the fourth type of activation map, λ1 and λ2 represent the preset weights, and ||.||1 represents regularization;

[0171] The two-branch network is collaboratively trained using the semantic regularization loss to determine a target two-branch network.

[0172] Optionally, the target semantic segmentation model determination module is specifically used to:

[0173] Each time the semantic segmentation model is iteratively trained, the background threshold is updated by a cosine annealing decay dynamic threshold adjustment strategy; wherein the formula of the cosine annealing decay dynamic threshold adjustment strategy is as follows:

[0174]

[0175] Where t represents the current iteration, T represents the total number of iteration training, th(t) represents the background threshold corresponding to the current iteration, and th(0) represents the set initial threshold;

[0176] Derive pseudo labels from the target class activation map according to the background threshold corresponding to the current iteration;

[0177] The pseudo label corresponding to the current iteration and the target class activation map are input into the segmentation decoder for current iteration training.

[0178] An embodiment of the present invention also provides a computing device, including a memory, a manager, and a program stored in the memory and running on the manager. When the manager executes the program, it implements some or all steps of the above-mentioned training method of a semantic segmentation model.

[0179] Among them, the computing device can be a computer, and correspondingly, its program is computer software. The above-mentioned parameters and steps in the computing device of the present invention can refer to the parameters and steps in the embodiment of a training method for a semantic segmentation model in the above text, and will not be repeated here.

[0180] Those skilled in the art know that the present invention can be implemented as a system, method or computer program product. Therefore, the present disclosure can be specifically implemented in the following forms, namely: it can be complete hardware, it can be complete software (including firmware, resident software, microcode, etc.), or it can be a combination of hardware and software, generally referred to as "circuit", "module" or "system" herein. In addition, in some embodiments, the present invention can also be implemented in the form of a computer program product in one or more computer-readable media, and the computer-readable medium contains computer-readable program code. Computer-readable storage media can be, for example, but not limited to - electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or devices, or any combination of the above.

[0181] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0182] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A method for training a semantic segmentation model, characterized in that: include: Obtain a training dataset, wherein the training dataset includes at least one image and at least one image-level label; Inputting the training data set into a semantic segmentation model, the semantic segmentation model comprising a dual-branch network and a segmentation decoder; wherein the dual-branch network comprises a first feature extraction network with Resnet50 as a branch and a second feature extraction network with Vit as a branch; The first feature extraction network extracts features from the image and the image-level label to determine a first type of activation map, and the second feature extraction network extracts features from the image and the image-level label to determine a second type of activation map; Co-training the dual-branch network by using the first type of activation map and the second type of activation map to determine a target dual-branch network; The target dual-branch network outputs a target class activation map, and a pseudo label is derived from the target class activation map and input into a segmentation decoder to iteratively train the semantic segmentation model to determine a target semantic segmentation model.

2. The method according to claim 1, characterized in that The first feature extraction network includes four convolution branches and three activation modulation modules; wherein an activation modulation module is inserted between two adjacent convolution branches; The first feature extraction network extracts features from the image and the image-level label to determine a first type of activation map, including: S11, starting from the first stage of the convolution branch until the third stage of the convolution branch, repeatedly execute S111-S112 to determine the output target adjustment feature map; S111, inputting the image and the image-level label into the convolution branch for feature extraction to determine an initial feature map; S112, inputting the initial feature map into an activation modulation module, adjusting the activation distribution of the initial feature map from the channel space dimension using a modulation function, and determining a target modulation feature map; S12, input the output target modulation feature map into the fourth stage of the convolution branch for feature extraction, and determine the first type of activation map.

3. The method according to claim 2, characterized in that The activation modulation module includes a channel modulation module and a space modulation module; The initial feature map is input into the activation modulation module, and the activation distribution of the initial feature map is adjusted from the channel space dimension using a modulation function to determine a target modulation feature map, including: Inputting the initial feature map into a channel modulation module, adjusting the feature distribution of the initial feature map by a Gaussian modulation function, and determining a first modulation feature map; The first modulation feature map is input into a spatial modulation module, the activation distribution of the first modulation feature map is adjusted by a Gaussian modulation function, a second modulation feature map is determined, and the first modulation feature map and the second modulation feature map are multiplied pixel by pixel to fuse features to determine a target modulation feature map.

4. The method according to claim 3, characterized in that Inputting the initial feature map into a channel modulation module, adjusting the feature distribution of the initial feature map by a Gaussian modulation function, and determining a first modulation feature map, including: Inputting the initial feature map into the channel modulation module, performing global average pooling on the initial feature map, and determining a global vector; Calculating the mean and variance of the global vector, and inputting the mean and variance into a Gaussian modulation function to adjust the characteristic distribution to determine a third modulation characteristic graph after modulation; Expanding the characteristic distribution of the third modulation characteristic graph to the same shape and size as the initial characteristic graph to determine a fourth modulation characteristic graph; The fourth modulation characteristic map and the initial characteristic map are multiplied pixel by pixel to determine the first modulation characteristic map.

5. The method according to claim 3, characterized in that: Inputting the first modulation characteristic map into a spatial modulation module, adjusting the activation distribution of the first modulation characteristic map by a Gaussian modulation function, and determining a second modulation characteristic map, including: Input the first modulation feature map into the spatial modulation module, perform max operation and mean operation on the first modulation feature map in the channel dimension, and splice along the channel dimension to determine a splicing tensor; Input the concatenated tensor into a convolution layer with a convolution kernel size of 7×7 to determine a feature tensor; The feature tensor is input into a Gaussian modulation function, the activation distribution of the feature tensor is adjusted, and a second modulation feature map is determined.

6. The method according to claim 2, characterized in that The second feature extraction network includes a classifier and an auxiliary classifier added in the tenth Transformer module in Vit; Then the second feature extraction network extracts features from the image and the image-level label to determine a second type of activation map, including: Extracting features from the image and the image-level label using a classifier to determine a third type of activation map; Extracting features from the image and the image-level label using an auxiliary classifier to determine a fourth type of activation map; The second type of activation map includes the third type of activation map and the fourth type of activation map.

7. The method according to claim 6, characterized in that The dual-branch network is collaboratively trained by using the first type of activation map and the second type of activation map to determine a target dual-branch network, including: The semantic consistency regularization loss is calculated according to the first type of activation map, the third type of activation map, and the fourth type of activation map to determine a first loss value, wherein the formula for calculating the semantic consistency regularization loss is as follows: l er =λ1||M CNN -M Aux ||1+λ2||M CNN -M Vit ||1 Among them, l er Represents the first loss value, M CNN represents the first type of activation map, M Vit represents the third type of activation map, M Aux represents the fourth type of activation map, λ1 and λ2 represent the preset weights, and ||·||1 represents regularization; The two-branch network is collaboratively trained using the semantic regularization loss to determine a target two-branch network.

8. The method according to claim 1, characterized in that The step of deriving a pseudo label from the target class activation map and inputting it into a segmentation decoder to iteratively train the semantic segmentation model comprises: Each time the semantic segmentation model is iteratively trained, the background threshold is updated by a cosine annealing decay dynamic threshold adjustment strategy; wherein the formula of the cosine annealing decay dynamic threshold adjustment strategy is as follows: Where t represents the current iteration, T represents the total number of iteration training, th(t) represents the background threshold corresponding to the current iteration, and th(0) represents the set initial threshold; Derive pseudo labels from the target class activation map according to the background threshold corresponding to the current iteration; The pseudo label corresponding to the current iteration and the target class activation map are input into the segmentation decoder for current iteration training.

9. A training system for a semantic segmentation model, characterized in that: include: A training data set acquisition module, used to acquire a training data set, wherein the training data set includes at least one image and at least one image-level sticky note; A semantic segmentation model module, used for inputting the training data set into a semantic segmentation model, wherein the semantic segmentation model comprises a dual-branch network and a segmentation decoder; wherein the dual-branch network comprises a first feature extraction network with Resnet50 as a branch and a second feature extraction network with Vit as a branch; a class activation map determination module, configured for the first feature extraction network to perform feature extraction on the image and the image-level label to determine a first class activation map, and for the second feature extraction network to perform feature extraction on the image and the image-level label to determine a second class activation map; a target dual-branch network determination module, configured to perform collaborative training on the dual-branch network through the first type of activation map and the second type of activation map to determine a target dual-branch network; The target semantic segmentation model determination module is used to output a target class activation map through the target dual-branch network, and derive a pseudo label from the target class activation map to input into a segmentation decoder to iteratively train the semantic segmentation model to determine the target semantic segmentation model.

10. A computing device comprising a memory, a processor, and a program stored in the memory and running on the processor, characterized in that: When the processor executes the program, the steps of a method for training a semantic segmentation model as described in claims 1-8 are implemented.