A weakly supervised semantic segmentation method and system balancing scribble annotation preference

Abstract

The application provides a weakly supervised semantic segmentation method for balancing scribble annotation preferences, comprising: constructing a semantic segmentation model; performing scribble annotation on a training picture; extracting deep semantic features of the training picture, obtaining a preliminary prediction map of the training picture based on the deep semantic features; extracting shallow features of the training picture, performing local aggregation on the preliminary prediction map, and obtaining a final prediction map of the training picture; obtaining a label probability map of the training picture; adding a pixel-wise weighted cross-entropy loss of the semantic segmentation model and a segmentation prediction learning loss to obtain a total loss function; iteratively updating model parameters of the semantic segmentation model until the total loss function converges, and taking the semantic segmentation model at this time as a final segmentation model. The application also provides a weakly supervised semantic segmentation system for balancing scribble annotation preferences, and a data processing device for performing weakly supervised semantic segmentation.

Description

Technical Field

[0001] This invention relates to the field of computer vision, specifically to a semantic segmentation method and system under graffiti supervision. Background Technology

[0002] Current image semantic segmentation methods are mainly based on deep neural networks and their variations. These methods require collecting a large number of images and corresponding pixel-by-pixel semantic segmentation labels, and then training the deep neural network to obtain good model prediction results. Models trained in this way are called strongly supervised semantic segmentation models. However, manual annotation of pixel-by-pixel semantic segmentation labels is time-consuming and costly, requiring significant upfront resource investment for practical applications. Currently, the main solution to this problem is to utilize various relatively inexpensive weak labels, such as category labels, bounding box labels, and doodle labels, to train the neural network model. Models trained in this way are called weakly supervised semantic segmentation models. Among these, doodle supervision (also known as doodle labeling) is one of the most commonly used weak labels in the field of weakly supervised semantic segmentation due to its fast annotation speed and high annotation accuracy.

[0003] Existing graffiti-supervised semantic segmentation methods share some commonalities with existing strongly supervised semantic segmentation methods. However, during the neural network training phase, these methods only calculate the loss function for labeled pixels, ignoring pixels not labeled by the graffiti. Furthermore, they introduce various regularization terms or expand the coverage area of ​​the graffiti labels to increase the supervision signal, thereby improving the prediction accuracy of the segmentation model. For example, the Chinese national patent "A Graffiti-Based Weakly Supervised Semantic Segmentation Method and System" (Publication No.: CN110443818A) designs a boundary regression network and corresponding boundary regression loss to constrain the segmentation prediction results of image edge regions during model training, thus improving the accuracy of segmentation prediction. The Chinese national patent "A Graffiti-Supervised High-Resolution Saliency Target Detection Method" (Publication No.: CN114332490A) maintains an edge structure preservation branch while using a deep network to learn semantic features, enabling it to extract edge features of the target at higher resolution, thereby improving the quality of the segmentation results in subsequent fusion. The Chinese national patent "A Weakly Supervised Semantic Segmentation Method Based on Guided Attention Reasoning Network" (Publication No.: CN108647684A) uses a guided attention mechanism and an attention mining method to ensure that all category-related regions are included in the network's attention, making the attention map more complete and accurate. The RAWKS method, on the other hand, expands the spatial area of ​​the originally sparse doodle labels by statistically analyzing the pixel hit probability of random walks.

[0004] While existing deep neural network models based on graffiti supervision have significantly outperformed traditional segmentation methods based on hand-crafted features, they still lag considerably behind strongly supervised semantic segmentation models. The main differences lie in two aspects: First, graffiti supervision, as a sparse annotation method, typically only calculates the training loss for the labeled pixels during model training, resulting in low data utilization and lower performance compared to strongly supervised semantic segmentation models that fully utilize data. Second, graffiti labels inherently exhibit bias during annotation, leading to spatial distribution differences between the labeled pixels in the graffiti labels and the pixel-by-pixel segmentation labels. Training the same segmentation model with labels exhibiting these distributional differences inevitably results in inconsistent training outcomes, which is one of the key reasons why graffiti-supervised semantic segmentation methods struggle to approach the performance of strongly supervised semantic segmentation methods. Summary of the Invention

[0005] To address the aforementioned problems, this invention proposes a weakly supervised semantic segmentation method and system that balances doodle annotation preferences. The method proposes a neural network architecture called LibraSeg, which includes five parts: a backbone network, a shallow feature extractor, a doodle branch, a segmentation branch, and a local aggregation module. Simultaneously, it uses two loss functions matched to this neural network architecture—weighted cross-entropy loss and segmentation prediction learning loss—to enable the network model to overcome the performance degradation caused by doodle label annotation preferences during training. Specifically, this invention includes:

[0006] Step 1: Build the LibraSeg model by collecting a certain number of images and labeling them with doodle tags, and use the obtained image-doodle tag pairs to form the training set;

[0007] Step 2: Sample from the training set and input the images into the skeleton network of the LibraSeg model to obtain the deep semantic features of the current image;

[0008] Step 3: Input the deep semantic features of the current image (obtained in Step 2) into the segmentation branch of the LibraSeg model to obtain the preliminary semantic segmentation prediction map of the current image;

[0009] Step 4: Input the original image into the shallow feature extractor in the LibraSeg model to obtain the shallow feature map of the current image; then, combine the preliminary semantic segmentation prediction result of the current image (obtained in Step 3) with this shallow feature map. Figure 1 Starting with the local aggregation module in the LibraSeg model, the segmentation probability map P of the current image is obtained. seg This refers to the final segmentation result of the model.

[0010] Step 5: Input the deep semantic features of the current image (obtained in Step 2) into the graffiti branch of the LibraSeg model to obtain the labeled probability map p of the current image. scrib ;

[0011] Step 6: Assign an additional category to the unlabeled pixels in the graffiti labels (obtained in Step 1) corresponding to the current image, and calculate the pixel-wise weighted cross-entropy loss for label preference learning by combining it with the label probability map (obtained in Step 5).

[0012] Suppose the current graffiti label y has K candidate categories. First, assign an additional "unlabeled" category to the pixels not marked by the graffiti, making the number of candidate categories for the label K+1, and denote the newly obtained label as y'. Then calculate the weighted cross-entropy loss pixel by pixel for the current image:

[0013]

[0014] in This represents the predicted annotation probability of the doodle branch of the LibraSeg model for class i. wi represents the class weights, used to balance the roles of samples from different classes during training.

[0015] Step 7: Calculate the LibraLoss loss used for model segmentation prediction learning based on the current image's labeled probability map (obtained in Step 5), the final semantic segmentation prediction map (obtained in Step 4), and the graffiti tags (obtained in Step 1).

[0016] The Libra Loss calculation is as follows:

[0017]

[0018] Where y i ∈{0,1} represents the graffiti label of the i-th class. This represents the prediction result of the segmentation probability map on the i-th class. If the current pixel has a low labeling probability, A value close to 0 indicates that the current pixel's spatial position is biased towards not being marked by the graffiti label. If a graffiti label is present at the current pixel, i.e., y... i =1 indicates that the current graffiti label has been placed in an area that is more likely to be unlabeled, indicating high rarity. Loss weight Taking a value close to 1 reflects the rarity of the label, which makes the rare label play a relatively crucial supervisory role in the training of the LibraSeg network model, thereby overcoming the negative impact of the spatial distribution difference of the labeled pixels of the graffiti label and the segmentation label.

[0019] Step 8: Combine the pixel-wise cross-entropy loss obtained in Step 6 with the LibraLoss loss obtained in Step 7 to construct the total loss function:

[0020] L=CE(p scrib ,y')+LL(p seg ,p scrib ,y)

[0021] Step 9: Update the LibraSeg model parameters using the gradient backpropagation algorithm, and determine whether the total loss function has converged. If it has, stop training; otherwise, continue with steps 2-8.

[0022] Step 10: Using the trained LibraSeg model as the final segmentation model, perform semantic segmentation on the target image with completed graffiti annotations.

[0023] The advantage of the LibraSeg model proposed in this invention is that it introduces an additional graffiti branch on top of the traditional segmentation branch. This branch can statistically analyze the distribution characteristics of graffiti labels in the graffiti annotation dataset and dynamically estimate the preferred annotation regions of graffiti labels in the current image. This allows it to be used to assess the rarity of the actual graffiti labels.

[0024] The local aggregation module in the LibraSeg model proposed in this invention effectively realizes the label propagation function for sparse annotations from the perspective of network structure design, effectively improves the utilization rate of annotation information, achieves plug-and-play functionality, and has good practical prospects.

[0025] The weighted cross-entropy loss and the segmentation prediction learning loss called Libra Loss proposed in this invention are training loss functions specifically designed for the LibraSeg model. The weighted cross-entropy loss effectively overcomes the drawback of excessive unlabeled pixels dominating training by appropriately reducing the loss weight of extra unlabeled pixels during training, thus enabling the LibraSeg network to train and obtain a label probability map that reflects graffiti annotation preferences. Libra Loss, on the other hand, evaluates the rarity of the corresponding graffiti label in the current image based on the label probability map, dynamically increasing the training weight of graffiti labels appearing in rare positions, thereby overcoming the potential performance degradation caused by graffiti annotation preferences and improving the prediction accuracy of the final semantic segmentation result.

[0026] The present invention also proposes a computer-readable storage medium storing computer-executable instructions, characterized in that, when the computer-executable instructions are executed, the weakly supervised semantic segmentation method for balancing doodle annotation preferences as described above is implemented.

[0027] The present invention also proposes a data processing apparatus, including a computer-readable storage medium as described above, wherein when the processor of the data processing apparatus retrieves and executes computer-executable instructions in the computer-readable storage medium, it performs weakly supervised semantic segmentation that balances graffiti annotation preferences. Attached Figure Description

[0028] Figure 1 This is a flowchart of the weakly supervised semantic segmentation method for balancing graffiti annotation preferences according to the present invention.

[0029] Figure 2 This is an architecture diagram of the weakly supervised semantic segmentation method for balancing graffiti annotation preferences according to the present invention.

[0030] Figure 3A , 3B 3C are graffiti annotation examples of the present invention.

[0031] Figure 4 This is a schematic diagram of the data processing device of the present invention. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0033] The purpose of this invention is to address the problem of poor training results in graffiti-supervised semantic segmentation models caused by the inconsistency in the spatial distribution of labeled pixels between graffiti labels and pixel-by-pixel semantic segmentation labels. This invention proposes a neural network model architecture called LibraSeg (hereinafter referred to as the LibraSeg model) and a corresponding segmentation prediction learning loss function called Libra Loss. By predicting the label preference region of the current image through the graffiti branch in the LibraSeg model, the rarity of graffiti labels in the dataset is assessed. This allows for dynamic adjustment of the supervision signal weights during model training, achieving a balanced label preference effect and improving the model's semantic segmentation prediction performance.

[0034] In their research on graffiti-supervised semantic segmentation, the inventors noticed a natural bias in graffiti labeling. Labelers tend to annotate areas of the image that are easily labeled, leading to spatial differences in the distribution of labeled pixels between graffiti labels and pixel-by-pixel segmentation labels. This phenomenon is not considered in existing graffiti-supervised semantic segmentation methods. To address this labeling bias, both the model structure and the loss function used for model training need to be rationally designed and modified. Therefore, the inventors proposed a neural network model architecture called LibraSeg, and a corresponding segmentation prediction learning loss function called Libra Loss. In LibraSeg, the graffiti branch predicts the preferred labeling regions of the current image, assessing the rarity of graffiti labels in the dataset. Simultaneously, the segmentation branch in LibraSeg uses a local aggregation module to spatially diffuse graffiti supervision, increasing the area of ​​the supervision signal and improving data utilization efficiency during model training. The two outputs are used to calculate the loss function through Libra Loss, dynamically adjusting the supervision signal weights during model training to balance labeling bias. Experiments demonstrate that this method achieves performance closer to that of strongly supervised semantic segmentation.

[0035] In the LibraSeg model proposed in this invention's weakly supervised semantic segmentation method, the graffiti prediction branch has the ability to predict the current image's preferred labeling region. Based on this, it can be used to evaluate the rarity of the graffiti label corresponding to the image in the dataset. In addition, the local aggregation module in the LibraSeg model can propagate the supervision signal generated by the loss function from the labeled pixels to more unlabeled pixel positions during model training based on pixel similarity, improving data utilization during training. The proposed Libra Loss loss function also dynamically adjusts the weight of the current sample during model training based on the rarity of the graffiti label, encouraging the model to learn more from the supervision signal brought by rare graffiti labels, thereby overcoming the negative impact of the difference in the spatial distribution of labeled pixels between graffiti labels and pixel-by-pixel segmentation labels, and improving the semantic segmentation prediction accuracy of the model.

[0036] Figure 1 This is a flowchart of the weakly supervised semantic segmentation method for balancing graffiti annotation preferences according to the present invention. Figure 2 This is an architecture diagram of the weakly supervised semantic segmentation method for balancing graffiti annotation preferences according to the present invention. Figure 1 , 2 As shown, the weakly supervised semantic segmentation method of the present invention specifically includes:

[0037] Step S1: Construct the LibraSeg model; collect images of a certain scale and label them with doodle tags, forming a training set from the obtained image-doodle tag pairs. The LibraSeg model of this invention comprises five parts: a skeleton network, a shallow feature extractor, a doodle branch, a segmentation branch, and a local aggregation module. In this embodiment, a network structure combining ResNet101 and UpperNet is used as the skeleton network; a shallow feature extractor is used, consisting of a convolutional layer with a stride of 4 and a 4×4 kernel size, and a fully connected layer; a network combination using 1×1 convolutions with ReLU activation functions is repeated three times and stacked (forming a convolution-ReLU-convolution-ReLU-convolution-ReLU structure) as the doodle branch; a network combination using 1×1 convolutions with ReLU activation functions is repeated three times and stacked (forming a convolution-ReLU-convolution-ReLU-convolution-ReLU structure) as the segmentation branch. Other network structures can also be used to construct the various parts of the LibraSeg model. For example, FCN, DeepLab, Transformer and other model structures can be used as the skeleton network part in this invention. Alternatively, convolutional layers with different kernel sizes, different lengths, different activation functions, or different numbers of layers can be used to replace the current structures used in the shallow feature extractor, graffiti branch, segmentation branch and other parts in this invention. This invention is not limited to these.

[0038] Step S2: Sample from the training set and input the image into the skeleton network of the LibraSeg model to obtain the deep semantic features of the current image;

[0039] Step S3: Input the deep semantic features of the current image (obtained in step S2) into the segmentation branch of the LibraSeg model to obtain the preliminary semantic segmentation prediction map of the current image;

[0040] Step S4: Input the original image into the shallow feature extractor in the LibraSeg model to obtain the shallow feature map of the current image; combine the preliminary semantic segmentation prediction result of the current image (obtained in step S3) with the shallow feature map. Figure 1 Starting with the local aggregation module in the LibraSeg model, the segmentation probability map P of the current image is obtained. seg This refers to the final segmentation result of the model, and the specific calculation process is as follows:

[0041] First, the original image is input into the shallow feature extractor to obtain the feature vector f of the shallow features corresponding to each pixel. Then, the shallow feature similarity is calculated pairwise for all pixels (N pixels in total). Where i, j, k represent pixel indices. Simultaneously, based on the spatial coordinates s of each pixel, the pairwise positional similarity between all pixels is calculated. Thus, the pixel similarity matrix is ​​obtained. The calculation method is as follows:

[0042]

[0043] Let the initial semantic segmentation prediction result input from the local aggregation module be: The corresponding dimension is K, and the number of pixels is N. The forward computation process of the local aggregation module is as follows:

[0044] X out =X in A

[0045] Here The output segmentation probability map p seg The matrix representation of this is used. This calculation process achieves local aggregation of segmentation prediction results based on pixel similarity, thus ensuring that the following formula is satisfied during gradient updates during network backpropagation:

[0046]

[0047] Figure 3A , 3B 3C is a graffiti annotation diagram of the present invention. For example... Figure 3A , 3B As shown in 3C, among which, Figure 3A For the original image, Figure 3B For graffiti labels, Figure 3C This is the image after the original image and graffiti labels are superimposed and aligned. Red, pink, and black graffiti represent different categories of graffiti labels, and the pixels they cover are the "labeled pixels." The remaining pixels not covered by the graffiti ( Figure 3A quilt Figure 3B The pixels covered by the white area are considered "unlabeled pixels." Since pixels can be categorized as "labeled pixels" and "unlabeled pixels" under graffiti labels, the input and output of the local aggregation module can be formalized as follows: Since the training loss is calculated only on the "labeled pixels", Therefore, the gradient calculation formula for the local aggregation module can be updated as follows:

[0048]

[0049] As can be seen, the gradient calculated on the labeled pixels is distributed to more unlabeled pixels according to pixel similarity, thereby realizing the spatial propagation of the graffiti label supervision signal.

[0050] Step S5: Input the deep semantic features of the current image (obtained in step S2) into the graffiti branch of the LibraSeg model to obtain the labeled probability map p of the current image. scrib ;

[0051] Step S6: Assign an additional category to the unlabeled pixels in the graffiti labels corresponding to the current image (obtained in step S1), and calculate the pixel-wise weighted cross-entropy loss for label preference learning with the label probability map (obtained in step S5);

[0052] Suppose the current graffiti label y has K candidate categories. First, assign an additional "unlabeled" category to the pixels not marked by the graffiti, making the number of candidate categories for the label K+1, and denote the newly obtained label as y'. Then calculate the weighted cross-entropy loss pixel by pixel for the current image:

[0053]

[0054] in This represents the predicted annotation probability of the doodle branch of the LibraSeg model for class i. wi represents the class weights, used to balance the roles of samples from different classes during training.

[0055] Step S7: Based on the current image's labeled probability map (obtained in step S5), the final semantic segmentation prediction map (obtained in step S4), and the graffiti label (obtained in step S1), calculate the LibraLoss loss used for model segmentation prediction learning.

[0056] The Libra Loss calculation is as follows:

[0057]

[0058] Where y i ∈{0,1} represents the graffiti label of the i-th class. This represents the prediction result of the segmentation probability map on the i-th class. If the current pixel has a low labeling probability, A value close to 0 indicates that the current pixel's spatial position is biased towards not being marked by the graffiti label. If a graffiti label is present at the current pixel, i.e., y... i =1 indicates that the current graffiti label has been placed in an area that is more likely to be unlabeled, indicating high rarity. Loss weight Taking a value close to 1 reflects the rarity of the label, which makes the rare label play a relatively crucial supervisory role in the training of the LibraSeg network model, thereby overcoming the negative impact of the spatial distribution difference of the labeled pixels of the graffiti label and the segmentation label.

[0059] Step S8: Sum the pixel-wise cross-entropy loss obtained in step S6 and the LibraLoss loss obtained in step S7 to construct the total loss function:

[0060] L=CE(p scrib ,y')+LL(p seg ,p scrib ,y)

[0061] Step S9: Update the LibraSeg model parameters using the gradient backpropagation algorithm and determine whether the total loss function has converged. If it has, stop training; otherwise, continue with steps S2-S8.

[0062] Step S10: Using the trained LibraSeg model as the final segmentation model, perform semantic segmentation on the target image with completed graffiti annotations.

[0063] This invention also proposes a weakly supervised semantic segmentation system that balances graffiti annotation preferences, specifically including:

[0064] The initialization module is used to build a semantic segmentation model and perform graffiti annotation on the training images to obtain the graffiti-labeled and unlabeled regions of the training images. The backbone network is a combination of ResNet101 and UpperNet. The shallow feature extractor is a combination of a convolutional layer with a stride of 4 and a kernel size of 4×4 and a fully connected layer. The graffiti branch and the segmentation branch are a combination of three 1×1 convolutions and ReLU activation functions.

[0065] The semantic segmentation prediction module is used to obtain the final prediction image for semantic segmentation of the training image;

[0066] The loss function generation module is used to generate the total loss function. Specifically, it constructs the pixel-wise weighted cross-entropy loss and the segmentation prediction learning loss for the model, and then sums these two losses to obtain the total loss function for the semantic segmentation model.

[0067] Pixel-wise weighted cross-entropy loss

[0068] Segmentation prediction learning loss

[0069] Then the total loss function L = CE(p) scrib ,y')+LL(p seg ,p scrib ,y);

[0070] The model training module is used to train and generate the final segmentation model. It updates the model parameters of the semantic segmentation model and sequentially calls the semantic segmentation prediction module and the loss function generation module until the total loss function converges. The semantic segmentation model at this point is used as the final segmentation model. The final segmentation model is used to perform semantic segmentation on the target image with completed graffiti annotation.

[0071] Figure 4 This is a schematic diagram of the data processing apparatus of the present invention. Figure 4 As shown, embodiments of the present invention also provide a computer-readable storage medium and a data processing apparatus. The computer-readable storage medium of the present invention stores computer-executable instructions. When these computer-executable instructions are executed by the processor of the data processing apparatus, the aforementioned weakly supervised semantic segmentation method for balancing graffiti annotation preferences is performed. Those skilled in the art will understand that all or part of the steps in the above method can be implemented by a program instructing related hardware (e.g., processor, FPGA, ASIC, etc.), and the program can be stored in a readable storage medium, such as a read-only memory, a disk, or an optical disk. All or part of the steps in the above embodiments can also be implemented using one or more integrated circuits. Accordingly, each module in the above embodiments can be implemented in hardware, for example, by implementing its corresponding function through an integrated circuit, or it can be implemented in the form of a software functional module, for example, by a processor executing a program / instruction stored in memory to implement its corresponding function. Embodiments of the present invention are not limited to any particular combination of hardware and software.

[0072] To verify the effectiveness of the method, the inventors conducted experiments on the PASCAL VOC 2012 and PASCALContext datasets. The PASCAL VOC 2012 dataset used 10,582 images with doodle labels as the training set, requiring differentiation of 21 categories, including background. It also provided 1,449 images with pixel-wise segmentation labels as the validation set to verify the method's performance. The PASCAL Context dataset used 4,998 images with doodle labels as the training set, requiring differentiation of 60 categories, including background. It also provided 5,105 images with pixel-wise segmentation labels as the validation set. The doodle labels used in both datasets were obtained from the paper "Scribblesup: Scribble-supervised convolutional networks for semanticsegmentation." (In: Proc. IEEE Conference on Computer Vision and Pattern Recognition.).

[0073] First, partial experiments with controlled variables were conducted on PASCAL VOC, and the results are shown in Table 1:

[0074]

[0075] Table 1: Comparison of model performance resulting from adding different method components in PASCAL VOC 2012

[0076] The model performance under different experimental settings was statistically analyzed by averaging three training iterations. It can be seen that adding a local aggregation module and using Libra Loss to train the model both improve performance.

[0077] Furthermore, the method of this invention is compared with the best existing method on the PASCAL VOC 2012 dataset and the PASCAL Context dataset, as shown in Tables 2 and 3. Under the same experimental settings, the method of this invention has higher accuracy in semantic segmentation prediction results.

[0078]

[0079] Table 2: Performance comparison of PASCAL VOC 2012 dataset and state-of-the-art methods

[0080]

[0081] Table 3: Performance comparison of PASCAL Context dataset and state-of-the-art methods

[0082] The above embodiments are only used to illustrate the present invention and are not intended to limit the present invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, all equivalent technical solutions also fall within the scope of the present invention, and the patent protection scope of the present invention should be defined by the claims.

Claims

1. A weakly supervised semantic segmentation method that balances graffiti annotation preferences, characterized in that, include: Step 1: Construct a semantic segmentation model, which includes a backbone network, a shallow feature extractor, a doodle branch, a segmentation branch, and a local aggregation module; Add annotations to the training images and obtain the annotated and unannotated areas of the training images; Step 2: Extract deep semantic features from the training image using the skeleton network; use the deep semantic features through the segmentation branch to obtain a preliminary prediction map for semantic segmentation of the training image. Step 3: Extract shallow features from the training image using the shallow feature extractor, and use the shallow features to perform local aggregation on the preliminary prediction image using the local aggregation module to obtain the final prediction image for semantic segmentation of the training image. Step 4: Obtain the labeled probability map of the training image using the deep semantic features through the graffiti branch; The unlabeled area is labeled as an additional category, and the total category of the training image is the sum of the additional category and all the graffiti categories of the labeled area. Using the labeled probability map and the total category, the pixel-wise weighted cross-entropy loss is calculated; using the labeled probability map, the final prediction map, and the label region, the segmentation prediction learning loss is calculated; and the total loss function of the semantic segmentation model is obtained by summing the pixel-wise weighted cross-entropy loss and the segmentation prediction learning loss. Step 5: Update the model parameters of the semantic segmentation model, and repeat Step 2 to Step 4 in sequence until the total loss function converges. Use the semantic segmentation model at this point as the final segmentation model; use the final segmentation model to perform semantic segmentation on the target image with completed graffiti annotation.

2. The weakly supervised semantic segmentation method as described in claim 1, characterized in that, The backbone network is a combination of ResNet101 and UpperNet; the shallow feature extractor is a combination of a convolutional layer with a stride of 4 and a kernel size of 4×4 and a fully connected layer. The network combination of three 1×1 convolutions with ReLU activation function is used as the graffiti branch and the segmentation branch.

3. The weakly supervised semantic segmentation method as described in claim 1, characterized in that, The process of obtaining the final predicted map in step 3 includes: Extract the feature vector f of the shallow feature. For all N pixels of the training image, obtain the shallow feature similarity between pixel i and pixel j. i≠j; Based on the spatial coordinates s of each pixel, obtain the positional similarity between pixel i and pixel j. Obtain the pixel similarity matrix Then the final predicted graph p seg The matrix representation of X out =X in A; where X in This is the matrix representation of the preliminary prediction map. K represents the candidate dimension for this graffiti category.

4. The weakly supervised semantic segmentation method as described in claim 3, characterized in that, The pixel-wise weighted cross-entropy loss The segmentation prediction learning loss The total loss function L = CE(p) scrib ,y')+LL(p seg ,p scrib ,y); Where y is the label for this graffiti category, y i ∈{0,1}, y' is the label of the overall category, which has K+1 candidate dimensions, p scrib This indicates the labeled probability map. w represents the predicted labeled probability of the semantic segmentation model on category i. i This represents the class weight of i. p seg In the labeling probability of category i, γ represents the hyperparameter used to adjust the sample loss weights.

5. The weakly supervised semantic segmentation method as described in claim 1, characterized in that, In step 5, the gradient backpropagation algorithm is used to update the model parameters.

6. A weakly supervised semantic segmentation system that balances graffiti annotation preferences, characterized in that, include: The initialization module is used to build a semantic segmentation model, which includes a backbone network, a shallow feature extractor, a doodle branch, a segmentation branch, and a local aggregation module. The training images are then annotated with graffiti to obtain the labeled and unlabeled areas of the training images. The semantic segmentation prediction module is used to obtain the final prediction map for semantic segmentation of the training image. Specifically, the deep semantic features of the training image are extracted through the backbone network; the segmentation branch uses these deep semantic features to obtain a preliminary prediction map for semantic segmentation of the training image; the shallow feature extractor extracts shallow features of the training image; and the local aggregation module uses these shallow features to locally aggregate the preliminary prediction map to obtain the final prediction map for semantic segmentation of the training image. The loss function generation module is used to generate the total loss function of the semantic segmentation model. Specifically, it obtains the labeled probability map of the training image using the deep semantic features through the graffiti branch; labels the unlabeled region as an additional category, and uses the additional category and all graffiti categories of the labeled region as the total category of the training image; calculates the pixel-wise weighted cross-entropy loss using the labeled probability map and the total category; calculates the segmentation prediction learning loss using the labeled probability map, the final prediction map, and the labeled region; and sums the pixel-wise weighted cross-entropy loss and the segmentation prediction learning loss to obtain the total loss function of the semantic segmentation model. The model training module is used to train and generate the final segmentation model. It updates the model parameters of the semantic segmentation model and sequentially calls the semantic segmentation prediction module and the loss function generation module until the total loss function converges. The semantic segmentation model at this point is used as the final segmentation model. The final segmentation model is used to perform semantic segmentation on the target image with completed graffiti annotation.

7. The weakly supervised semantic segmentation system as described in claim 6, characterized in that, In this initialization module, a network structure combining ResNet101 and UpperNet is used as the backbone network; a network structure combining a convolutional layer with a stride of 4 and a kernel size of 4×4 and a fully connected layer is used as the shallow feature extractor. The network combination of three 1×1 convolutions with ReLU activation function is used as the graffiti branch and the segmentation branch.

8. The weakly supervised semantic segmentation system as described in claim 6, characterized in that, The loss function generation module includes: The pixel-wise weighted cross-entropy loss The segmentation prediction learning loss The total loss function L = CE(p) scrib ,y')+LL(p seg ,p scrib ,y); Where y is the label of the graffiti category, K is the candidate dimension of the graffiti category, yi∈{0,1}, y' is the label of the overall category, and the overall category has K+1 candidate dimensions, p scrib This indicates the labeled probability map. w represents the predicted labeled probability of the semantic segmentation model on category i. i This represents the loss weight for category i. p seg In the labeling probability of category i, γ represents the hyperparameter used to adjust the sample loss weights.

9. A computer-readable storage medium storing computer-executable instructions, characterized in that, When the computer-executable instructions are executed, the weakly supervised semantic segmentation method for balancing graffiti annotation preferences as described in any one of claims 1 to 5 is implemented.

10. A data processing apparatus comprising a computer-readable storage medium as claimed in claim 9, wherein when a processor of the data processing apparatus retrieves and executes computer-executable instructions in the computer-readable storage medium, it performs weakly supervised semantic segmentation that balances graffiti annotation preferences.

Images