A pixel-level pseudo-label generation and image semantic segmentation system

Abstract

The application discloses a pixel-level pseudo-label generation system, comprising a classification subsystem, an adversarial erasing discrimination subsystem and a foreground saliency correction subsystem; the classification subsystem is used for generating pixel-level pseudo-labels for input images; the adversarial erasing discrimination subsystem is used for constructing background loss functions and foreground loss functions respectively to perform adversarial training on the classification subsystem; the foreground saliency correction subsystem is used for generating foreground prediction maps and constructing saliency loss functions; the classification subsystem is further used for constructing global loss functions according to the background loss functions, the foreground loss functions and the saliency loss functions to perform training constraints on model parameters, so as to update the pixel-level pseudo-labels. The application further discloses an image semantic segmentation system. The technical scheme disclosed by the application improves the expression ability of pseudo-labels on the overall structure of a target, reduces the error allocation probability between adjacent target regions, suppresses the foreground and background aliasing, and improves the accuracy of image semantic segmentation.

Description

Technical Field

[0001] This invention belongs to the field of deep neural network technology, and in particular relates to a pixel-level pseudo-label generation and image semantic segmentation system. Background Technology

[0002] Semantic segmentation is a crucial task in computer vision, aiming to assign a category label to each pixel in an image, thereby segmenting image regions based on these category labels. This technology has wide applications in various fields, including autonomous driving, robot navigation, image editing, and augmented reality. With the development of convolutional neural networks (CNNs), significant progress has been made in fully supervised semantic segmentation in recent years. However, training a fully supervised semantic segmentation network requires pixel-level labels, which necessitates manually labeling each pixel of the image and assigning a category label, thus consuming substantial time and resources.

[0003] To save the manpower and time required for manual pixel-level labeling, much research in recent years has focused on weakly supervised semantic segmentation. Weakly supervised semantic segmentation essentially replaces manual work with deep learning models, using these models to annotate images pixel by pixel. The goal of weakly supervised semantic segmentation is to use only low-cost, weak annotations (such as image-level annotations, doodles, points, or bounding boxes) as supervision, generating pixel-level pseudo-labels by training a deep learning model. Image-level labels only provide information about the categories present in the image, lacking information about the location and contours of those categories.

[0004] Most existing weakly supervised semantic segmentation methods based on image-level labels rely on feature maps that visually represent the model's focus on key features of different image categories. However, due to the nature of image-level labels, the generated feature maps typically only cover the most discriminative regions of the image. Specifically, models trained with image-level labels lacking positional and contour constraints can only distinguish the regions with the most class-specific features, lacking the ability to discriminate complete class regions. Therefore, existing weakly supervised semantic segmentation still faces several technical problems, including: the class localization map only covers a small portion of the target object region; misclassification occurs at class boundaries in images covering multiple categories; and misclassification occurs between class and non-class regions. In summary, the core challenge of existing weakly supervised semantic segmentation methods based on image-level labels is the contradiction between generated pseudo-labels and target integrity. The core reasons are the defects in pseudo-label generation methods, the accumulation of self-training errors caused by pseudo-label noise, and the lack of structural information due to the lack of spatial and boundary supervision.

[0005] Therefore, how to gradually narrow the semantic gap between image-level supervision and pixel-level segmentation by constructing an accurate pseudo-label generation network, fusing multi-source information, and implementing a robust training mechanism is the main technical problem that this invention urgently needs to solve. Summary of the Invention

[0006] To address the technical problems in existing technologies, such as only covering highly discriminative local regions of the target object, making it difficult to fully depict the overall structure of the target object, and the blurring of spatial boundaries between different categories, as well as significant foreground / background confusion between target category regions (i.e., foreground) and non-target regions (i.e., background), this invention proposes a pixel-level pseudo-label generation and image semantic segmentation system. This system effectively alleviates the problem of incomplete coverage of target regions for each category in existing weakly supervised semantic segmentation methods based on image-level labels, improves the expressive power of the generated pixel-level pseudo-labels for the overall structure of the target, reduces the probability of misassignment of category labels between adjacent target regions, and makes the generated pixel-level pseudo-labels more accurate in foreground region localization and background region suppression. This improves the reliability and robustness of pseudo-labels during weakly supervised semantic segmentation training, thereby further enhancing the accuracy of image semantic segmentation based on the generated pixel-level pseudo-labels.

[0007] To achieve the above objectives, the present invention provides a pixel-level pseudo-label generation system, comprising: a classification subsystem, an adversarial erasure discrimination subsystem, and a foreground saliency correction subsystem; The classification subsystem is used to extract features from the input image, aggregate the semantic features of the input image according to the image-level labels, generate corresponding category activation maps for each target category, predict the category label for each pixel, and generate pixel-level pseudo-labels for the input image. The adversarial erasure discrimination subsystem is used to erase the foreground region pixels and background region pixels of the input image according to the category activation map of one of the target categories, and generate a foreground mask map and a background mask map accordingly. The adversarial erasure discrimination subsystem is further configured to perform adversarial prediction on the foreground mask and the background mask respectively, and construct background loss function and foreground loss function respectively using the corresponding prediction results; The foreground saliency correction subsystem is used to generate a saliency map of the input image and calculate the similarity between the category activation map of each target category and the saliency map one by one. The foreground saliency correction subsystem is also used to merge the category activation maps of all similar target categories to generate a foreground prediction map; The foreground saliency correction subsystem is further configured to construct a saliency loss function based on the pixel distance between the foreground prediction map and the saliency map; The classification subsystem is further configured to construct a global loss function based on the background loss function, the foreground loss function, and the saliency loss function, and to use the global loss function to constrain and train the model parameters of the classification subsystem in order to update the pixel-level pseudo-labels.

[0008] Furthermore, the adversarial erasure discrimination subsystem includes a mask generator, a foreground eraser, a background eraser, and an adversarial discriminator; wherein, The mask generator is used to randomly find the activation region of the category activation map of a target category as a foreground mask; and to flip the foreground mask to generate a background mask. The foreground eraser is used to erase corresponding pixels in the input image using the foreground mask to generate a foreground mask image; The background eraser is used to erase corresponding pixels in the input image using the background mask to generate a background mask image; The adversarial discriminator is used to extract features and perform classification prediction on the foreground mask image, and the prediction result obtained is the background response value; The adversarial discriminator is also used to extract features and perform classification prediction on the background mask image, and the prediction result is the foreground response value.

[0009] In a preferred embodiment, the background loss function is used to force the classification subsystem to penalize residual target features in the background region during training in order to find pixel values ​​belonging to the erased target category in the foreground mask image; the foreground loss function is used to force the classification subsystem to apply classification consistency constraints to the foreground region of the category activation map extracted from the background mask image during training.

[0010] Furthermore, the adversarial erasure discrimination subsystem is also used to perform adversarial prediction on the foreground mask and the background mask respectively, and to construct a background loss function and a foreground loss function using the corresponding prediction results. Specifically, this includes: using the prediction result generated by the adversarial erasure discrimination subsystem on the foreground mask, performing cross-entropy calculation with the background reference label of the foreground mask to construct the background loss function; and using the prediction result generated by the adversarial erasure discrimination subsystem on the background mask, performing cross-entropy calculation with the foreground reference label of the background mask to construct the foreground loss function.

[0011] In a preferred embodiment, the foreground saliency correction subsystem includes: a saliency map generator; the saliency map generator includes a saliency extraction model; the saliency map generator is used to pre-train the saliency extraction model based on a saliency dataset, and after the pre-training is completed, transmit the input image to the saliency extraction model for inference to obtain a saliency map of the input image.

[0012] Furthermore, the foreground saliency correction subsystem further includes: a similarity filter; the similarity filter is used to calculate the intersection-union ratio (IUR) between the saliency map of the input image and the class activation maps of each target category one by one, and compare the IUR with a preset confidence threshold to obtain the similarity value between the class activation maps of each target category and the saliency map of the input image.

[0013] Furthermore, the foreground saliency correction subsystem further includes: a saliency foreground synthesizer; the saliency foreground synthesizer is used to synthesize the foreground prediction map based on the similarity value, the category activation maps of all target categories of the input image and their corresponding labels.

[0014] Compared with the prior art, the present invention has at least the following advantages and technical effects: This invention first proposes a pixel-level pseudo-label generation system. On one hand, by introducing an adversarial erasure discrimination mechanism consisting of a foreground eraser and a background eraser, bidirectional constraints are applied to the high-discriminative regions and non-target regions of the target category during training. This enables the model to suppress local over-response while mining the potential discriminative regions of the target object, thereby effectively alleviating the problem of incomplete target region coverage in existing weakly supervised semantic segmentation methods based on class activation maps and improving the ability of pixel-level pseudo-labels to express the overall structure of the target. On the other hand, this invention introduces foreground loss functions and background loss functions to the foreground mask image and background mask image respectively, so that different target categories form clearer discrimination boundaries in space. This reduces the mutual interference of category responses in the boundary region in multi-category coexistence scenarios, thereby reducing the probability of misassignment of category labels between adjacent target regions and improving the category consistency and discriminativeness of pixel-level pseudo-labels. Furthermore, this invention introduces a foreground saliency correction subsystem, which effectively suppresses the confusion between target category regions and non-target background regions by constraining the spatial consistency between the category activation map and the saliency information of the input image. This makes the generated pixel-level pseudo-labels more accurate in terms of foreground region localization and background region suppression, thereby improving the reliability and robustness of pseudo-labels in the weakly supervised semantic segmentation training process.

[0015] Furthermore, based on the aforementioned pixel-level pseudo-label generation system, this invention proposes an image semantic segmentation system. It utilizes the pixel-level pseudo-label generation system to convert input images in the training set that only contain image-level labels into pixel-level pseudo-labels with spatial semantic information. This allows image-level labeled image data, which would otherwise be unsuitable for direct semantic segmentation training, to participate in model optimization in a near-fully supervised manner, significantly reducing the dependence of semantic segmentation tasks on manual pixel-level annotation. By training a fully supervised semantic segmentation model under the supervision of the generated pixel-level pseudo-labels, the segmentation model can simultaneously learn semantic discriminative information and spatial structure information of the target category, thereby improving the model's accuracy and generalization ability when performing semantic segmentation prediction on arbitrary input images during the actual inference stage. Therefore, this invention effectively reduces data annotation costs while ensuring segmentation performance, improving the feasibility and practical value of weakly supervised semantic segmentation systems in real-world applications.

[0016] In summary, this invention can generate high-quality pixel-level pseudo-labels using image data containing only image-level labels during the training phase. This allows for fully supervised training of the semantic segmentation model using pixel-level pseudo-labels, significantly reducing reliance on manual pixel-level annotation while ensuring semantic segmentation accuracy and stability. It is particularly suitable for application scenarios where large-scale image data is difficult to annotate finely or where annotation costs are high, including fields such as autonomous driving and intelligent medical image analysis. Attached Figure Description

[0017] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a schematic diagram of a pixel-level pseudo-tag generation system provided in Embodiment 1 of the present invention; Figure 2 This is a schematic diagram of the structure of the adversarial erasure discrimination subsystem provided in Embodiment 1 of the present invention; Figure 3 This is another structural schematic diagram of the pixel-level pseudo-tag generation system provided in Embodiment 2 of the present invention; Figure 4 This is a schematic diagram of the image semantic segmentation system provided in Embodiment 3 of the present invention; Figure 5 This is a visual comparison chart of pixel-level pseudo-tags and real tags generated step by step by the pixel-level pseudo-tag generation system provided in Embodiment 4 of the present invention; Figure 6 This is a visual comparison diagram of pixel-level pseudo-labels, semantic segmentation maps, and real segmentation results generated by the image semantic segmentation system provided in Embodiment 5 of the present invention. Detailed Implementation

[0018] It should be noted that, without causing conflict, the embodiments and their apparent features mentioned in this application can be flexibly combined. To illustrate the content of this application in detail, the following will use accompanying drawings and specific embodiments.

[0019] It should be noted that the steps in the flowchart shown in the attached figure can be executed by a computer system, specifically through a series of computer-executable instructions. Furthermore, although the flowchart illustrates the logical execution order of the steps, in some cases, the execution order of these steps may differ from the order shown or described.

[0020] Example 1 Figure 1 This is a schematic diagram of a pixel-level pseudo-tag generation system provided in an embodiment of the present invention.

[0021] In specific implementation, the pixel-level pseudo-tag generation system provided in this embodiment of the invention, such as... Figure 1 As shown, it includes: a classification subsystem 100, an adversarial erasure discrimination subsystem 200, and a foreground salience correction subsystem 300.

[0022] The classification subsystem 100 is used to extract features from the input image, aggregate the semantic features of the input image according to the image-level labels, generate corresponding category activation maps for each target category, predict the category label for each pixel, and generate pixel-level pseudo-labels for the input image.

[0023] The adversarial erasure discrimination subsystem 200 is used to erase the foreground region pixels and background region pixels of the input image according to the category activation map of one of the target categories, and generate a foreground mask map and a background mask map accordingly.

[0024] The adversarial erasure discrimination subsystem 200 is further configured to perform adversarial prediction on the foreground mask and the background mask respectively, and construct background loss function and foreground loss function respectively using the corresponding prediction results.

[0025] The foreground saliency correction subsystem 300 is used to generate a saliency map of the input image and calculate the similarity between the category activation map of each target category and the saliency map.

[0026] The foreground saliency correction subsystem 300 is also used to merge the category activation maps of all similar target categories to generate a foreground prediction map.

[0027] The foreground saliency correction subsystem 300 is further configured to construct a saliency loss function based on the pixel distance between the foreground prediction map and the saliency map.

[0028] The classification subsystem 100 is further configured to construct a global loss function based on the background loss function, the foreground loss function, and the saliency loss function, and to use the global loss function to constrain the model parameters of the classification subsystem 100 in order to update the pixel-level pseudo-labels.

[0029] In practical implementation, the corresponding category activation graphs generated for each target category can be represented as a set A = {A}. i | i ∈[1,C]}, where C represents the total number of labels for the target category, i.e., the total number of categories that classification subsystem 100 can identify for the target object; A i Then the corresponding number i A category activation map for each target category. Image-level labels can be represented as... , representing the image-level label vector of the entire original image. For target category i The image-level label indicates the first i Does the target category exist in the original image? (i.e., "0" means not present, "1" means present).

[0030] Depend on Figure 1 It can be seen that category activation map A1 and category activation map A C The activation regions are different, indicating that the pixel regions of the foreground predicted by the classification subsystem 100 for the first target category and the Cth target category are different and may overlap or be confused. Therefore, relying solely on the classification subsystem 100 is far from sufficient to generate high-precision pixel-level pseudo-labels. To this end, embodiments of the present invention further introduce an adversarial erasure discrimination subsystem 200 and a foreground saliency correction subsystem 300 to train, constrain, and optimize the classification subsystem 100.

[0031] The adversarial erasure discrimination subsystem 200 randomly selects a category activation map of a target category, uses the activation region of the category activation map as the foreground region, and generates a foreground mask and a background mask respectively. Then, it erases the pixels in the foreground region and the background region of the input image respectively, generating a foreground mask map and a background mask map accordingly. Further, it performs adversarial prediction on the foreground mask map and the background mask map, and uses the corresponding prediction results to construct a background loss function and a foreground loss function respectively. The foreground saliency correction subsystem 300 calculates the similarity between the category activation map of each target category and the saliency map through a specific similarity calculation method, merges all similar category activation maps to generate a foreground prediction map, and constructs a saliency loss function based on the pixel distance between the foreground prediction map and the saliency map.

[0032] Figure 2 A schematic diagram of one embodiment of the adversarial erasure discrimination subsystem 200 is provided.

[0033] In a preferred embodiment, the adversarial erasure discrimination subsystem 200 includes a mask generator 201, a foreground eraser 202, a background eraser 203, and an adversarial discriminator 204. The mask generator 201 randomly selects an activation region of the activation map of a target category as a foreground mask; and flips the foreground mask to generate a background mask. Specifically, the target category... i The corresponding foreground mask is marked as ; and the foreground mask Flip the object to generate the target category. i The corresponding background mask ( The foreground refers to the region in the input image that belongs to the target category. In multi-label classification tasks, the foreground can be the overlapping region corresponding to multiple target categories; the background refers to the region other than the region occupied by the target category. It should be noted that, in this embodiment, the activated region of the category activation map is used as the pixel region corresponding to the foreground.

[0034] The foreground eraser 202 is used to erase corresponding pixels in the input image using the foreground mask to generate a foreground mask image.

[0035] The background eraser 203 is used to erase corresponding pixels in the input image using the background mask to generate a background mask image.

[0036] In a preferred embodiment, the foreground eraser 202 is used to generate a foreground mask image according to the following equation:

[0037] The background eraser 203 is used to generate a background mask image according to the following equation:

[0038] in, For target category i Foreground mask; For target category i The corresponding background mask; I is the input image; This refers to the element-wise multiplication operation between matrices. To erase the target category on the input image I i The foreground mask image following the corresponding pixels; To retain only the target category on the input image I i The background mask image corresponding to the foreground pixels.

[0039] In practice, the mask only has two values: "0" or "1". In the foreground mask, since pixels in the foreground region need to be erased, the most prominent activation region in the currently randomly selected category activation map is considered the foreground region of the current target category. Therefore, the pixels at the corresponding positions in this region should be set to zero in the foreground mask (other positions are set to "1" to be retained), and the corresponding background mask should be set to "1" (other positions are set to "0" to be erased). Therefore, the foreground mask is a spatial mask generated based on the category activation map, used to indicate the foreground pixel region in the input image that is related to the target category; the background mask is the complement of the foreground mask, used to indicate the background pixel region in the input image that is unrelated to the target category.

[0040] The adversarial discriminator 204 is used to extract features and perform classification prediction on the foreground mask image, and the prediction result obtained is the background response value; The adversarial discriminator 204 is also used to extract features and classify the background mask image, and the obtained prediction result is the foreground response value.

[0041] Weakly supervised semantic segmentation models suffer from knowledge confusion in the boundary regions of adjacent categories when segmenting multi-class images. Existing techniques typically employ foreground erasure methods that simultaneously erase high-response regions of all categories or consistently erase the category with the strongest activation. However, simultaneously erasing high-response regions of all categories leads to a large loss of semantic information, especially in scenes with dense targets, which can easily introduce noisy supervision. On the other hand, consistently erasing the category with the strongest activation results in the category with the strongest activation (e.g., large targets) being repeatedly focused on during training, while small targets or weakly responsive categories are not adequately learned over time.

[0042] The embodiments of the present invention employ a strategy of randomly selecting only a single foreground category, the purpose of which is to guide the classification model of the classification subsystem 100 to focus on learning the features of a single category in the image each time during training iterations.

[0043] Therefore, the mask generator 201 of this embodiment randomly selects a target category. i The activation regions of the category activation map are used as foreground masks. The foreground mask is then flipped to generate a background mask. This erasure strategy eliminates interference from other category information when the model learns the features of the target category. In this embodiment, a foreground mask is constructed randomly for only one target category at a time, while the remaining categories retain their original visual information. This reduces interference from other category information and effectively avoids training instability caused by excessive erasure. Furthermore, randomly selecting a target category ensures that each category has a chance to be erased during training, thereby increasing the learning probability of weak categories and small targets, and helping to alleviate class imbalance and uneven activation strength.

[0044] Furthermore, since fixed erasure strategies make it easy for models to remember erasure patterns and form path dependencies on specific regions, while randomly selecting target categories and dynamically changing foreground positions and shapes in different input images can force classification models to complete category predictions under different contextual conditions, thereby enhancing the robustness of the model to category context and discrimination cues.

[0045] In summary, the single-target-category random erasure strategy designed in this embodiment of the invention can avoid problems such as excessive loss of semantic information and category interference caused by simultaneous erasure of multiple categories. At the same time, it reduces the dominant role of strong response categories in the model training process, allowing different target categories to obtain more balanced optimization opportunities during training, thereby improving the robustness of the model to multi-category scenarios and further improving the overall quality of the generated pixel-level pseudo-labels in terms of category coverage completeness and category discrimination.

[0046] In a preferred embodiment, the adversarial erasure discrimination subsystem 200 is further configured to perform adversarial prediction on the foreground mask image and the background mask image respectively, and construct a background loss function and a foreground loss function respectively using the corresponding prediction results, specifically including: Using the prediction result generated by the adversarial erasure discrimination subsystem 200 through the foreground mask image, cross-entropy is calculated between it and the background reference label of the foreground mask image to construct the background loss function; and, Using the prediction results generated by the adversarial erasure discrimination subsystem 200 through the background mask image, the foreground reference label of the background mask image is cross-entropy calculated to construct the foreground loss function.

[0047] In this embodiment, the background loss function is used to force the classification subsystem 100 to penalize the target features remaining in the background region during training, so as to find the pixel value belonging to the erased target category in the foreground mask map. The foreground loss function is used to force the classification subsystem 100 to apply classification consistency constraints to the foreground regions of the category activation maps extracted from the background mask map during training.

[0048] In practice, by fusing the category activation map of the target category with the features of each region of the input image, a foreground mask map and a background mask map are synthesized, which are then input into the adversarial discriminator 204 to calculate the foreground loss function and background loss function required to achieve adversarial training of the classification subsystem 100.

[0049] In practical implementation, the background loss function calculated using cross-entropy is:

[0050] Where C is the total number of labels for all target categories; To erase the first i After feature extraction and prediction by the adversarial discriminator 204, the foreground mask image following the foreground pixels of the nth target category outputs the nth... i Background response values ​​corresponding to each target category; To erase the first i The background baseline label of the foreground mask image after each target category's foreground pixels, and ,in, The image-level label vector representing the entire original image. For target category i The image-level label indicates whether the i-th target category exists in the original image (i.e., "0" means not present, and "1" means present). It is a non-linear function used to non-linearly map input values ​​to the interval [0, 1].

[0051] In practical implementation, the background response value The raw score (Logit) of the output is calculated for the adversarial discriminator 204. (Nonlinear function) The preferred activation function is the Sigmoid activation function or a Sigmoid activation function constrained by a temperature coefficient, which will reduce the background response value. Mapped to the probability value interval [0, 1], such that It can indicate that there are still objects belonging to the target category in the remaining background area. i The probability of a pixel. In practical applications, Other nonlinear functions, such as Softmax, Sparsemax, and ReLU, can be used depending on the distribution of the dataset or the choice of classification model. However, experiments have shown that the Sigmoid activation function and the Sigmoid activation function constrained by the temperature coefficient are more suitable for the requirements of the classification subsystem 100 provided in this embodiment of the invention, effectively suppressing noise and better conforming to the statistical significance of the loss function. It is the natural logarithm function with base e. When calculating the gradient through backpropagation based on the loss function, taking the logarithm of the probability value can implement a non-linear penalty mechanism where the more wrong the prediction, the greater the penalty. This forces the model to quickly correct severe prediction biases in the early stages of training. Furthermore, it can transform the probability multiplication into a summation operation, making model training more stable and efficient. It also avoids gradient vanishing during training, making the model training and optimization process smoother.

[0052] Adversarial discriminator 204, used to determine the background loss function The classification subsystem 100 is forced to penalize residual target features in the background region during training, in order to improve the foreground mask map. Find the category that belongs to the target category i The pixel value.

[0053] Accordingly, the foreground loss function calculated using cross-entropy is:

[0054] in, The output of the background mask image after feature extraction and prediction by the adversarial discriminator is the first... i Foreground response values ​​for each target category; The foreground reference label is the background mask image, and , For target category i Image-level labels; other symbols are the same as the background loss function mentioned above. The symbols in the cross-entropy function are defined in the same way.

[0055] In this embodiment, the label is set to This is to give the adversarial discriminator 204 a very clear signal: to only discriminate image-level labels at a time. The single target category is selected based on its features in the input image; other categories are not allowed, thus achieving single-target category discrimination. The design principle of single-target category discrimination has been explained in the introduction to the mask generator 201 and will not be repeated here. Foreground loss function. Nonlinear functions in The selection of the background loss function mentioned above For its nonlinear function The selection scheme and its principle are the same, and will not be repeated here. It is the natural logarithm function, and its core function is the same as that of the background loss function mentioned above, so it will not be repeated here.

[0056] In specific implementation, the adversarial discriminator 204 is also used to determine the foreground loss function. This forces the classification subsystem 100 to train on the target category. i The classification consistency constraint is applied to the foreground regions extracted from the category activation map. The activation regions of the category activation map extracted by the classification subsystem 100 are considered foreground regions by the classification subsystem 100; however, for the adversarial discriminator 204, the foreground information labeled by this activation region may be local foreground or inaccurate foreground. If the adversarial discriminator 204 identifies the activation region of the current target category i as another category or fails to identify it, it indicates that the foreground region extracted by the classification subsystem 100 is inconsistent with the original image label. The adversarial discriminator 204 utilizes... The resulting gradient-forced classification subsystem 100 corrects the target category. i The logic for generating the category activation graph is used to adjust the model weight parameters of the classification subsystem 100.

[0057] Compared with the prior art, the pixel-level pseudo-tag generation system provided in this invention has many beneficial effects, including: (1) The foreground erasure method is used to cover the active regions in the image that are identified as foreground categories by the classification subsystem 100, so that the image retains only non-target category information. The adversarial discriminator 204 checks whether the target category information still exists in the image to determine whether the classification subsystem 100 has the ability to distinguish complete categories. During the training process, the judgment of the adversarial discriminator 204 will force the classification subsystem 100 to improve its classification ability as much as possible and identify more complete target category regions. The background erasure method is used to cover all information except for the foreground region, so that the input image retains only target category information. The adversarial discriminator 204 determines whether only the target category exists in the image, prompting the classification subsystem 100 to reduce the misidentification of non-target categories. This embodiment of the invention proposes a combination of foreground erasure and background erasure, which takes into account the classification incompleteness and misclassification problems of the classification subsystem 100. The adversarial erasure discriminator 200 prevents the classification subsystem 100 from over-learning irrelevant content in the learning of the corresponding category while the classification subsystem 100 learns new category knowledge, effectively improving the quality and coverage of pseudo-labels.

[0058] (2) Foreground erasure weakens the influence of the most significant region of the target category features, forcing the classification subsystem 100 to learn information that has a low target category feature response but belongs to the target category, thereby expanding the target category region based on the incomplete target category; background erasure forces the classification subsystem 100 to reduce the coverage of non-category regions during the target category expansion process, reducing the model's misjudgment problem; through adversarial training between foreground erasure and background erasure, the classification subsystem 100 can reach a balance state and generate relatively complete and clean pixel-level pseudo-labels.

[0059] (3) The foreground saliency correction subsystem 300 introduces a saliency map, which provides more accurate foreground information and contours. By judging the category activation map of each category and combining it with image-level labels, categories that truly exist, have obvious features, and fit well with the foreground region of the saliency map are selected. The category activation maps of these categories are then fused to generate a foreground prediction map. The saliency map is used as supervision to correct the classification subsystem 100's erroneous judgments on the boundaries and guide the classification subsystem 100 in learning knowledge between the foreground and background boundaries. The foreground saliency correction subsystem 300 improves the model's ability to distinguish pixels in the boundary region, provides more reliable guidance for the synthesis of category activation maps, and further improves the accuracy and generation efficiency of semantic segmentation pseudo-labels.

[0060] Example 2 Figure 3 This is a schematic diagram of another embodiment of the pixel-level pseudo-label generation system provided by the present invention. Based on the aforementioned embodiment 1, this embodiment further provides a specific implementation of the adversarial erasure discrimination subsystem 200 and the foreground saliency correction subsystem 300, and the implementation of training constraints on the classification subsystem 100.

[0061] For the adversarial erasure discrimination subsystem 200, the mask generator 201 further includes: a target category random filter 2011 and a foreground mask synthesizer 2012.

[0062] The target category random filter 2011 is used to randomly select a target category to be erased from a known set of target categories; The foreground mask synthesizer 2012 is used to binarize the pixel values ​​of the category activation map corresponding to the target category to be erased according to the following equation to generate a foreground mask:

[0063] Among them, P x,y Is the target category i The corresponding category is the pixel value at coordinates (u, v) in the activation map; θ It is the binarization threshold; Is the target category i The mask value at coordinates (u, v) in the corresponding foreground mask; For target category i The foreground mask. H and W are the height and width of the input image, respectively.

[0064] like Figure 3As shown, the foreground discrimination path and the background discrimination path transmit opposite mask images, but share the same adversarial discriminator 204. That is, when performing discrimination prediction on the foreground mask image and the background mask image respectively, the model weight parameters of the adversarial discriminator 204 are the same or share a set of weight parameters. This is the key to maintaining semantic space consistency. When training the classification subsystem 100, the background discrimination path continuously pushes the activation region represented by the foreground mask outward, attempting to cover the entire object; the foreground discrimination path continuously pulls the activation region represented by the forward mask inward, attempting to maintain semantic purity. The dual-path adversarial mechanism provided in this embodiment of the invention constructs a semantically complementary game loop by using a shared adversarial discriminator 204 with frozen parameters as the gold standard: the background discrimination path uses the recognition of residual information after erasing salient features to generate outward push, inducing the classification model to mine a more comprehensive object contour and improve recall; while the foreground discrimination path uses the classification accuracy of isolated feature regions to generate inward pull, forcing the model to eliminate irrelevant background noise and improve accuracy. The dynamic balance between the two ensures that the category activation map generated by the classification subsystem 100 can fully cover the target while maintaining rigorous semantic purity.

[0065] In practice, the background discrimination path for foreground erasure utilizes the background loss function L. bg This forces the model to learn and allocate attention to regions where target category features are not obvious, improving the model's ability to identify regions of the complete foreground category. A category is randomly and uniformly selected from the set of real-world categories labeled in the image at the image level as the target category to be erased. i (The corresponding tags are) y i ), using foreground mask Remove the selected target category predicted by the current classification subsystem 100 from the original input image. i The most prominent activation region, if the adversarial discriminator 204 can still assign a target category at this time. i Background response value A high score indicates a foreground mask. The identified activation regions are not comprehensive enough; during training, a background loss function L needs to be applied. bg Backpropagation along path 2 forces the target category i The corresponding category activation map (CAM) is expanded to find categories belonging to the target category. i The remaining, less significant feature regions. Conversely, when the image input to the adversarial discriminator 204 is a background mask, i.e., the background of the original input image is erased, theoretically the background mask only contains the area covered by the foreground mask or only pixels containing the target category. However, if the adversarial discriminator 204 cannot assign a target category... i Foreground response value A high score, or a score assigned to other categories, indicates that the current classification subsystem 100 has incorrectly identified the activation region or that it contains too much background noise. Therefore, by adjusting the foreground loss function L... fg Backpropagation along path 3 forces the classification subsystem 100 to further refine its output class activation map by continuing to train it, thus eliminating non-target pixels that misclassify the system.

[0066] It should be noted that when training the classification subsystem 100, the adversarial discriminator 204 provided in this embodiment does not need to participate in the training process of the classification subsystem 100; the classification subsystem 100 and the adversarial discriminator 204 are trained independently. Preferably, the adversarial discriminator 204 has a built-in artificial neural network model, which is pre-trained using massive amounts of image data, and its model parameters remain unchanged after pre-training. Therefore, for the classification subsystem 100, the adversarial discriminator 204 is a discriminator with shared prior knowledge and frozen parameters. Regardless of whether the image has had its foreground or background erased, they are ultimately mapped to the same semantic space for classification prediction, and sharing the same adversarial discriminator 204 with fixed parameters ensures the objectivity of the evaluation criteria. In specific implementations, the deep neural network model built into the adversarial discriminator 204 can be configured as a deep convolutional neural network (CNN) or a Transformer model.

[0067] Furthermore, in the pixel-level pseudo-label generation system provided in this embodiment of the invention, residual feature recognition is used to drive foreground expansion, while local feature purity is used to drive foreground convergence. Under this mechanism, the foreground saliency correction subsystem 300 provided in this embodiment of the invention is particularly important because it affects the foreground loss function. and background loss function In addition to the game theory, a saliency map is introduced as a visual geometric constraint to prevent the game imbalance from causing the foreground area to be too large and overflow into the background or too small and get stuck in the local foreground area.

[0068] Specifically, the foreground saliency correction subsystem 300 includes a saliency map generator 301. The saliency map generator 301 includes a saliency extraction model. In specific implementations, the saliency map generator 301 is used to pre-train the saliency extraction model based on a saliency dataset, and after pre-training, transmits the input image to the saliency extraction model for inference to obtain the saliency map of the input image. In specific implementations, a class-agnostic neural network model is preferably used as the saliency extraction model built into the saliency map generator 301. By finding a more complete target contour in the geometric visual dimension, the saliency map generator 301 enables the saliency extraction model to possess independent visual common sense after pre-training on a completely different large dataset (saliency dataset), thus serving as an external error correction mechanism to prevent the classification subsystem 100 from getting trapped in local optima. Preferably, a CNN model with large model parameters but rich boundary details (such as U2-Net) or a CNN model with fast inference speed suitable for preprocessing large-scale datasets (such as PoolNet) can be used to implement the saliency extraction model. Alternatively, a self-supervised or unsupervised visual attention (ViT) model that does not require dedicated dataset labels can be used.

[0069] Furthermore, the foreground saliency correction subsystem 300 also includes a similarity filter 302.

[0070] The similarity filter 302 is used to calculate the intersection-union ratio (IUR) between the saliency map of the input image and the class activation maps of each target category, and compare the IUR with a preset confidence threshold to obtain the similarity value between the class activation maps of each target category and the saliency map of the input image. Specifically, the similarity filter 302 obtains the similarity value between the class activation maps of each target category and the saliency map S of the input image using the following equation:

[0071] In the above formula, For target category i The category activation map, where S is the saliency map of the input image. For target category i Category activation graph Similarity value between the saliency map S and the saliency map S; The preset confidence threshold; This is a binarization function used to perform element-wise binarization of the saliency map and the category activation map. Specifically, the saliency map S is compared with the category activation maps of each target category. The intersection-union ratio between them is , The function computes the elements of the input tensor X at coordinates (u, v) as follows:

[0072] in, This represents the element values ​​of the input tensor X at coordinates (u, v). This is the intersection / merge threshold; Indicates the target category i Category activation graph Binarize the elements at each position; similarly, This involves binarizing the elements at each position in the saliency map S. Calculating the intersection-union ratio (IUR) between these elements reveals the similarity between the saliency map S and the activation maps of each class. Higher similarity indicates a larger overlap, i.e., a higher IUR. Therefore, an IUR can be preset by setting an IUR threshold. To control the boundaries of similarity judgments between them. Preferably, set .

[0073] Furthermore, such as Figure 3 As shown, the foreground salience correction subsystem 300 further includes a salience foreground synthesizer 303.

[0074] The salient foreground synthesizer 303 is used to synthesize the foreground prediction map based on the similarity value, the category activation maps of all target categories in the input image, and their corresponding labels. Specifically, for the similarity value... Category activation graph of all target categories and its corresponding tags The foreground prediction map is synthesized using the following equations. :

[0075] Where C is the total number of labels for all target categories in the dataset; For target category i Tags; For target category i Category activation graph; For target category i The similarity value between the category activation map and the saliency map; This refers to the foreground prediction map. In specific implementation, the label... Ensure that only the categories that actually exist in the graph are activated by the corresponding category graph. Participating in synthesis, and The semantic response strength and similarity value extracted by the model were preserved. Equivalent to similarity weighting, it spatially corrects the activation regions predicted by the classification subsystem 100. When the overall correction effect of the activation maps of all C categories is combined, a more accurate foreground prediction map is generated. .

[0076] In a preferred embodiment, the significance loss function is:

[0077] in, S represents the foreground prediction map; S represents the saliency map of the input image; H represents the height of the foreground prediction map; and W represents the width of the foreground prediction map. (Mathematical symbols) The square of the second norm is used to calculate the foreground prediction map. The squared Euclidean distance between the target image and the saliency map S. Using the squared Euclidean distance minimizes the gradient distance as the image approaches the target during training, contributing to convergence stability and tending to reduce the average error across the entire image rather than directly reducing pixel values ​​to zero. For example... Figure 3 As shown, during the training process, the saliency loss function L sal The data is then transmitted back along path 4 to the classification subsystem 100 for training. Furthermore, this embodiment takes into account the design of the three newly added loss functions mentioned above, as well as the original classification loss function of the classification subsystem 100. A novel global loss function is proposed to provide a more comprehensive and integrated training constraint for the classification subsystem 100.

[0078] Specifically, in the pixel-level pseudo-label generation system provided in this embodiment, the classification subsystem 100 uses the following global loss function to constrain the model parameters during training:

[0079] In the above formula, The original classification loss function of the classification subsystem is... The significance loss function; Foreground loss function; Use the background loss function; , , and The weights for each loss function are listed in order. Let be the global loss function.

[0080] In practical implementation, the original classification loss function of the base layer To ensure the model's most basic classification ability, it serves as the source of all localization information, guaranteeing the semantic meaning of each category's activation map, and directly feeding back to the classification subsystem 100 along path 1; the adversarial enhancement layer... Achieve adversarial erasure and uncover global features. Responsible for extrapolating to discover non-significant but necessary features; Responsible for inward refining features; spatial prior layer generated. The geometric shape of the category activation map is corrected by introducing visual common sense to achieve saliency correction, and the region found by the classification subsystem 100 is ensured to conform to the outline of the object in the physical world by using an external reference model.

[0081] Through the above four weights ( , , The allocation of the four loss function components adjusts their influence, optimizing the quality of generated pixel-level pseudo-labels during the training of the classification subsystem 100. When actually deploying this global loss function, a phased weighting strategy is preferred. Specifically, a high weighting is maintained in the early stages of training. First, let the classification subsystem 10 develop basic classification capabilities. At this stage, although the category activation map is not accurate enough, the semantic direction is correct. During the mid-training phase, gradually increase... and By introducing saliency constraints and background erasure, the model is forced to shift from viewing "points" to viewing "surfaces," thus completing the object's outline; in the later stages of training, high... Finally, feature extraction is performed to ensure that the generated foreground pseudo-labels have clear edges and no background residue. It's not just a combination of four formulas; it actually defines an automated pixel-level labeler. Through the mutual influence of the four loss functions, the classification model in classification subsystem 100 can ultimately produce high-quality pixel-level labels even with only image-level category labels.

[0082] Original classification loss function As the foundation of weakly supervised semantic segmentation, supervision enables the model to understand semantics and drives it to generate initial activation regions with class discriminative capabilities. However, image-level labels lack specific image contours and boundary information, resulting in sparse and inaccurate foreground regions generated by models using only classification loss functions. This leads to a discussion of classification loss functions... To address the existing deficiencies, the adversarial erasure discrimination subsystem 200 and the foreground saliency correction subsystem 300 proposed in this embodiment can provide two new training constraints for the classification subsystem 100.

[0083] The adversarial erasure discriminant subsystem 200 achieves accurate extraction and spatial localization of target features through a dual-path collaborative optimization mechanism of foreground erasure and background erasure. The foreground erasure path masks identified highly activated regions (such as the center of an object), forcing the model to expand its feature learning to non-significantly activated regions of the target category (such as object edges), thereby improving the completeness of target coverage. The background erasure path, on the other hand, effectively reduces category interference information in the background region by suppressing feature responses from non-target categories, ensuring the model focuses on feature extraction of the target category. Foreground erasure drives the model to deeply mine target category regions, while background erasure maintains the purity of target category features. During this alternating optimization process, the model gradually converges to the optimal state, ultimately generating pseudo-labels that not only have higher spatial coverage but also maintain accurate category segmentation.

[0084] The foreground saliency correction subsystem 300 provides boundary and foreground information for the classification model of the classification subsystem 100, and provides constraints for foreground expansion of the classification model. This embodiment of the invention fully utilizes the complementary characteristics between category activation maps and saliency maps: category activation maps can provide semantic localization information for different categories in an image, but their spatial resolution is low and their boundaries are blurry, making it difficult to accurately delineate object outlines; saliency maps, on the other hand, can effectively capture visually prominent foreground object regions and have clearer spatial boundaries, but lack specific semantic category information.

[0085] This invention innovatively combines the advantages of these two feature maps: on the one hand, it utilizes the semantic discriminativeness of the category activation map to distinguish the approximate regions of different categories; on the other hand, it leverages the spatial accuracy of the saliency map to refine object boundaries, compensating for the insufficient localization of the category activation map in boundary regions through the spatial detail of the saliency map. This complementary fusion strategy significantly improves the quality of pseudo-labels, enabling them to possess both accurate semantic classification capabilities and refined spatial segmentation effects. Therefore, the pixel-level pseudo-label generation system provided by this invention can effectively improve the quality of pseudo-labels.

[0086] Example 3 Furthermore, based on the pixel-level pseudo-label generation system provided in the above embodiments, this invention further provides an image semantic segmentation system.

[0087] like Figure 4 The diagram shown is a structural schematic of an embodiment of the image semantic segmentation system provided by the present invention.

[0088] Specifically, the image semantic segmentation system provided in this embodiment includes: any of the aforementioned pixel-level pseudo-label generation systems 10, and a fully supervised semantic segmentation system 20.

[0089] The pixel-level pseudo-label generation system 10 is used to access each input image of the training set and its corresponding image-level label, and generate corresponding pixel-level pseudo-labels for each input image. The fully supervised semantic segmentation system 20 is used to access each input image of the training set, train the semantic segmentation model under the supervision of the pixel-level pseudo-labels, and use the trained semantic segmentation model to perform semantic segmentation prediction on any input image.

[0090] In this embodiment, by cascading the pixel-level pseudo-label generation system 10 with the fully supervised semantic segmentation system 20, training data with only image-level labels can participate in the training of the semantic segmentation model in a near-fully supervised manner, thereby reducing the cost of manual annotation while improving the accuracy and stability of semantic segmentation prediction results.

[0091] Example 4 This embodiment, based on the pixel-level pseudo-label generation system in Embodiment 2 above, further provides a more specific implementation example of a pixel-level pseudo-label generation system. Specifically, this embodiment conducts extensive comparative experiments based on two standard datasets, PASCAL VOC 2012 and MSCOCO 2014, and generates high-quality pixel-level pseudo-labels.

[0092] The PASCAL VOC 2012 dataset contains 21 categories (20 foreground (target class) and 1 background class), using 10,582 images for training, 1,464 for validation, and 1,456 for testing. The MS COCO 2014 dataset contains 81 categories (80 foreground and background classes), using 82,081 images for training and 40,137 for validation after selection. All input images were uniformly adjusted to a resolution of 512×512×3 (height×width×channels), and the same preprocessing workflow was used. Mean IoU (mean Intersection over Union, mIoU) was used as the primary evaluation metric. PASCAL VOC 2012 contains a rich variety of everyday objects and is suitable for scene understanding tasks with medium-sized categories; while MS COCO 2014 supports finer-grained object recognition, featuring more complex scenes and diverse object combinations. Both datasets provide comprehensive benchmarks for validating algorithm performance.

[0093] The pixel-level pseudo-label generation system for semantic segmentation proposed in this embodiment adopts an end-to-end training process. During model training, the input image is first subjected to feature extraction by the classification subsystem 100, simultaneously generating class activation maps and class prediction probabilities. Subsequently, the adversarial erasure discriminant subsystem 200 randomly selects target classes based on image-level labels, performs operations to preserve and erase target regions on the image, and uses an adversarial discriminator 204 to determine the authenticity of the processed image. Simultaneously, the foreground saliency correction subsystem 300 selects high-confidence class activation maps for fusion, generates a foreground prediction map, and then uses the saliency map for spatial alignment optimization. Experiments show that the model performance of the classification subsystem 100 tends to stabilize after 20 rounds of iterative training. During the inference phase, the classification model in the trained classification subsystem 100 first generates class activation maps, which are then converted into pixel-level pseudo-labels after thresholding.

[0094] As shown in Table 1, the pixel-level pseudo-label generation system proposed in this embodiment demonstrates excellent performance on the PASCAL VOC 2012 training set. Specifically, the pixel-level pseudo-label generation system, without any post-optimization processing, achieves a semantic segmentation pixel-level pseudo-label accuracy of 72.8% mIoU. After processing with a dense conditional random field (CRF), the accuracy is further improved by 2.2% mIoU; and after optimization using an affinity network (such as AffinityNet), the accuracy is improved by another 0.6% mIoU. The pixel-level pseudo-labels generated in this embodiment surpass the accuracy of existing models.

[0095] Table 1. Experimental results of pseudo-labels generated on the PASCAL VOC 2012 training set.

[0096] This embodiment further conducted ablation experiments to investigate the impact of each system component. As shown in Table 2, the results of these ablation experiments fully validate the effectiveness of the innovative system components in the pixel-level pseudo-label generation system proposed in this embodiment. Specifically, using foreground erasure alone improves the pseudo-label accuracy by 8.6% mIoU compared to the baseline model (47.4% mIoU); using the adversarial erasure discrimination subsystem 200 alone (i.e., foreground erasure + background erasure) brings a significant accuracy improvement of 11.1% mIoU. When the adversarial erasure discrimination subsystem 200 and the foreground saliency correction subsystem 300 (i.e., "foreground correction" in Table 2) are integrated simultaneously, the pseudo-label accuracy reaches 72.8% mIoU, an improvement of 25.4% mIoU compared to the baseline model, achieving a significant leap in accuracy. These data fully demonstrate the synergistic effect of the various technical components in this embodiment.

[0097] Table 2. Ablation experiment results of pixel-level pseudo-label generation accuracy based on PASCAL VOC 2012 training set.

[0098] Furthermore, this embodiment also addresses the confidence threshold used by the similarity filter 302 in the foreground saliency correction subsystem 300. Ablation experiments were conducted for verification. Confidence threshold. It is a hyperparameter not obtained through learning and training; therefore, its value usually depends on extensive experimental verification rather than being obtained by training a model.

[0099] Table 3 shows the confidence thresholds in the prospect saliency correction subsystem 300. The impact of different values ​​on semantic segmentation performance. This hyperparameter This method is used to filter similar or qualified foreground categories, avoiding the systematic error problem of saliency maps. Experiments show that when When the value is 0.4, the pixel-level classification model performs optimally and generates pseudo-labels of higher quality.

[0100] Table 3 Confidence Thresholds Results of pseudo-tag ablation experiments in PASCAL VOC 2012

[0101] In addition to the experimental results presented above, which show objective segmentation accuracy data, this embodiment of the invention also includes a comparison of the subjective effects of visualized pseudo-label generation.

[0102] like Figure 5 The image shown is a visual comparison of pixel-level pseudo-tags and real tags generated step by step by the pixel-level pseudo-tag generation system provided in this embodiment of the invention.

[0103] in, Figure 5 The first row represents different samples of the input images in the PASCAL VOC 2012 dataset; the second row represents the overall overlay of activation maps of various categories generated by the baseline model for the input images in the first row (i.e., the activation regions containing all predicted categories); the third row represents the overall overlay of activation maps of categories generated by using only foreground erasure in the model training of classification subsystem 100; the fourth row represents the overall overlay of activation maps of categories generated after adding adversarial erasure (foreground erasure + background erasure) to optimize the model training of classification subsystem 100; the fifth row represents the overall overlay of activation maps of categories generated by the pixel-level pseudo-label generation system used in this embodiment; and the last row represents the real pixel-level label map corresponding to each input image.

[0104] The advantages of the embodiments of the present invention compared to the benchmark model are very obvious. Specifically, from Figure 5 As can be observed, the category activation map generated by the baseline model trained with image-level labels only activates the foreground region, which is the most discriminative category in the image. The region classified by the classification subsystem 100 is incomplete and sparse, making it difficult to achieve accurate semantic segmentation. Foreground erasure can guide the system to discover more detailed and complete image semantic information; however, the system using only foreground erasure suffers from misclassification of the background region during the foreground region expansion process. A system that uses both foreground and background erasure can provide supervision for the expansion process to stop, thus mitigating the misclassification problem of the background region. When foreground correction is added to the system, the pixel-level pseudo-label generation system provided in this embodiment can further learn the foreground boundary knowledge in the input image, generating pixel-level pseudo-labels with higher accuracy, more complete foreground regions, and closer to the real labels.

[0105] Example 5 Based on Example 3, this embodiment proposes a specific implementation method for the overall image semantic segmentation system.

[0106] Furthermore, this embodiment utilizes the pixel-level pseudo-labels generated in Embodiment 4 as the experimental output of the first stage of the image semantic segmentation system, and uses them as the input signal (pixel-level supervision signal) for the second stage of the image semantic segmentation system. This input signal is then transmitted to the image semantic segmentation system for final semantic segmentation, thereby training the semantic segmentation model within the system. The trained semantic segmentation model no longer requires pixel-level pseudo-labels and can perform semantic segmentation reasoning on any input image. This is similar to the implementation process of conventional fully supervised semantic segmentation, but the architecture of this embodiment is more efficient and achieves higher accuracy.

[0107] Specifically, this embodiment uses DeepLab-v1 / v2 architecture models based on ResNet-101, ResNet-38, and VGG-16 backbones as the semantic segmentation models in the fully supervised semantic segmentation system 20. The effectiveness of pixel-level pseudo-label supervision is verified through fully supervised training of the semantic segmentation models. Experimental results on general datasets such as PASCAL VOC2012 and MSCOCO2014 demonstrate that this embodiment achieves a significant improvement in semantic segmentation performance.

[0108] To comprehensively evaluate the quality of the generated semantic segmentation pseudo-labels, this embodiment conducted in-depth experimental verification on the PASCAL VOC 2012 and MSCOCO 2014 datasets. Using the generated pixel-level pseudo-labels as supervisory signals, fully supervised semantic segmentation models in the fully supervised semantic segmentation system 20, based on the DeepLab-v1 and DeepLab-v2 architectures, were trained respectively. ResNet-38, ResNet-101, and VGG-16 backbone networks were selected as classification models in the classification subsystem 100 of the pixel-level pseudo-label generation system 10. Ablation experiments were then conducted, and the experimental results are summarized in Table 4 for performance comparison.

[0109] The experimental results, shown in Table 4, demonstrate that under the same network architecture, our system significantly outperforms the benchmark model. Specifically, on the PASCAL VOC 2012 dataset, the VGG-16-based DeepLab-v1 model achieves segmentation accuracies of 68.9% mIoU and 70.4% mIoU on the validation and test sets, respectively; while the ResNet-101-based DeepLab-v2 model achieves segmentation accuracies of 73.3% mIoU and 72.7% mIoU, respectively, with excellent average accuracy and pixel accuracy of 83.4% and 93.5%, respectively. In Table 4, the backbone network refers to the backbone network (classification model) of the classification subsystem in the pixel-level pseudo-label generation system; the network model refers to the semantic segmentation model in the fully supervised semantic segmentation system.

[0110] Table 4 Performance evaluation of the image semantic segmentation system based on PASCAL VOC 2012

[0111] On the more challenging MS COCO 2014 dataset, as shown in Table 5, the DeepLab-v2 models based on ResNet-101 and VGG-16 achieved segmentation accuracies of 40.4% and 43.1% mIoU, respectively, on the MS COCO 2014 validation set. In Table 5, "backbone network" refers to the built-in classification model of the classification subsystem in the pixel-level pseudo-label generation system, and "network model" refers to the semantic segmentation model in the fully supervised semantic segmentation system.

[0112] Table 5 Performance evaluation of the image semantic segmentation system based on the MS COCO 2014 validation set

[0113] The pixel-level pseudo-labels generated in this embodiment are used to train the semantic segmentation model in the fully supervised semantic segmentation system 20 in a fully supervised manner. They show excellent performance in various indicators such as average intersection-union ratio, average accuracy and pixel accuracy on the validation set (Val.), which fully proves that the proposed pixel-level pseudo-label generation system and image semantic segmentation system have strong generalization ability and practical value.

[0114] Figure 6 This demonstrates a visualization of the supervised learning effect of the semantic segmentation system 20 using generated pixel-level pseudo-labels. Specifically, in this embodiment, multiple samples of the input image on the PASCAL VOC 2012 dataset are used... Figure 6 It is displayed in the first line of the document. Figure 6 The second row of images consists of pixel-level pseudo-label images generated for each input image in the first stage, which are used for fully supervised semantic segmentation in the second stage. Figure 6 The third row of images is the semantic segmentation map generated by the semantic segmentation model trained with pixel-level pseudo-labels in the second stage; Figure 6 The last line shows the actual semantic segmentation result of the input image.

[0115] In summary, both the objective experimental data and the visualization results demonstrate that the image semantic segmentation system provided in this embodiment, after training based on the aforementioned pixel-level pseudo-labels, can effectively generate high-precision semantic segmentation pseudo-labels while significantly reducing the workload of manual annotation, ultimately achieving high semantic segmentation accuracy. Therefore, this invention can efficiently complete large-scale weakly labeled image semantic segmentation tasks.

[0116] It should be noted that the above content is merely a preferred embodiment of this application and does not imply that the scope of protection of this application is limited to this. Any equivalent modifications or alternatives made by those skilled in the art within the technical framework disclosed in this application, based on existing technical knowledge, should fall within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope defined in the appended claims.

Claims

1. A pixel-level pseudo-tag generation system, characterized in that, include: Classification subsystem, adversarial erasure discrimination subsystem, foreground salience correction subsystem; The classification subsystem is used to extract features from the input image, aggregate the semantic features of the input image according to the image-level labels, generate corresponding category activation maps for each target category, predict the category label for each pixel, and generate pixel-level pseudo-labels for the input image. The adversarial erasure discrimination subsystem is used to erase the foreground region pixels and background region pixels of the input image according to the category activation map of one of the target categories, and generate a foreground mask map and a background mask map accordingly. The adversarial erasure discrimination subsystem is further configured to perform adversarial prediction on the foreground mask and the background mask respectively, and construct background loss function and foreground loss function respectively using the corresponding prediction results; The foreground saliency correction subsystem is used to generate a saliency map of the input image and calculate the similarity between the category activation map of each target category and the saliency map one by one. The foreground saliency correction subsystem is also used to merge the category activation maps of all similar target categories to generate a foreground prediction map; The foreground saliency correction subsystem is further configured to construct a saliency loss function based on the pixel distance between the foreground prediction map and the saliency map; The classification subsystem is further configured to construct a global loss function based on the background loss function, the foreground loss function, and the saliency loss function, and to use the global loss function to constrain and train the model parameters of the classification subsystem in order to update the pixel-level pseudo-labels.

2. The pixel-level pseudo-tag generation system according to claim 1, characterized in that, The adversarial erasure discrimination subsystem includes a mask generator, a foreground eraser, a background eraser, and an adversarial discriminator; wherein, The mask generator is used to randomly find the activation region of the category activation map of a target category as a foreground mask; and to flip the foreground mask to generate a background mask. The foreground eraser is used to erase corresponding pixels in the input image using the foreground mask to generate a foreground mask image; The background eraser is used to erase corresponding pixels in the input image using the background mask to generate a background mask image; The adversarial discriminator is used to extract features and perform classification prediction on the foreground mask image, and the prediction result obtained is the background response value; The adversarial discriminator is also used to extract features and perform classification prediction on the background mask image, and the prediction result is the foreground response value.

3. The pixel-level pseudo-tag generation system according to claim 1, characterized in that, The background loss function is used to force the classification subsystem to penalize the residual target features in the background region during training, so as to find the pixel value belonging to the erased target category in the foreground mask map. The foreground loss function is used to force the classification subsystem to apply classification consistency constraints to the foreground regions of the category activation maps extracted from the background mask map during training.

4. The pixel-level pseudo-tag generation system according to claim 1, characterized in that, The adversarial erasure discrimination subsystem is further configured to perform adversarial prediction on the foreground mask and the background mask respectively, and construct background loss functions and foreground loss functions respectively using the corresponding prediction results, specifically including: Using the prediction result generated by the adversarial erasure discrimination subsystem from the foreground mask image, cross-entropy is calculated between it and the background reference label of the foreground mask image to construct the background loss function; and, The foreground loss function is constructed by using the prediction result generated by the adversarial erasure discrimination subsystem of the background mask image and the foreground reference label of the background mask image to perform cross-entropy calculation.

5. The pixel-level pseudo-tag generation system according to claim 1, characterized in that, The foreground saliency correction subsystem includes: a saliency map generator; the saliency map generator includes a saliency extraction model; The saliency map generator is used to pre-train the saliency extraction model based on a saliency dataset, and after the pre-training is completed, transmit the input image to the saliency extraction model for inference to obtain the saliency map of the input image.

6. The pixel-level pseudo-tag generation system according to claim 5, characterized in that, The foreground saliency correction subsystem further includes: a similarity filter; The similarity filter is used to calculate the intersection-union ratio (IUR) between the saliency map of the input image and the class activation maps of each target category, and compare the IUR with a preset confidence threshold to obtain the similarity value between the class activation maps of each target category and the saliency map of the input image.

7. The pixel-level pseudo-tag generation system according to claim 6, characterized in that, The foreground saliency correction subsystem further includes: a saliency foreground synthesizer; The salient foreground synthesizer is used to synthesize the foreground prediction map based on the similarity value, the category activation maps of all target categories in the input image, and their corresponding labels.

8. The pixel-level pseudo-tag generation system according to claim 1, characterized in that, The significance loss function is: in, S is the foreground prediction map; S is the saliency map of the input image; H represents the height of the foreground prediction map; and W represents the width of the foreground prediction map.

9. The pixel-level pseudo-tag generation system according to claim 1, characterized in that, The classification subsystem uses the following global loss function to constrain the model parameters during training: in, The original classification loss function of the classification subsystem is... The significance loss function; Foreground loss function; Use the background loss function; , , and The weights for each loss function are listed in order. Let be the global loss function.

10. An image semantic segmentation system, characterized in that, The image semantic segmentation system includes: a pixel-level pseudo-label generation system as described in any one of claims 1 to 9, and a fully supervised semantic segmentation system; The pixel-level pseudo-label generation system is used to access each input image of the training set and its corresponding image-level label, and generate corresponding pixel-level pseudo-labels for each input image. The fully supervised semantic segmentation system is used to access each input image in the training set, train the semantic segmentation model under the supervision of the pixel-level pseudo-labels, and use the trained semantic segmentation model to perform semantic segmentation prediction on any input image.

Images