Image Weakly Supervised Semantic Segmentation Method Based on High-Quality Local Regions

Abstract

This invention discloses a weakly supervised semantic segmentation method for images based on high-quality local regions, comprising: 1) randomly generating diverse local regions; 2) quantifying and measuring local regions using foreground proportion and confidence level to select high-quality local regions such as target object edges and regions difficult for the model to identify; 3) constructing positive sample pairs with the same foreground category and large background differences, and negative sample pairs with different foreground categories and similar backgrounds based on high-quality local regions, and using a triplet loss constraint model to bring the class probabilities of positive samples closer and push away the class probabilities of negative samples further away, thereby improving the model's complete attention to the target object region. This invention proposes a systematic process for the generation, evaluation, selection, and use of local regions, which can fully mine and utilize the information in local regions and effectively improve the performance of weakly supervised semantic segmentation.

Description

Technical Field

[0001] This invention relates to the technical field of weakly supervised semantic segmentation and metric learning, and in particular to a weakly supervised semantic segmentation method for images based on high-quality local regions. Background Technology

[0002] Semantic segmentation, as a crucial computer vision task, has a wide range of applications in the real world, such as road scene segmentation in autonomous driving, map region segmentation in remote sensing, pathological image segmentation in medicine, robot vision, image cutout in video software, and virtual try-on functionality in e-commerce software. However, the difficulty in obtaining pixel-level segmentation labels required for training semantic segmentation tasks hinders their application in many real-world scenarios. Weakly supervised semantic segmentation, as a weakly supervised training task, can train semantic segmentation models using only image-level category labels, greatly reducing the cost of applying semantic segmentation to real-world scenarios and making it easier for machines to "see" and understand the world.

[0003] However, due to the difference in supervision granularity between image-level labels and pixel-level labels, weakly supervised semantic segmentation models typically only focus on the most discriminative regions of the target object, lacking complete coverage of the target object. To address the problems caused by training semantic segmentation models based on image-level labels, existing methods introduce local regions during model training to assist model learning, emphasizing complete coverage of the target object. This effectively improves the model's focus on the complete target object region and demonstrates the effectiveness and importance of local regions in improving the performance of weakly supervised semantic segmentation tasks. However, existing methods lack diversity in the generated local regions, cannot guarantee the quality of local regions, have inaccurate local region labels, and do not fully utilize the information in the local regions, thus failing to fully explore the local regions to better improve the performance of weakly supervised semantic segmentation. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings and deficiencies of the prior art and propose a weakly supervised semantic segmentation method for images based on high-quality local regions. This method can generate diverse local regions, achieve quantitative evaluation of the quality of local regions, set reasonable labels for local regions, and make full use of the information in local regions for metric learning. This maximizes the role of local regions in the weakly supervised semantic segmentation task, improves the model's ability to fully recognize target objects, and thus improves the segmentation accuracy of weakly supervised semantic segmentation.

[0005] To achieve the above objectives, the technical solution provided by this invention is: a weakly supervised semantic segmentation method for images based on high-quality local regions, comprising the following steps:

[0006] 1) Generate a local region candidate set: Given a training dataset, randomly generate the position and size data of rectangular local regions within the size range of each image in the training dataset to produce a local region candidate set;

[0007] 2) Evaluate and screen the local regions to obtain high-quality local regions: Use a classifier network to calculate the class activation map of the local regions, calculate the pseudo-segmentation labels of the local regions and the class confidence of each label class based on the class activation map, calculate the proportion of each class in the local region using the pseudo-segmentation labels, take the class with the largest proportion as the local region class, and take the corresponding proportion as the foreground proportion evaluation value of the local region, take the corresponding class confidence as the confidence evaluation value of the local region, and select high-quality local regions according to the foreground proportion evaluation value and the confidence evaluation value;

[0008] 3) Use high-quality local regions: Use the triplet loss in contrastive learning to constrain the class probabilities of the local regions; The selection criteria for positive and negative sample pairs of the triplet loss are: When an anchor sample is determined, from the local regions in the same training batch, select those with the same foreground class and large background differences as positive samples, and at the same time select those with different foreground classes and similar backgrounds as negative samples; Use the triplet loss to constrain the class probabilities of the anchor and positive samples to be similar, and at the same time constrain the class probabilities of the anchor and negative samples to be dissimilar.

[0009] Furthermore, in step 1), given a training dataset D, for each image i in the training dataset D, randomly generate J rectangular local regions that fall within the image size range. These local regions have random positions and aspect ratios, but should satisfy: When the upper left endpoint of the image is placed at the origin of the two-dimensional coordinate axis, and the length and width on the x-axis and y-axis are h and w respectively, for any local region j, j = 0, 1, 2,..., J, its length should satisfy h j < h, and the width should satisfy w j < w, and the coordinates should satisfy:

[0010]

[0011] where x lt 、y lt 、x rb 、y rb respectively represent the abscissa of the upper left endpoint of the local region, the ordinate of the upper left endpoint, the abscissa of the lower right endpoint, and the ordinate of the lower right endpoint;

[0012] Subsequently, initially screen according to the size and aspect ratio, that is, given a minimum size requirement s and a maximum aspect ratio r, filter out local regions with a size less than s and an aspect ratio less than 1 / r or greater than r to form a local region candidate set.

[0013] Furthermore, step 2) includes the following steps:

[0014] 2.1) Use a classifier network to compute the class activation map of the local region;

[0015] 2.2) Based on the category activation map, calculate the foreground percentage of each category and the category confidence of each category in the local region, as follows:

[0016] First, pseudo-segmentation labels, or pseudo-labels, are calculated for local regions based on the category activation map. The formula for calculating the pseudo-label for each pixel position is as follows;

[0017]

[0018] In the formula, Represents pixel p t The pseudo-label; k represents the k-th category, with a value ranging from 1 to K, and there are a total of K categories; Pixel p in the category activation graph represents the k-th category. t The category activation value of the location; This means iterating through all categories and selecting the category with the largest X value as the pseudo-label.

[0019] After obtaining the pseudo-labels, we can calculate the proportion of each foreground category in the local region. Here, the foreground category refers to all the categories of interest in the given dataset D, and the background category is the opposite of the foreground category. The formula for calculating the foreground proportion of category k is as follows:

[0020]

[0021] In the formula, fgratio k T represents the foreground proportion of category k, ranging from [0,1]; k This represents the number of pixels in the local region whose pseudo-label is category k; T is the total number of pixels in the local region.

[0022] The category confidence score for each category is calculated based on the category activation graph, using the following formula:

[0023]

[0024] In the formula, confid k This represents the category confidence score of the k-th category; This indicates that all cases from 0 to T are traversed and accumulated;

[0025] 2.3) Determine the local region category and local region evaluation value: Take the category with the largest foreground proportion as the local region category, take the proportion corresponding to the category as the foreground proportion evaluation value of the local region, and take the category confidence as the confidence evaluation value of the local region.

[0026] 2.4) When selecting local regions, selection can be based on the foreground proportion assessment value, the confidence level assessment value, or a combination of both, thereby obtaining high-quality local regions. A selection quantity p is set; when selecting based on the foreground proportion assessment value, the local region is selected according to the foreground proportion assessment value, i.e., fgratio. k Sort the regions and select the p middle local regions; when selecting based on the confidence score, sort the local regions according to the confidence score, i.e., confidence. k Sort the regions and select the p middle local areas. When making a selection based on the combined result of foreground proportion and confidence assessment value, first use the following formula to obtain the combined assessment value:

[0027] overall k =λ*fgratio k +(1-λ)*confid k

[0028] In the formula, overall k The comprehensive evaluation value is represented by λ, which is a weighting coefficient ranging from [0,1] and determines the ratio between the evaluation value of the prospect of use and the evaluation value of confidence.

[0029] After obtaining the overall evaluation value, the local area is divided according to the overall evaluation value. k Sort the data and select the p middle local regions;

[0030] After sorting the local regions according to their evaluation values, the method for obtaining the p middle local regions is as follows: for all J values ​​in a training batch... batch There are p local regions, with the middle p local regions as targets, and each on both sides... The nth local region is not the target; therefore, for all sorted local regions, obtain the nth... One to the first A local region; among which, J batch ≥p; This is the floor symbol.

[0031] Furthermore, step 3) includes the following steps:

[0032] 3.1) Calculate the background feature representation of the local region:

[0033]

[0034] In the formula, F bg Background feature representation; T bg This represents the number of background pixels in the local region. This indicates that the value of t is traversed and accumulated from 0 to T. bg All cases; Represents pixel p in the feature map t Feature representation of location;

[0035] 3.2) Construct positive and negative sample pairs for triplet loss: Traverse each local region in the current training batch, using the local region as an anchor point, and based on the background feature representation calculated in step 3.1), select local regions with the same foreground category and large background differences as positive samples; select local regions with different foreground categories and similar backgrounds as negative samples; wherein, the background feature representation of the local region is a one-dimensional vector, and the similarity between feature representations can be measured using Euclidean distance or cosine distance;

[0036] 3.3) Calculate the class probability of local regions: Use a classifier network to calculate the class probability of local regions; if the trained network is a classifier network, the classification layer of the network is directly used to predict the class probability output based on the feature representation of the local region; if it is a non-classifier network, the classification layer can be connected after the features are extracted from the backbone network of the model, and a classification prediction task is added to train this layer of the network.

[0037] 3.4) The triplet loss is used to constrain the class probabilities of anchor points and positive samples to be similar, and the class probabilities of anchor points and negative samples to be dissimilar. This triplet loss is used to assist in constraining model training. The formula for calculating the triplet loss is as follows:

[0038] loss(d ap ,d an ) = max(d ap -d an +margin,0)

[0039] In the formula, loss(d ap ,d an ) represents the triplet loss; d ap d an , respectively, represent the Euclidean distance between the class probabilities of positive and negative sample pairs; margin is an adjustment term that controls the difference in distance between positive and negative sample pairs; the max function indicates taking the larger of the two.

[0040] Furthermore, the classifier network is any neural network classifier trained using a given training dataset D or a pre-trained neural network classifier.

[0041] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0042] 1. The random local region generation strategy of the present invention can ensure the diversity of local regions, thereby being able to cope with diverse scenarios in the real world, such as objects of different sizes, different aspect ratios, and objects of different sizes at different distances.

[0043] 2. This invention proposes two methods for quantitatively evaluating the quality of local regions: foreground proportion assessment and confidence assessment, as well as related selection methods. These methods enable reasonable evaluation of local regions and further selection of high-quality local regions such as the edge regions of target objects and regions that are difficult for the model to identify.

[0044] 3. This invention proposes to construct positive and negative samples of anchor points by simultaneously considering the foreground and background categories of local regions. This not only makes full use of the background information in the local regions, but also constructs targeted hard samples for metric learning through the background information, so that the model's attention can focus more on the foreground category of the target.

[0045] 4. This invention proposes a complete process for generating, evaluating, selecting, and using local regions, which can fully mine and utilize information in local regions and effectively improve the performance of weakly supervised semantic segmentation.

[0046] 5. The method based on high-quality local regions proposed in this invention can be encapsulated as a plug-and-play module and applied to other weakly supervised semantic segmentation tasks, thus exhibiting high flexibility.

[0047] 6. The local region evaluation and selection strategy of the present invention is a sampling strategy that can effectively and efficiently sample high-quality local regions for assisting model learning, thereby reducing the total number of local regions used for model learning, reducing the interference of low-quality local regions on model learning, accelerating the convergence speed of the model, reducing the training time of the model, and improving training efficiency. Attached Figure Description

[0048] Figure 1 This is a schematic diagram of the logic flow of the present invention.

[0049] Figure 2 This is a schematic diagram illustrating the implementation of the present invention in the first stage of a weakly supervised semantic segmentation training task to assist in classifier training; ResNet38 is a convolutional network implementation. Detailed Implementation

[0050] The present invention will be further described in detail below with reference to the embodiments and accompanying drawings, but the embodiments of the present invention are not limited thereto.

[0051] The implementation background of this embodiment is as follows: With the development of intelligent technology, various types of robots have been manufactured and applied to various fields, such as picking robots on industrial assembly lines, intelligent service robots in supermarkets or showrooms, and robotic vacuum cleaners in homes. The ability of robots to see and understand the world is a key step in achieving intelligent interaction. Through semantic segmentation tasks, machines can identify and locate people or objects in the real world, thereby completing tasks such as precise obstacle avoidance, garbage collection, and target object grasping. This embodiment uses the real-world general object recognition and segmentation dataset PASCALVOC2012 to illustrate how the method proposed in this invention can perform semantic segmentation of real-world general objects with only image-level labels.

[0052] This embodiment employs a common two-stage training framework. The first stage trains a classification model using an image-level labeled dataset, and generates a class activation map for each image in the training set using the trained model. This class activation map serves as the seed region for the target object. In the second stage, the class activation map is further refined to obtain pseudo-labels, and a semantic segmentation model is trained in a fully supervised manner based on these pseudo-labels. This semantic segmentation model can be used for semantic segmentation of general objects in the real world. This embodiment focuses on using a method based on high-quality local regions for the first stage of classification model training, maximizing the role of local regions through systematic local region generation, evaluation, selection, and utilization.

[0053] like Figure 1 and Figure 2 As shown in the figure, this embodiment discloses a weakly supervised semantic segmentation method for images based on high-quality local regions, which is used to optimize the classifier training in the first stage of the weakly supervised semantic segmentation training task. The specific details are as follows:

[0054] 1) Given a training dataset D, for each image i in the training dataset D, randomly generate J rectangular local regions falling within the image size range, where J = 20. These local regions have random positions and lengths and widths, but should satisfy the following: when the upper left endpoint of the image is placed at the origin of the two-dimensional coordinate axis, and the length and width on the x-axis and y-axis are h and w respectively, the length of any local region j, j = 0, 1, 2, ..., J, should satisfy h j <h, width satisfies w j <w, the coordinates must satisfy:

[0055]

[0056] In the formula, x lt y lt x rb y rbThese represent the x-coordinate of the upper left endpoint, the y-coordinate of the upper left endpoint, the x-coordinate of the lower right endpoint, and the y-coordinate of the lower right endpoint, respectively.

[0057] Subsequently, a preliminary selection process is performed based on size and aspect ratio to form a candidate set of local regions. That is, given a minimum size requirement s and a minimum aspect ratio r, local regions with a size smaller than s and an aspect ratio smaller than r are filtered out. Here, s = 80, meaning that local regions with fewer than 80 pixels are removed, and r = 4 and 1 / 4, meaning that local regions with an aspect ratio greater than 4 or less than 1 / 4 are removed.

[0058] 2) Calculate the category activation map and pseudo-label corresponding to each local region, thereby calculating the category, foreground proportion evaluation value, and confidence evaluation value of the local region, and using the evaluation values ​​to obtain high-quality local regions, including the following steps:

[0059] 2.1) As Figure 2 As shown, the classification model uses ResNet38 as the feature extraction network to generate feature maps. A classification layer and a global average pooling layer are added after the feature extraction network to obtain the class activation map and the model's output class probability, respectively. During the forward propagation process, assuming the input image is i, the feature map of the image is obtained after forward propagation. i and category activation map cam i By scaling proportionally, local regions can be mapped to corresponding positions in the feature map and class activation map, thus obtaining the feature map and class activation map of the local region. Specifically, when the aspect ratio of the image and feature map is xratio... if ,yratio if At that time, the coordinates of the local region on the feature map are:

[0060]

[0061] In the formula, These represent the x-coordinate of the upper left endpoint, the y-coordinate of the upper left endpoint, the x-coordinate of the lower right endpoint, and the y-coordinate of the lower right endpoint of the local region in the feature map.

[0062] When the aspect ratio of the image and the category activation map is xratio ic ,yratio ic At that time, the index in the category activation graph is:

[0063]

[0064] In the formula, These represent the x-coordinate of the top left endpoint, the y-coordinate of the top left endpoint, the x-coordinate of the bottom right endpoint, and the y-coordinate of the bottom right endpoint of the local region in the category activation map.

[0065] 2.2) Calculate the foreground percentage of each category and the category confidence of each category in the local region based on the category activation map, including the following steps:

[0066] 2.2.1) First, the pseudo-segmentation labels of the local region are calculated based on the category activation map, which are referred to as pseudo-labels. The formula for calculating the pseudo-label at each pixel position is as follows;

[0067]

[0068] In the formula, Represents pixel p t The pseudo-label; k represents the k-th category, with a value ranging from 1 to K, and there are a total of K categories; Pixel p in the category activation graph represents the k-th category. t The category activation value of the location; This means iterating through all categories and selecting the category with the largest X value as the pseudo-label.

[0069] 2.2.2) After obtaining the pseudo-labels, we can calculate the proportion of each foreground category in the local region. Here, foreground categories refer to all the categories we are interested in in the dataset. The opposite of foreground categories is the background category. If we use the PASCALVOC2012 dataset, there are 20 foreground categories, i.e., K = 20. The formula for calculating the foreground proportion of category k is as follows:

[0070]

[0071] In the formula, fgratio k T represents the foreground proportion of category k, ranging from [0,1]; k This represents the number of pixels in the local region with the pseudo-label of category k; T is the total number of pixels in the local region.

[0072] 2.2.3) The category confidence score for each category is calculated based on the category activation graph, using the following formula:

[0073]

[0074] In the formula, confid k This represents the category confidence score of the k-th category; This represents iterating through and accumulating all cases where t takes values ​​from 0 to T.

[0075] 2.3) Determine the local region category and local region evaluation value: Take the category with the largest foreground proportion as the local region category, take the proportion corresponding to the category as the foreground proportion evaluation value of the local region, and take the category confidence as the confidence evaluation value of the local region.

[0076] 2.4) Select a local region. Selection can be based on the foreground proportion assessment value, the confidence level assessment value, or a combination of both. Set the selection quantity p. When selecting based on the foreground proportion assessment value, the local region will be selected according to the foreground proportion assessment value, i.e., fgratio. k Sort the regions and select the p middle local regions; when selecting based on the confidence score, sort the local regions according to the confidence score, i.e., confidence. k The selection process involves sorting the regions and choosing the p middle local areas. However, this embodiment selects regions based on a combination of foreground proportion and confidence level assessment. The comprehensive assessment value is first obtained using the following formula:

[0077] overall k =λ*fgratio k +(1-λ)*confid k

[0078] In the formula, overall k λ is the overall evaluation value; λ is the trade-off coefficient, ranging from [0,1], which determines the ratio of the use prospect percentage evaluation value to the confidence evaluation value. In this example, it is set to 0.5.

[0079] After obtaining the overall evaluation value, the local area will be divided according to the overall evaluation value. k Sort and select the first one. One to the first A local region. Among them, J batch J represents the number of local regions in a training batch. batch ≥p; This is the floor sign. In this embodiment, p = 0.3 * J. batch .

[0080] 3) Use local regions to assist classifier training. Constrain the classifier to predict the same class probability for local regions containing the same foreground, and different class probabilities for local regions containing different foregrounds. Simultaneously, fully consider the background information of the local regions when constructing positive and negative pairs. The specific steps are as follows:

[0081] 3.1) Calculate the feature representation of the local region's background:

[0082]

[0083] In the formula, F bg Background feature representation; T bg This represents the number of background pixels in the local region. This indicates that the value of t is traversed and accumulated from 0 to T. bg All cases; Represents pixel p in the feature map t Feature representation of location;

[0084] 3.2) Constructing positive and negative sample pairs: Traverse each local region in the current training batch, using the current local region as an anchor point. Based on the background feature representation calculated in step 3.1), select local regions with the same foreground category and large background differences as positive samples; select local regions with different foreground categories and similar backgrounds as negative samples. The background feature representation of a local region is a one-dimensional vector, and the similarity between feature representations can be measured using Euclidean distance, cosine distance, etc. This embodiment uses Euclidean distance as the similarity measurement method.

[0085] 3.3) Calculate the class probability of local regions: Calculate the class probability of local regions based on their feature maps using the network classification layer and global average pooling layer, such as... Figure 2 The flowchart shows the process of inputting medium-to-high quality local regions into the network for computation.

[0086] 3.4) Using triplet loss, the class probabilities of the anchor point and positive samples are similar, while the class probabilities of the anchor point and negative samples are dissimilar:

[0087] loss(d ap ,d an ) = max(d ap -d an +margin,0)

[0088] In the formula, loss(d ap ,d an ) represents the triplet loss; d ap d an , which represent the Euclidean distance between the class probabilities of positive and negative sample pairs, respectively; margin is an adjustment term that controls the difference in distance between positive and negative sample pairs, which is set to 28 in this embodiment; max function means taking the larger of the two values.

[0089] 3.5) After obtaining the constraint loss based on the local region, use both this loss and the classification loss to optimize the training of the classification model, i.e., the overall loss is:

[0090]

[0091] In the formula, Loss represents the overall loss. cls For classification loss, Loss local For loss based on local regions, i.e., loss(d) ap ,d an); γ is an adjustment coefficient that controls the proportion of the two losses. In this embodiment, the value of γ is set to 0.05. This parameter is affected by the size of the training batch. When there are many samples in each batch, a smaller initial value of γ is taken, and vice versa.

[0092] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A weakly supervised semantic segmentation method for images based on high-quality local regions, characterized in that: Includes the following steps: 1) Generate a local region candidate set: Given a training dataset, randomly generate the location and size data of rectangular local regions within the size range of each image in the training dataset to generate a local region candidate set; 2) Evaluate and filter local regions to obtain high-quality local regions: Use a classifier network to calculate the class activation map of the local region, calculate the pseudo segmentation label of the local region and the class confidence of each label class based on the class activation map, use the pseudo segmentation label to calculate the proportion of each class in the local region, take the class with the largest proportion as the local region class, and take the corresponding proportion as the foreground proportion evaluation value of the local region, take the corresponding class confidence as the confidence evaluation value of the local region, and select high-quality local regions based on the foreground proportion evaluation value and the confidence evaluation value; 3) Use high-quality local regions: Use triple loss in contrastive learning to constrain the class probability of local regions; the selection criteria for positive and negative sample pairs of triple loss are: after determining an anchor sample, select positive samples from local regions of the same training batch that have the same foreground class and large background differences, and select negative samples that have different foreground classes and similar backgrounds; use triple loss to constrain the class probabilities of anchor points and positive samples to be similar, and constrain the class probabilities of anchor points and negative samples to be dissimilar.

2. The image weakly supervised semantic segmentation method based on high-quality local regions according to claim 1, characterized in that, In step 1), given a training dataset D, for each image i in the training dataset D, J rectangular local regions falling within the image size range are randomly generated. These local regions have random positions and lengths and widths, but should satisfy the following: when the upper left endpoint of the image is placed at the origin of the two-dimensional coordinate axis, and the length and width on the x-axis and y-axis are h and w respectively, the length of any local region j, j = 0, 1, 2, ..., J, should satisfy h j <h, width satisfies w j <w, the coordinates must satisfy: Where x lt 、y lt 、x rb 、y rb These represent the x-coordinate of the upper left endpoint, the y-coordinate of the upper left endpoint, the x-coordinate of the lower right endpoint, and the y-coordinate of the lower right endpoint, respectively. Subsequently, preliminary filtering is performed based on size and aspect ratio. That is, given the minimum size requirement s and the maximum aspect ratio r, local regions with a size smaller than s and an aspect ratio less than 1 / r or greater than r are filtered out, forming a local region candidate set.

3. The image weakly supervised semantic segmentation method based on high-quality local regions according to claim 2, characterized in that, Step 2) includes the following steps: 2.1) Use a classifier network to compute the class activation map of the local region; 2.2) Based on the category activation map, calculate the foreground percentage of each category and the category confidence of each category in the local region, as follows: First, pseudo-segmentation labels, or pseudo-labels, are calculated for local regions based on the category activation map. The formula for calculating the pseudo-label for each pixel position is as follows; Where, L pt Represents pixel p t The pseudo-label; k represents the k-th category, with a value ranging from 1 to K, and there are a total of K categories; Pixel p in the category activation graph represents the k-th category. t The category activation value of the location; This means iterating through all categories and selecting the category with the largest X value as the pseudo-label. After obtaining the pseudo-labels, we can calculate the proportion of each foreground category in the local region. Here, the foreground category refers to all the categories of interest in the given training dataset D, and the background category is the opposite of the foreground category. The formula for calculating the foreground proportion of category k is as follows: In the formula, fgratio k T represents the foreground proportion of category k, ranging from [0,1]; k This represents the number of pixels in the local region whose pseudo-label is category k; T is the total number of pixels in the local region. The category confidence score for each category is calculated based on the category activation graph, using the following formula: In the formula, confid k This represents the category confidence score of the k-th category; This indicates that all cases from 0 to T are iterated and accumulated; 2.3) Determine the local region category and local region evaluation value: Take the category with the largest foreground proportion as the local region category, take the proportion corresponding to the category as the foreground proportion evaluation value of the local region, and take the category confidence as the confidence evaluation value of the local region. 2.4) When selecting local regions, selection can be based on the foreground proportion assessment value, the confidence level assessment value, or a combination of both, thereby obtaining high-quality local regions. A selection quantity p is set; when selecting based on the foreground proportion assessment value, the local region is selected according to the foreground proportion assessment value, i.e., fgratio. k Sort the regions and select the p middle local regions; when selecting based on the confidence score, sort the local regions according to the confidence score, i.e., confidence. k Sort the regions and select the p middle local areas. When making a selection based on the combined result of foreground proportion and confidence assessment value, first use the following formula to obtain the combined assessment value: overall k =λ*fgratio k +(1-λ)*confid k In the formula, overall k The comprehensive evaluation value is represented by λ, which is a weighting coefficient ranging from [0,1] and determines the ratio between the evaluation value of the prospect of use and the evaluation value of confidence. After obtaining the overall evaluation value, the local area is divided according to the overall evaluation value. k Sort the data and select the p middle local regions; After sorting the local regions according to their evaluation values, the method for obtaining the p middle local regions is as follows: for all J values ​​in a training batch... batch There are p local regions, with the middle p local regions as targets, and each on both sides... The nth local region is not the target; therefore, for all sorted local regions, obtain the nth... One to the first A local region; among which, J batch ≥p; This is the floor symbol.

4. The image weakly supervised semantic segmentation method based on high-quality local regions according to claim 3, characterized in that, Step 3) includes the following steps: 3.1) Calculate the background feature representation of the local region: In the formula, F bg Background feature representation; T bg This represents the number of background pixels in the local region. This indicates that the value of t is traversed and accumulated from 0 to T. bg All cases; Represents pixel p in the feature map t Feature representation of location; 3.2) Construct positive and negative sample pairs for triplet loss: Traverse each local region in the current training batch, using the local region as an anchor point, and based on the background feature representation calculated in step 3.1), select local regions with the same foreground category and large background differences as positive samples; select local regions with different foreground categories and similar backgrounds as negative samples; wherein, the background feature representation of the local region is a one-dimensional vector, and the similarity between feature representations can be measured using Euclidean distance or cosine distance; 3.3) Calculate the class probability of local regions: Use a classifier network to calculate the class probability of local regions; if the trained network is a classifier network, the classification layer of the network is directly used to predict the class probability output based on the feature representation of the local region; if it is a non-classifier network, the classification layer can be connected after the features are extracted from the backbone network of the model, and a classification prediction task is added to train this layer of the network. 3.4) The triplet loss is used to constrain the class probabilities of anchor points and positive samples to be similar, and the class probabilities of anchor points and negative samples to be dissimilar. This triplet loss is used to assist in constraining model training. The formula for calculating the triplet loss is as follows: loss(d ap ,d an )=max(d ap -d an +margin,0) In the formula, loss(d ap d an ) represents the triplet loss; d ap d an , respectively, represent the Euclidean distance between the class probabilities of positive and negative sample pairs; margin is an adjustment term that controls the difference in distance between positive and negative sample pairs; the max function indicates taking the larger of the two.

5. The image weakly supervised semantic segmentation method based on high-quality local regions according to claim 4, characterized in that, The classifier network is any neural network classifier trained using a given training dataset D or a pre-trained neural network classifier.

Images