Weakly supervised semantic segmentation method, device, electronic device and storage medium

Abstract

The embodiments of the present application disclose a weakly supervised semantic segmentation method, device, electronic device and storage medium. The method includes: obtaining a picture to be identified, inputting the picture to be identified into a semantic segmentation model, and obtaining a semantic segmentation result; the semantic segmentation model is obtained by training a basic semantic segmentation model based on training pseudo-labels; the training pseudo-labels are obtained by a dual-branch model after identifying the picture; the dual-branch model is obtained after iterative training based on a first training label and a second training label; the first training label is an initial label generated by a CAM; the second training label is an online label output by the dual-branch model. The present application trains a dual-branch model through iterative optimization so that it can predict higher-quality object boundaries and segmentation results, and finally generates high-quality pseudo-labels for training the basic semantic segmentation model based on the object boundaries and segmentation results, thereby training a high-precision semantic segmentation model.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a weakly supervised semantic segmentation method, device, electronic device, and storage medium. Background Art

[0002] Semantic segmentation is an important and classic computer vision task with wide applications in image editing, scene analysis, and other fields. Although semantic segmentation based on deep neural networks has made significant progress, these methods rely heavily on time-consuming and labor-intensive pixel-level image segmentation labels.

[0003] To reduce the cost of image annotation, weakly supervised semantic segmentation methods based on image category labels have been widely studied. Currently, most methods train a classification network using category labels, then use the activation map (CAM) of the last convolutional layer of the classification network to obtain the position and shape information of foreground objects, generate initial labels (seed labels), and use these initial labels to train a standard semantic segmentation model. Finally, the trained semantic segmentation model predicts the semantic segmentation results of the image to be identified. Because the foreground region of the activation map is typically only partially highlighted, the initial label generally only marks part of the foreground object, resulting in low foreground category recall, which affects the performance of the segmentation model.

[0004] Currently, some work uses seed labels to train boundary detection models to extract foreground object boundaries (contours). These models then propagate foreground category scores under the constraints of these contours, making the highlighted foreground regions in activation maps more complete. However, the object boundary maps predicted by these boundary detection models contain many false positives (edges within objects), which hinder the propagation of foreground category scores. This results in incomplete highlighted foreground regions in the corrected activation maps and low recall of the initial labels. Summary of the Invention

[0005] Since the existing methods have the above-mentioned problems, the embodiments of the present application provide a weakly supervised semantic segmentation method, device, electronic device and storage medium, focusing on solving the problem of low initial label recall.

[0006] Specifically, the embodiments of the present application provide the following technical solutions:

[0007] In a first aspect, an embodiment of the present application provides a weakly supervised semantic segmentation method, comprising:

[0008] Obtaining a picture to be identified, and inputting the picture to be identified into a semantic segmentation model to obtain a semantic segmentation result of the picture to be identified;

[0009] In which, the semantic segmentation model is obtained by training the basic semantic segmentation model based on the training pseudo-label; the training pseudo-label is obtained after the dual-branch model recognizes the image; the dual-branch model is obtained after iterative training based on the first training label and the second training label; wherein, the first training label is the initial label generated by the classification network activation map CAM; the initial label includes the foreground object position and shape information of the image; the second training label is the online label output by the dual-branch model; the online label is generated based on the semantic segmentation branch prediction result and the object boundary detection branch prediction result; the dual-branch model is composed of the semantic segmentation branch and the object boundary detection branch, and the semantic segmentation branch and the object boundary detection branch share a trunk branch for extracting image features.

[0010] Optionally, the CAM is obtained after a classification network model performs feature recognition on an image; the classification network model is obtained after training based on image category labels.

[0011] Optionally, the training pseudo-label is obtained by recognizing the image using a dual-branch model, including:

[0012] The training pseudo labels are obtained based on the semantic segmentation prediction results obtained after the semantic segmentation branch identifies the image, and the object boundary results obtained after the object boundary detection branch identifies the image.

[0013] Optionally, the dual-branch model is obtained through iterative training based on the first training label and the second training label, including:

[0014] The CAM is processed offline to generate a first training label; under the constraint of the object boundary map generated by the object boundary detection branch, the foreground category score in the initial segmentation probability map generated by the semantic segmentation branch is propagated using a foreground category score propagation method to obtain a revised segmentation probability map, and a second training label is generated based on the revised segmentation probability map;

[0015] Supervise the training of the object boundary detection branch and the semantic segmentation branch in the dual-branch model according to the first training label and the second training label;

[0016] Based on a dense conditional random field (DCR), the initial segmentation probability map is processed to obtain a background reference label, and the second training label is corrected according to the background reference label to obtain a corrected second training label. Based on the first training label and the corrected second training label, the object boundary sub-model in the dual-branch model is supervised and trained.

[0017] Optionally, obtaining the training pseudo label based on a semantic segmentation prediction result obtained after recognizing the image according to the semantic segmentation branch and an object boundary result obtained after recognizing the image according to the object boundary sub-model includes:

[0018] After performing multi-scale scaling and horizontal flipping on the image, the image is input into the trained semantic segmentation branch to obtain a semantic segmentation prediction result, and the image is input into the trained object boundary detection branch to obtain an object boundary result;

[0019] The training pseudo label is generated according to the semantic segmentation prediction result and the object boundary result.

[0020] In a second aspect, an embodiment of the present application provides a weakly supervised semantic segmentation apparatus, comprising:

[0021] A processing module is used to obtain a picture to be identified and input the picture to be identified into a semantic segmentation model to obtain a semantic segmentation result of the picture to be identified;

[0022] In which, the semantic segmentation model is obtained by training the basic semantic segmentation model based on the training pseudo-label; the training pseudo-label is obtained after the dual-branch model recognizes the image; the dual-branch model is obtained after iterative training based on the first training label and the second training label; wherein, the first training label is the initial label generated by the classification network activation map CAM; the initial label includes the foreground object position and shape information of the image; the second training label is the online label output by the dual-branch model; the online label is generated based on the semantic segmentation branch prediction result and the object boundary detection branch prediction result; the dual-branch model is composed of the semantic segmentation branch and the object boundary detection branch, and the semantic segmentation branch and the object boundary detection branch share a trunk branch for extracting image features.

[0023] Optionally, the CAM is obtained after a classification network model performs feature recognition on an image; the classification network model is obtained after training based on image category labels.

[0024] Optionally, the processing module is specifically configured to:

[0025] The training pseudo labels are obtained based on the semantic segmentation prediction results obtained after the semantic segmentation branch identifies the image, and the object boundary results obtained after the object boundary detection branch identifies the image.

[0026] In a third aspect, an embodiment of the present invention further provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the weakly supervised semantic segmentation method as described in the first aspect is implemented.

[0027] In a fourth aspect, an embodiment of the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the weakly supervised semantic segmentation method as described in the first aspect.

[0028] It can be seen from the above technical solution that the embodiment of the present application inputs the picture to be identified into the semantic segmentation model to obtain the semantic segmentation result of the picture to be identified; wherein, the semantic segmentation model is obtained after training the basic semantic segmentation model based on the training pseudo-label; the training pseudo-label is obtained after the dual-branch model identifies the picture; the dual-branch model is obtained after iterative training based on the first training label and the second training label; wherein, the first training label is the initial label generated by the classification network activation map CAM; the initial label includes the foreground object position and shape information of the picture; the second training label is the online label output by the dual-branch model; the online label is generated based on the semantic segmentation branch prediction result and the object boundary detection branch prediction result; the dual-branch model is composed of the semantic segmentation branch and the object boundary detection branch, and the semantic segmentation branch and the object boundary detection branch share a trunk branch for extracting picture features. It can be seen that the embodiment of the present application iteratively optimizes the two branches of the dual-branch model through online labels. During forward propagation, under the constraint of the object boundary, the foreground category score in the segmentation result is propagated to the surrounding area to generate a second training label. The label predicts a more complete and accurate foreground area, and well optimizes the two branches of the dual-branch model during back propagation. Compared with the existing scheme that only uses the initial label (first training label) to supervise the object boundary branch, the embodiment of the present application can effectively suppress false positives in the object boundary (internal boundary of the object) and help the foreground category score to propagate from the salient area to the non-salient area. The embodiment of the present application uses score propagation to optimize the segmentation results of the segmentation sub-model and generate training pseudo-labels. Compared with the traditional CAM-based method, the generated training pseudo-labels are more accurate, so that a higher-performance basic semantic segmentation model can be trained, thereby improving the accuracy of the semantic segmentation results. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0030] Figure 1 This is one of the step flow charts of the weakly supervised semantic segmentation method provided in an embodiment of the present application;

[0031] Figure 2 This is the second step flow chart of the weakly supervised semantic segmentation method provided in the embodiment of the present application;

[0032] Figure 3 This is a framework diagram of the iterative training dual-branch model provided in an embodiment of the present application;

[0033] Figure 4 This is one of the network structure diagrams of the dual-branch model provided in the embodiment of the present application;

[0034] Figure 5 This is the second network structure diagram of the dual-branch model provided in the embodiment of the present application;

[0035] Figure 6 This is the third network structure diagram of the dual-branch model provided in the embodiment of the present application;

[0036] Figure 7 Schematic diagram of the structure of the weakly supervised semantic segmentation device provided in an embodiment of the present application;

[0037] Figure 8 It is a schematic structural diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0038] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0039] Figure 1 FIG1 shows one of the flow charts of the weakly supervised semantic segmentation method provided in an embodiment of the present application. Figure 2 This is the second step flow chart of the weakly supervised semantic segmentation method provided in the embodiment of the present application. Figure 3 This is a framework diagram of the iterative training dual-branch model provided in the embodiment of the present application. Figure 4 This is one of the network structure diagrams of the dual-branch model provided in the embodiment of the present application. Figure 5 This is the second network structure diagram of the dual-branch model provided in the embodiment of the present application. Figure 6 This is the third network structure diagram of the dual-branch model provided in the embodiment of the present application. Figures 1 to 6 The weakly supervised semantic segmentation method provided in the embodiment of the present application is explained and illustrated in detail. Figure 1 As shown, the embodiment of the present application provides a weakly supervised semantic segmentation method, including:

[0040] Step 101: Obtain a picture to be identified, and input the picture to be identified into a semantic segmentation model to obtain a semantic segmentation result of the picture to be identified;

[0041] In which, the semantic segmentation model is obtained by training the basic semantic segmentation model based on the training pseudo-label; the training pseudo-label is obtained after the dual-branch model recognizes the image; the dual-branch model is obtained after iterative training based on the first training label and the second training label; wherein, the first training label is the initial label generated by the classification network activation map CAM; the initial label includes the foreground object position and shape information of the image; the second training label is the online label output by the dual-branch model; the online label is generated based on the semantic segmentation branch prediction result and the object boundary detection branch prediction result; the dual-branch model is composed of the semantic segmentation branch and the object boundary detection branch, and the semantic segmentation branch and the object boundary detection branch share a trunk branch for extracting image features.

[0042] In this step, it should be noted that first, a classification network model needs to be trained using image category labels, such as Figure 3 The backbone weights of the classification network model are initialized using the backbone of the ImageNet pre-trained model. The fully connected layers used for classification have no bias and their weights are randomly initialized. During training, the input images are randomly augmented and fed into the network for training, with SGD optimization used.

[0043] In this step, after the classification network model training is completed, the training image is input into the classification network model. The feature map output by the last convolution layer is F, which is not pooled. The weight of the fully connected layer is converted into a 1×1 convolution kernel to convolve F and the result is input into the Relu activation function to obtain the activation map CAM (Class Activation Map).

[0044] In this step, after obtaining the activation map CAM, the CAM is upsampled to the input image size, and the channels corresponding to the categories that do not appear in the input image are set to 0. A background channel is added to the CAM, and the background channel is set to τ1 to obtain CAM1, and the background channel is set to τ2 (τ2 < τ1) to obtain CAM2. CAM1 and CAM2 are respectively input into the argmax function to obtain two initial labels. The two labels are corrected using the pydensecrf package to obtain Y fg , Y bg . fg The background pixel labels in are modified if they are in Y bg If it belongs to the foreground, it is re-marked as an uncertain pixel (take 255), as shown below (Y init [i] indicates Y init The i-th pixel in Y fg [i],Y bg [i] Similar), for Y fg After modification, the first training label Y is obtained init .

[0045]

[0046] In this step, if Figure 4 As shown, the dual-branch model selects resnet50 or resnet101 as the backbone, changes the stride of the resnet network stage4 and stage5 from 2 to 1, and stage5 finally outputs a feature map F with a stride of 8 s8 , adjust the dilation of the convolutional layers of the network stage4 and stage5 to make F s8 The receptive field of each position is as large as the receptive field of the corresponding position in the original ResNet network. It should be noted that the dual-branch model consists of a semantic segmentation branch (segmentation sub-model) and an object boundary detection branch (object boundary sub-model), such as Figure 5 As shown, the segmentation sub-model is constructed by adding a seg head after stage5. The seg head adopts the Aspp model and consists of four 3×3 convolution kernels with different dilations. s8 The results of the four convolutions are added together and then upsampled by a factor of 2 in the spatial dimension. Finally, softmax is input to obtain the semantic segmentation probability map M. Figure 6As shown in the figure, the object boundary submodule is constructed as follows: the output features of stage1 to stage5 are respectively reduced to 32 channels through 5 edge_layers, the 5 obtained feature maps are concat together and input into edge_layer6 (1×1 convolution), and the output channel is 1 object boundary map contour map. The contour map is mapped to [0,1] by the sigmoid function, which is denoted as B.

[0047] In this step, in addition to the initial label Y init (First training label), in the dual-branch model training stage, the segmentation result M and the object boundary B are also used to generate the online label Y online (Second training label). The training image is randomly scaled, cropped and enhanced before being input into the dual-branch model. Only a certain rectangular area R in the input image I contains the original image content, and the other areas are the zero-filled areas during image enhancement. online The 0-filled area in is set to 255 (uncertain label), Y online The label of the rectangular area R in M and B is obtained by score propagation using the valid area R' in M and B (R' corresponds to R). online The information of M and B is integrated, and the result is more accurate. On the other hand, the small amount of highlighted background area in M will expand rapidly after fractional propagation, resulting in the generated Y online In these areas, there are a lot of false positive labels predicted as foreground, so it is necessary to online Correct these wrong labels to background labels and get the corrected second label Y refine .

[0048] In this step, after obtaining the first training label Y init , the second training label Y online and the corrected second training label Y refine Then, use Y init and Y online The segmentation prediction result M of the dual-branch model is supervised by CrossEntropy loss, and Y init and Y refine The semantic correlation matrix between different pixels in B is obtained, and this matrix is used to supervise the semantic correlation matrix generated based on B, thereby indirectly supervising the object boundary prediction result B of the dual-branch model.

[0049] In this step, after the dual-branch model training is completed, the images in the training set are multi-scaled and horizontally flipped, and then input into the trained dual-branch model. The semantic segmentation prediction results are obtained by the segmentation sub-model, and the object boundary results are obtained by the object boundary sub-model. Then, based on the semantic segmentation prediction results and the object boundary results, training pseudo labels are generated through score propagation.

[0050] In this step, the generated pseudo labels are used to train a basic semantic segmentation model (such as deeplab). After the training is completed, the identified image is input into the semantic segmentation model to obtain the semantic segmentation result of the image.

[0051] As can be seen from the above technical solution, the embodiment of the present application inputs the image to be identified into the semantic segmentation model to obtain the semantic segmentation result of the image to be identified; wherein, the semantic segmentation model is obtained by training the basic semantic segmentation model based on the training pseudo-label; the training pseudo-label is obtained after the image is identified by the dual-branch model; the dual-branch model is obtained after iterative training based on the first training label and the second training label; wherein, the first training label includes the initial label Y generated by the classification network activation map CAM init ; said Y init Including the position and shape information of the foreground object in the picture; the second training label Y onlime is the online label output by the dual-branch model; online Based on the prediction results of the semantic segmentation branch and the object boundary detection branch, the dual-branch model is composed of a segmentation branch and an object boundary detection branch, and these two branches share a trunk branch for extracting features from the input image. online The two branch sub-models of the dual-branch model are iteratively optimized. During the forward propagation, the foreground category scores in the segmentation results are propagated to the surrounding area under the constraint of the object boundary. The generated Y online The foreground area is marked more completely and accurately, and the two sub-model branches of the dual-branch model can be optimized during back propagation. init Compared with the supervised object boundary branching scheme (first training label), it can effectively suppress false positives in object boundaries (internal boundaries of objects), and help propagate foreground category scores from salient areas to non-salient areas. The embodiment of the present application performs score propagation optimization based on the segmentation prediction results of the segmentation sub-model and generates training pseudo-labels. Compared with the traditional CAM-based method, the generated training pseudo-labels are more accurate, thereby training a higher-performance basic semantic segmentation model and improving the accuracy of the semantic segmentation results.

[0052] Based on the content of the above embodiment, in this embodiment, the CAM is obtained after the classification network model performs feature recognition on the image; the classification network model is obtained after training based on the image category label; the image category label is provided by the training data set.

[0053] Based on the content of the above embodiment, in this embodiment, the training pseudo labels are obtained by using a dual-branch model to recognize the image, including:

[0054] The training pseudo labels are obtained by using the semantic segmentation prediction results obtained after the semantic segmentation branch identifies the image, and the object boundary results obtained after the object boundary detection branch identifies the image.

[0055] In this embodiment, it should be noted that each image in the training set is multi-scaled and horizontally flipped to generate multiple images and input them into the trained dual-branch model to obtain multiple semantic segmentation prediction results and object boundary results. The average of the multiple semantic segmentation prediction results and object boundary results is taken, and a method similar to that of generating the second training label is used to generate training pseudo labels based on these average values.

[0056] Based on the content of the above embodiment, in this embodiment, the dual-branch model is obtained through iterative training based on the first training label and the second training label, including:

[0057] The CAM is processed offline to generate a first training label; under the constraint of the object boundary map generated by the object boundary detection branch, the foreground category score in the initial segmentation probability map generated by the semantic segmentation branch is propagated using a foreground category score propagation method to obtain a revised segmentation probability map, and a second training label is generated based on the revised segmentation probability map;

[0058] Supervise training of the object boundary detection branch in the dual-branch model based on the first training label and the second training label;

[0059] When using the second training label to supervise the object boundary detection branch, the second training label can be modified to a certain extent and used as the supervision signal. First, the initial segmentation probability map is processed based on the dense conditional random field (CRF) to obtain the background reference label. Then, the second training label is modified based on the background reference label to obtain the modified second training label.

[0060] Finally, the object boundary sub-model in the dual-branch model is supervised and trained based on the first training label and the corrected second training label.

[0061] In this embodiment, it should be noted that after using the first training label generated by the activation map CAM, the second training label is obtained by propagating the network foreground category score. According to the first training label and the second training label, the segmentation sub-model in the dual-branch model is supervised and trained. In addition, in order to better supervise the object boundary branch, the second training label can be corrected. This is because a small number of highlighted background areas in the segmentation probability map will expand rapidly after score propagation, resulting in the generated second training labels being predicted as foregrounds in these areas. There are a large number of false positive labels, so the second training labels need to be corrected and these erroneous labels are corrected to background labels. Therefore, the segmentation result is subjected to dense CR processing to obtain a reference label. The second training label is corrected according to the reference label to obtain a corrected second training label, and then the object boundary sub-model in the dual-branch model is supervised and trained according to the first training label and the corrected second training label. It can be seen that the embodiment of the present application uses the initial label (first training label) and the online label (second training label) to supervise the training of the semantic segmentation sub-model during the back propagation of the dual-branch model, and uses the initial label and the corrected second training label to supervise the object boundary sub-model. The initial labels serve to initialize and stabilize the training process, while the online labels and the revised second training labels integrate information from the two sub-model branches. During training, these two sub-models are iteratively optimized. The revised second training labels mitigate the negative impact of low-quality object segmentation results. After the dual-branch network is trained, the segmentation predictions of the sub-model are optimized using object boundary information to generate training pseudo-labels. Compared to traditional methods that optimize CAM, the generated training pseudo-labels are more accurate.

[0062] The present application will be described in detail below through specific embodiments.

[0063] First embodiment:

[0064] In this embodiment, a semantic segmentation database is taken as an example, including 21 semantic categories of background, with 10,582 training images and corresponding semantic segmentation labels. This embodiment only uses image category labels, which can be obtained through semantic segmentation label conversion.

[0065] Figure 2 As shown in the flowchart of the present invention, the weakly supervised semantic segmentation method provided by the embodiment of the present application specifically includes the following steps:

[0066] Step S0, use the image category labels to train a classification network, such as Figure 3Classic models such as ResNet-50 can be used. The network backbone weights are initialized using the backbone of an ImageNet pre-trained model. The fully connected layers used for classification have no bias and their weights are randomly initialized. During training, the input images are randomly scaled (with the long side in the range of 320-640), randomly flipped horizontally, and pixel values are normalized (first dividing by 255 to convert pixel values to [0, 1], then normalizing the RGB channels based on means of 0.485, 0.456, and 0.406, and variances of 0.229, 0.224, and 0.225). The images are then randomly cropped to 512×512, with undersized portions padded with zeros. The cropped images are fed into the network for training using SGD optimization with a backbone learning rate of 0.1 and a learning rate of 1.0 for the fully connected layers used for classification. The batch size is 16, and training is performed for 5 epochs.

[0067] Step S1: After the classification network training is completed, the training image is input into the network. The last convolutional layer outputs the feature map F. The fully connected layer weights are used as the weights of the 1×1 convolution kernel to convolve F and the result is input into the ReLU activation function. The activation map (CAM) with the same size as F and 20 channels (there are 20 types of foreground objects) is obtained, as shown in the following figure: Figure 3 .

[0068] In step S2, the input image size is 512×512, and the activation map (CAM) is upsampled to 512×512. The channels corresponding to the categories that do not appear in the input image are set to 0, and the remaining channels are normalized to [0,1] (each channel is divided by the maximum value of all its positions). A background channel with a value of 0.3 is added before the first channel to obtain a new activation map CAM1. CAM1 is input into the argmax function and the maximum value is taken in the channel dimension to obtain a segmentation label. The dense conditional random field processing is performed on it using the pydensecrf package to obtain the label Y. fg (addPairwiseGaussian parameters sxy = 3, compat = 3; addPairwiseBilateral parameters sxy = 50, srgb = 5, compat = 10, unary_from_labels parameters gt_prob = 0.7, zero_unsure = False, inference 10 times). Similarly, add a background channel with a value of 0.05 before the first channel and perform the same subsequent operations to obtain the label Y bg . fg The background tag in the Y bgIf the pixel is foreground, it is re-marked as uncertain pixel and Y init (512×512), as shown below (Y init [i] indicates Y init The i-th pixel in Y fg [i],Y bg [i] is similar, 0 represents the background category and 255 represents uncertain pixels).

[0069]

[0070] Step S3, build a dual-branch network backbone, such as Figure 4 The input image size of the dual-branch network is 512×512, and resnet50 or resnet101 is selected as the backbone. The stride of the resnet network stage4 and stage5 is changed from 2 to 1. At the same time, the dilation of the 3×3 convolution from the second to the last layer of stage4 is set to 2, and the dilation of the first 3×3 convolution is set to 1. The dilation of the 3×3 convolution from the second to the last layer of stage5 is set to 4, and the dilation of the first 3×3 convolution is set to 2. In this way, stage5 will finally output a feature map F with a stride of 8 s8 (size is 64×64), F s8 The receptive field of each position is as large as the receptive field of the corresponding position in the original ResNet network.

[0071] Step S4, construct a dual-branch network segmentation branch sub-model, add seg head after stage5, and use the Aspp model, such as Figure 5 , consists of 4 3×3 convolutions with bias, 21 output channels, and dilations of 6, 12, 18, and 24 respectively. s8 The results of the four convolutions are added together and then upsampled by a factor of 2. A softmax operation is performed on the channel dimension to obtain the semantic segmentation result M, which has 21 channels (foreground + background) and a size of 128×128.

[0072] Step S5: Construct a double-branch network object boundary branch sub-model. Figure 6The output features from stage 1 to stage 5 are passed through five edge layers, reducing the number of channels to 32. These five edge layers are called edge layer 1, edge layer 2, ..., edge layer 5. Each edge layer consists of a 1×1 convolution, a GroupNorm layer (with 4 groups), and a Relu layer. Edge layers 3, 4, and 5 are upsampled by a factor of 2 before the Relu layer. The resulting five feature maps are concatenated and fed into edge layer 6 (a 1×1 convolution). The output is a 1-channel object boundary map. The values in the object boundary map are mapped to the range [0, 1] using a sigmoid function. Note that B is 128×128 in size.

[0073] Step S6: Generate online label Y using segmentation result M and object boundary B online (Second training label, 512×512). The training images are randomly scaled, cropped, and enhanced before being input to the model. In the input image I, only a rectangular region R (h×w) contains the original image content, and the rest of the region is padded with zeros. Region R corresponds to the valid region R' (h / 4×w / 4) in M and B. The R' region in M and B is selected during the forward propagation of the two-branch model. and Generate Y by score propagation online Labels in the R region Y online The zero-filled area in is set to 255 (unsure label).

[0074] The following describes the score propagation process. In order to reduce the amount of calculation and facilitate batch processing, and Resized to 64×64 and First, based on Calculate the pixel correlation sparse matrix A of size 4096×4096. Consider The maximum value β of the boundary confidence of two pixels i and j whose distance from each other does not exceed 3 and the pixels near their connection (the pixels with the closest vertical distance to the connection line of i and j) is (1-β) 10 As the correlation between i and j, A i，j =A j，i =(1-β) 10 ,If the distance between pixels m,n exceeds 3, A m，n =A n，m = 0. The propagation of pixel correlation is realized by matrix multiplication, and the calculation right Normalize each column so that its sum is 1. It is dense and describes the semantic correlation between distant pixels. When two pixels are far apart, it is inaccurate to calculate the correlation based on the boundary confidence on the line connecting them. The present invention obtains the distant pixel correlation by continuous matrix multiplication.

[0075] get Afterwards, The channel corresponding to the category not included in the input image is set to 0, and the background channel is set to 0.25. For each category i included in the input image, adjust The i-th channel is a 1×4096 vector, and the value is normalized to [0,1]. Perform matrix multiplication to obtain a new vector V i , V i Resize V to 64×64 i 64×64 , which is the corrected i-th channel. Finally, the corrected segmentation result is obtained Input argmax function and calculate the maximum value in channel dimension to get Corresponding online label Adjust it to R size (h×w), and we get Fill the 0 area with 255 to get the complete Y online . V i and The calculation method is as follows: (Vec() means vectorization, express i-th channel, label I represents the category label of the input image):

[0076]

[0077] Using the above-obtained Y online Supervise the segmentation branch, in order to better supervise the object boundary branch, the above Y online Make some corrections. The input image does not contain the corresponding channel of the category, all are set to 0, the background channel is set to 0.05, and then the dense CRF is processed and adjusted to R size (h×w). (DenseCRF parameters are: iter_max=10,pos_xy_std=1,pos_w=3,bi_xy_std=67,bi_rgb_std=3,bi_w=4), generate The background threshold (0.05) is much smaller than the generated Background threshold (0.25), the background area of the former has a higher confidence, and the background area label of the former is used to correct the latter, and we get As shown below ( express The i-th pixel in ):

[0078]

[0079] Fill the 0 area with 255 to get the complete Y refine , use Y refine and Y init Supervised object boundary branch.

[0080] Step S7, training the dual-branch network, such as Figure 3 , using Y init and Y online Through CrossEntropy loss supervision M, while Y init and Y refine Obtain the semantic correlation matrix between different pixels in B, and use the semantic correlation matrix to indirectly supervise B.

[0081] Y init or Y refine Downsample to the size of B (128×128), only consider the semantic relevance between pixels with certain category labels and supervise the boundary confidence of related positions in B. For pixel p, consider the category labels of all other pixels with a distance of no more than 10 from it. If they are the same as p, they form a positive pair with p. If they are different, they form a negative pair with p. If they have uncertain labels, they are not considered. Supervise the maximum boundary confidence of two pixels on B and the pixels near their connection. For positive pairs, set the maximum value label to 0, and for negative pairs, set the maximum value label to 1. Supervise these maximum values through Binary CrossEntropy loss. The total loss function is (L CE is CrossEntropy loss, L A is the inter-pixel correlation loss):

[0082] L=L A (B,Y refine )+L A (B,Y init )+L CE (M,Y online )+L CE (M,Y init )

[0083] When training the two-branch network, the input image is randomly scaled to [0.5, 1.5], randomly flipped horizontally, pixel values normalized to [-1, 1], and randomly cropped to 512×512. Incomplete pixels are padded with zeros during cropping. The cropped images are fed into the network for training. The backbone learning rate is 0.0025, all edge_layers and seg head learning rates are 0.025, and the batch_size is 10. Training is performed for 19 epochs.

[0084] Step S8: After the model training is completed, the 10582 training images are horizontally flipped and enlarged by 1.5 and 2 times to obtain 6 images, which are input into the dual-branch network. The segmentation results and object boundaries are calculated using the average value M of the 6 results. ave 、B ave At this time M ave 、B ave All regions are valid, similar to the generation of Y in step S6 online The method based on B ave Generate a correlation matrix (When generating the sparse correlation matrix A, consider the pixel pairs whose distance does not exceed 5), ave Perform score propagation to obtain training pseudo labels.

[0085] Step S9: Use the generated pseudo labels to train a basic semantic segmentation model (such as deeplab). After the training is completed, the identified image is input into the semantic segmentation model to obtain the semantic segmentation result of the image.

[0086] It can be seen from the above technical solution that the embodiment of the present application uses image category labels to train a classification model, uses the activation map (CAM) to obtain the first training label (initial label) of the training picture, and uses it as a supervisory signal to train a two-branch model to predict object boundaries and semantic segmentation results. In the process of training the two-branch model, the object boundary and semantic segmentation prediction results are used to generate a second training label (online label), supervise the object boundary and semantic segmentation branches, and perform iterative optimization. After the model training is completed, the object boundary and semantic segmentation prediction results are used to generate high-quality training pseudo labels, train a standard semantic segmentation model, and use the model to perform semantic segmentation on the picture. On the one hand, the segmentation result predicted by the network is more accurate than the activation map (CAM). On the other hand, the false positive examples in the object boundary are reduced through iterative optimization, which helps to propagate the foreground category score. Therefore, the training pseudo label finally generated marks a more complete foreground area, making the segmentation result of the basic semantic segmentation model trained based on the training pseudo label more accurate.

[0087] Based on the same inventive concept, another embodiment of the present invention provides a weakly supervised semantic segmentation device, such as Figure 7As shown, the device includes:

[0088] Processing module 1 is used to obtain a picture to be identified and input the picture to be identified into a semantic segmentation model to obtain a semantic segmentation result of the picture to be identified;

[0089] In which, the semantic segmentation model is obtained by training the basic semantic segmentation model based on the training pseudo-label; the training pseudo-label is obtained after the dual-branch model recognizes the image; the dual-branch model is obtained after iterative training based on the first training label and the second training label; wherein, the first training label is the initial label generated by the classification network activation map CAM; the initial label includes the foreground object position and shape information of the image; the second training label is the online label output by the dual-branch model; the online label is generated based on the semantic segmentation branch prediction result and the object boundary detection branch prediction result; the dual-branch model is composed of the semantic segmentation branch and the object boundary detection branch, and the semantic segmentation branch and the object boundary detection branch share a trunk branch for extracting image features.

[0090] The weakly supervised semantic segmentation device described in this embodiment can be used to execute the above method embodiment. Its principles and technical effects are similar and will not be repeated here.

[0091] Based on the same inventive concept, another embodiment of the present invention provides an electronic device, see Figure 8 The structural diagram of the electronic device specifically includes the following contents: a processor 801, a memory 802, a communication interface 803 and a communication bus 804;

[0092] The processor 801, memory 802, and communication interface 803 communicate with each other via the communication bus 804; the communication interface 803 is used to implement information transmission between devices;

[0093] The processor 801 is used to call the computer program in the memory 802, and when the processor executes the computer program, it implements all steps of the above-mentioned weakly supervised semantic segmentation method, for example: obtaining a picture to be identified, and inputting the picture to be identified into a semantic segmentation model to obtain the semantic segmentation result of the picture to be identified; wherein, the semantic segmentation model is obtained by training the basic semantic segmentation model based on the training pseudo-label; the training pseudo-label is obtained after the dual-branch model recognizes the picture; the dual-branch model is obtained after iterative training based on the first training label and the second training label; wherein, the first training label is an initial label generated by the classification network activation map CAM; the initial label includes the foreground object position and shape information of the picture; the second training label is an online label output by the dual-branch model; the online label is generated based on the semantic segmentation branch prediction result and the object boundary detection branch prediction result; the dual-branch model is composed of the semantic segmentation branch and the object boundary detection branch, and the semantic segmentation branch and the object boundary detection branch share a trunk branch for extracting picture features.

[0094] Based on the same inventive concept, another embodiment of the present invention provides a non-transitory computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements all the steps of the above-mentioned weakly supervised semantic segmentation method, for example: obtaining a picture to be identified, and inputting the picture to be identified into a semantic segmentation model to obtain a semantic segmentation result of the picture to be identified; wherein, the semantic segmentation model is obtained by training a basic semantic segmentation model based on training pseudo labels; the training pseudo labels are obtained after a dual-branch model recognizes the picture; the dual-branch model is obtained after iterative training based on a first training label and a second training label; wherein, the first training label is an initial label generated by a classification network activation map CAM; the initial label includes the foreground object position and shape information of the picture; the second training label is an online label output by the dual-branch model; the online label is generated based on the semantic segmentation branch prediction result and the object boundary detection branch prediction result; the dual-branch model is composed of the semantic segmentation branch and the object boundary detection branch, and the semantic segmentation branch and the object boundary detection branch share a trunk branch for extracting picture features. In addition, the logical instructions in the above-mentioned memory can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0095] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the embodiments of the present invention. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0096] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus the necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a computer-readable storage medium such as ROM / RAM, a disk, an optical disk, etc., including several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the weakly supervised semantic segmentation method described in each embodiment or some part of the embodiment.

[0097] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A weakly supervised semantic segmentation method, characterized in that include: Obtaining a picture to be identified, and inputting the picture to be identified into a semantic segmentation model to obtain a semantic segmentation result of the picture to be identified; The semantic segmentation model is obtained by training the basic semantic segmentation model based on the training pseudo-labels; the training pseudo-labels are obtained by the dual-branch model after recognizing the image; the dual-branch model is obtained after iterative training based on the first training label and the second training label; the first training label is the initial label generated by the classification network activation map CAM; the initial label includes the position and shape information of the foreground object in the image; the second training label is the online label output by the dual-branch model; the online label is generated based on the prediction results of the semantic segmentation branch and the prediction results of the object boundary detection branch; the dual-branch model is composed of the semantic segmentation branch and the object boundary detection branch, and the semantic segmentation branch and the object boundary detection branch share a trunk branch for extracting image features; The dual-branch model is obtained through iterative training based on the first training label and the second training label, and includes: The CAM is processed offline to generate a first training label; under the constraint of the object boundary map generated by the object boundary detection branch, the foreground category score in the initial segmentation probability map generated by the semantic segmentation branch is propagated using a foreground category score propagation method to obtain a revised segmentation probability map, and a second training label is generated based on the revised segmentation probability map; Supervise the training of the object boundary detection branch and the semantic segmentation branch in the dual-branch model according to the first training label and the second training label; Based on a dense conditional random field (DCR), the initial segmentation probability map is processed to obtain a background reference label, and the second training label is corrected according to the background reference label to obtain a corrected second training label. Based on the first training label and the corrected second training label, the object boundary sub-model in the dual-branch model is supervised and trained.

2. The weakly supervised semantic segmentation method according to claim 1, characterized in that The CAM is obtained by performing feature recognition on an image using a classification network model; the classification network model is obtained by training based on image category labels.

3. The weakly supervised semantic segmentation method according to claim 1, characterized in that The training pseudo labels are obtained by using a dual-branch model to identify the image, including: The training pseudo labels are obtained based on the semantic segmentation prediction results obtained after the semantic segmentation branch identifies the image, and the object boundary results obtained after the object boundary detection branch identifies the image.

4. The weakly supervised semantic segmentation method according to claim 3, characterized in that The training pseudo labels are obtained based on the semantic segmentation prediction results obtained after the semantic segmentation branch recognizes the image, and the object boundary results obtained after the object boundary sub-model recognizes the image, including: After performing multi-scale scaling and horizontal flipping on the image, the image is input into the trained semantic segmentation branch to obtain a semantic segmentation prediction result, and the image is input into the trained object boundary detection branch to obtain an object boundary result; The training pseudo label is generated according to the semantic segmentation prediction result and the object boundary result.

5. A weakly supervised semantic segmentation device, characterized in that include: A processing module is used to obtain a picture to be identified and input the picture to be identified into a semantic segmentation model to obtain a semantic segmentation result of the picture to be identified; The semantic segmentation model is obtained by training the basic semantic segmentation model based on the training pseudo-labels; the training pseudo-labels are obtained by the dual-branch model after recognizing the image; the dual-branch model is obtained after iterative training based on the first training label and the second training label; the first training label is the initial label generated by the classification network activation map CAM; the initial label includes the position and shape information of the foreground object in the image; the second training label is the online label output by the dual-branch model; the online label is generated based on the prediction results of the semantic segmentation branch and the prediction results of the object boundary detection branch; the dual-branch model is composed of the semantic segmentation branch and the object boundary detection branch, and the semantic segmentation branch and the object boundary detection branch share a trunk branch for extracting image features; The dual-branch model is obtained through iterative training based on the first training label and the second training label, and includes: The CAM is processed offline to generate a first training label; under the constraint of the object boundary map generated by the object boundary detection branch, the foreground category score in the initial segmentation probability map generated by the semantic segmentation branch is propagated using a foreground category score propagation method to obtain a revised segmentation probability map, and a second training label is generated based on the revised segmentation probability map; Supervise the training of the object boundary detection branch and the semantic segmentation branch in the dual-branch model according to the first training label and the second training label; Based on a dense conditional random field (DCR), the initial segmentation probability map is processed to obtain a background reference label, and the second training label is corrected according to the background reference label to obtain a corrected second training label. Based on the first training label and the corrected second training label, the object boundary sub-model in the dual-branch model is supervised and trained.

6. The weakly supervised semantic segmentation device according to claim 5, characterized in that The CAM is obtained by performing feature recognition on an image using a classification network model; the classification network model is obtained by training based on image category labels.

7. The weakly supervised semantic segmentation device according to claim 5, characterized in that The processing module is specifically used to: The training pseudo labels are obtained based on the semantic segmentation prediction results obtained after the semantic segmentation branch identifies the image, and the object boundary results obtained after the object boundary detection branch identifies the image.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the weakly supervised semantic segmentation method according to any one of claims 1 to 4 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the weakly supervised semantic segmentation method according to any one of claims 1 to 4 are implemented.

Images