Weak supervision semantic segmentation method and device, electronic equipment and medium

Abstract

The invention discloses a weak supervision semantic segmentation method and device, electronic equipment and a medium, and the method comprises the steps: carrying out the salient target detection of an input image through a salient target detection module, so as to capture the complete contour and edge details of a salient target in the input image; generating a class activation graph by using the classification network to estimate the position of the target object in the image; a salient target selector module is designed, and a salient target detection result and the class activation graph are fused to generate a pseudo tag; carrying out weak supervision training on the semantic segmentation model by utilizing the generated pseudo tag so as to improve the overall performance of the semantic segmentation model; and obtaining a to-be-processed image, inputting the image into the trained semantic segmentation model, and outputting the segmented image. According to the method, the high-quality pseudo tag is generated through complementary fusion of the salient target detection and the class activation graph, the semantic segmentation precision is remarkably improved, the segmentation precision of the model in a complex scene is improved, and the method can be widely applied to the field of image processing.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a weakly supervised semantic segmentation method, device, electronic equipment and medium. Background Art

[0002] In computer vision, the application of weakly supervised semantic segmentation is mainly reflected in the following fields: 1) In the field of autonomous driving, weakly supervised semantic segmentation technology can effectively identify key elements such as roads, vehicles, pedestrians, etc., and improve the vehicle's perception of the road environment. By using a small amount of annotated data, this technology can train a semantic segmentation model with high accuracy, thereby assisting the autonomous driving system to make more accurate decisions and enhance the safety and reliability of autonomous driving. 2) In the field of remote sensing image analysis, weakly supervised semantic segmentation technology also shows great potential. Remote sensing images usually have the characteristics of high resolution and large amount of information, but annotated data is often difficult to obtain. Weakly supervised semantic segmentation technology can use a small amount of annotated information to perform efficient semantic segmentation on remote sensing images and extract key ground object information, such as urban buildings, farmland, waters, etc., providing strong support for urban planning, environmental monitoring and other fields. 3) In the field of medical image analysis, weakly supervised semantic segmentation technology is also of great significance. Medical image data usually contains rich pathological information, but annotated data is also scarce and the annotation process is complicated. Weakly supervised semantic segmentation technology can classify medical images at the pixel level without relying on a large amount of finely labeled data, helping doctors to more accurately identify lesions and diagnose diseases. In addition, this technology can also assist doctors in surgical planning, efficacy evaluation, and other tasks, improving the efficiency and quality of medical services.

[0003] Currently, the two most common weakly supervised semantic segmentation methods include: 1) Image-level label-based methods: Use image-level labels to train classification models, and generate pseudo labels through techniques such as class activation mapping (CAM) to achieve pixel-level segmentation. This method reduces the reliance on pixel-level annotations, but may face the problem of insufficient pseudo-label accuracy. Pseudo-labels with errors may further damage the accuracy of the model during the training process. 2) Introducing unsupervised learning methods: such as autoencoders or generative adversarial networks, train models through unlabeled data to increase the generalization ability of segmentation models. Since unsupervised learning does not rely on explicit supervisory signals (such as classification labels or regression values), it requires a large amount of data to mine the intrinsic structure and characteristics of the data itself. Summary of the invention

[0004] In order to solve at least one of the technical problems existing in the prior art to a certain extent, the purpose of the present invention is to provide a weakly supervised semantic segmentation method, device, electronic device and medium based on deep learning.

[0005] The first technical solution adopted by the present invention is:

[0006] A weakly supervised semantic segmentation method comprises the following steps:

[0007] Get the input image;

[0008] The salient object detection module performs salient object detection on the input image to capture the complete outline and edge details of the salient object in the input image.

[0009] For the input image, a classification network is used to generate a class activation map to estimate the location of the target object in the image;

[0010] Design a salient object selector module to fuse salient object detection results and class activation maps to generate high-quality pseudo labels;

[0011] Use the generated pseudo labels to perform weakly supervised training on the semantic segmentation model to improve the overall performance of the semantic segmentation model;

[0012] Get the image to be processed, input the image into the trained semantic segmentation model, and output the segmented image.

[0013] Furthermore, the salient object detection module is implemented using an RGB-SOD network;

[0014] The RGB-SOD network includes an encoder and a decoder, wherein the encoder is used to extract features of an input RGB image, and the decoder is used to generate a saliency map based on the extracted features to capture the edges and contours of salient objects in the image;

[0015] Among them, when training the RGB-SOD network, the NAMLAB superpixel segmentation algorithm is used to segment the input RGB image to generate a superpixel segmentation map; the boundaries between the superpixel segmentation maps are extracted to generate a binary map as a boundary guide to optimize the network's boundary perception ability.

[0016] Furthermore, the encoder is implemented using Swin-Transformer.

[0017] Furthermore, the classification network is used to generate a class activation map using image-level labels, and the class activation map is used to mark significant areas related to the classification results.

[0018] Furthermore, the salient object selector module calculates the consistency of the saliency map and the class activation map in object positioning through intersection over union (IoU), and determines the pseudo label according to the calculated consistency.

[0019] Further, determining the pseudo label according to a preset threshold comprises:

[0020] If the intersection-over-union ratio is greater than the preset threshold, that is, the overlap area between the saliency map and the class activation map is high, indicating that the two have good consistency in the target position. At this time, the saliency map is used as a pseudo-label to supplement the deficiency of the class activation map.

[0021] If the intersection-over-union ratio is less than the preset threshold, that is, the overlapping area between the saliency map and the class activation map is low, and the edge information provided by the saliency map is more refined, the weight of the saliency map is increased in the pseudo-label to improve the pseudo-label's ability to describe the target boundary.

[0022] Furthermore, the weakly supervised training of the semantic segmentation model includes:

[0023] Input the generated pseudo labels into the semantic segmentation model for training;

[0024] The model is optimized using the regional loss function and the boundary loss function to ensure that the model can maintain the fineness of the boundary when learning the target area.

[0025] The second technical solution adopted by the present invention is:

[0026] A weakly supervised semantic segmentation device, comprising:

[0027] An image input module, used to obtain an input image;

[0028] A boundary detection module, used to perform salient object detection on the input image through the salient object detection module, so as to capture the complete outline and edge details of the salient object in the input image;

[0029] The target detection module is used to generate a class activation map using a classification network for an input image to estimate the location of the target object in the image;

[0030] The label generation module is used to design a salient object selector module, fuse the salient object detection results and the class activation map to generate pseudo labels;

[0031] The model training module is used to use the generated pseudo labels to perform weakly supervised training on the semantic segmentation model to improve the overall performance of the semantic segmentation model;

[0032] The image segmentation module is used to obtain the image to be processed, input the image into the trained semantic segmentation model, and output the segmented image.

[0033] The third technical solution adopted by the present invention is:

[0034] An electronic device comprises a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement a weakly supervised semantic segmentation method as described above.

[0035] The fourth technical solution adopted by the present invention is:

[0036] A computer-readable storage medium stores at least one instruction, at least one program, a code set or an instruction set, wherein the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement a weakly supervised semantic segmentation method as described above.

[0037] The fifth technical solution adopted by the present invention is:

[0038] A computer program product or a computer program, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the above-mentioned weakly supervised semantic segmentation method.

[0039] The beneficial effects of the present invention include:

[0040] (1) Improve the quality of pseudo labels.

[0041] The present invention significantly improves the quality of pseudo-labels by combining salient object detection (SOD) with class activation maps (CAM), overcoming multiple limitations in traditional CAM methods. In traditional weakly supervised semantic segmentation methods, CAM can only capture partial areas of the target object due to its limited activation range, resulting in low accuracy of pseudo-labels, especially when the target boundary is not obvious or the contrast between the target and the background is not high, it is difficult for the pseudo-label to accurately reflect the complete form of the target. The present invention effectively integrates the fine boundary advantage of salient object detection (SOD) with the target positioning ability of CAM by designing a salient object selector module, and dynamically adjusts the weights of the two in the pseudo-label generation process, thereby significantly improving the overall quality of pseudo-labels.

[0042] Salient object detection can capture the complete outline and detailed information of the target, making up for the shortcomings of CAM in boundary recognition. By fusing the complementary information of SOD and CAM, the pseudo-labels generated by the present invention are not only significantly improved in the coverage of the target area, but also significantly improve the applicability of pseudo-labels in complex scenes. In addition, by measuring the degree of match between the salient object and CAM by the intersection over union (IoU), the salient object selector can dynamically adjust the generation of pseudo-labels according to the threshold, thereby further ensuring the high accuracy of the pseudo-labels. This combined method enables the generated pseudo-labels to have higher positioning accuracy and edge quality, providing more reliable supervision signals for subsequent semantic segmentation training.

[0043] (2) Enhance target edge perception.

[0044] In the semantic segmentation task, the capture of edge information is crucial to generate high-quality segmentation results, especially when the boundary between the target and the background is complex. The present invention effectively enhances the perception of target edge information by introducing a salient target selector module, so that the generated pseudo-label is significantly better than the traditional method in terms of boundary refinement. The salient target selector module can not only capture the overall outline of the salient target, but also enhance the detailed description of the target boundary in the pseudo-label, which plays an important role in improving the performance of the segmentation model in different scenarios.

[0045] Pseudo-labels generated by traditional CAM usually have blurred edges or inaccurate positioning problems, which is mainly because it is difficult to accurately delineate the boundary between the target and the background. The present invention uses salient target detection to provide clearer target edges, and through the dynamic fusion of the salient target selector module, the information of the boundary area in the pseudo-label is significantly enhanced. Specifically, the salient target selector module calculates the intersection and union ratio of the saliency map and the class activation map, evaluates the consistency between them, and gives a higher weight to the edge information when generating pseudo-labels. This method effectively solves the problem of blurred boundaries, so that the accuracy of the final segmentation result in the edge area is significantly improved, especially in scenes with complex backgrounds or relatively blurred boundaries between the target and the background. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the embodiments of the present invention or the drawings of related technical solutions in the prior art are introduced below. It should be understood that the drawings introduced below are only for the convenience of clearly describing some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0047] Figure 1is a schematic diagram of a weakly supervised semantic segmentation framework based on a salient target fusion method in an embodiment of the present invention;

[0048] Figure 2 is an intermediate result diagram of hierarchical image segmentation based on the NAMLab algorithm in an embodiment of the present invention;

[0049] Figure 3 is a schematic diagram of a network structure of a salient object detection (NAMLAB-SOD) based on NAMLAB-Edge edge information in an embodiment of the present invention;

[0050] Figure 4 is a flow chart of a salient object selection algorithm based on weakly supervised semantic segmentation in an embodiment of the present invention;

[0051] Figure 5 3 is a comparison effect diagram of the pseudo label generation method in the embodiment of the present invention. DETAILED DESCRIPTION

[0052] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and are not to be construed as limitations of the present invention. For the step numbers in the following embodiments, they are only provided for the convenience of explanation, and the order between the steps is not limited in any way, and the execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

[0053] In the description of the present invention, it should be understood that descriptions involving orientations, such as up, down, front, back, left, right, etc., and orientations or positional relationships indicated are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present invention.

[0054] In the description of the present invention, "several" means one or more, "more" means more than two, "greater than", "less than", "exceed" etc. are understood as not including the number itself, and "above", "below", "within" etc. are understood as including the number itself. If there is a description of "first" or "second", it is only used for the purpose of distinguishing the technical features, and cannot be understood as indicating or implying the relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0055] In the description of the present invention, unless otherwise clearly defined, terms such as setting, installing, connecting, etc. should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meanings of the above terms in the present invention based on the specific content of the technical solution.

[0056] Terminology explanation:

[0057] CAM: abbreviation of Class Activation Map, class activation map; it is a key means to locate the target object.

[0058] RGB-SOD network: A salient object detection network structure that generates a saliency map by encoding and decoding the input RGB image. The network uses a boundary-aware module to capture the complete outline and edge details of the salient object.

[0059] NAMLAB-Edge: An edge detection method based on the NAMLAB algorithm, which is used to enhance the boundary perception ability of the salient object detection network. It generates boundary guidance information by segmenting the superpixel area in the image and improves the perception of the target edge.

[0060] NAMLAB-SOD: A salient object detection method based on the NAMLAB algorithm. A saliency map is generated by performing superpixel segmentation and boundary detection on RGB images to locate and capture edge details of the target area, and a weak supervision method is combined to improve the boundary accuracy of the segmentation model.

[0061] Example 1

[0062] like Figure 1 As shown, this embodiment provides a weakly supervised semantic segmentation method, which generates high-quality pseudo labels through the complementarity of salient object detection and CAM, thereby effectively improving the segmentation accuracy. The method includes the following steps:

[0063] S1. Get input image.

[0064] S2. Perform salient object detection on the input image through the salient object detection module to capture the complete outline and edge details of the salient objects in the input image.

[0065] The salient object detection module is implemented using an RGB-SOD network, which includes an encoder and a decoder. The encoder is used to extract features of the input RGB image, and the decoder is used to generate a saliency map based on the extracted features to capture the edges and contours of salient objects in the image. During training, the RGB-SOD network uses the NAMLAB superpixel segmentation algorithm to segment the input RGB image and generate a superpixel segmentation map; the boundaries between the superpixel segmentation maps are extracted to generate a binary map as a boundary guide to optimize the network's boundary perception ability.

[0066] Specifically, in salient object detection, the goal is to find the most noticeable salient area in the image through image processing technology. In this embodiment, the input image is firstly subjected to salient object detection through the RGB-SOD network based on NAMLAB-Edge edge information. The network structure consists of an encoder and a decoder, which captures the complete outline and edge details of the salient object through feature extraction and generation of saliency maps.

[0067] Edge information enhancement: The NAMLAB-based edge detection module provides the network with additional boundary perception capabilities. Compared with traditional saliency detection methods, this module can effectively improve the ability to capture the edge area of ​​the target, thereby improving the quality of salient targets.

[0068] Encoding and decoding: First, the image is subjected to multi-scale feature extraction through an RGB encoder (e.g., a Swin-Transformer block). The extracted features contain semantic information from low to high levels. The decoder generates a saliency map based on these features. The saliency map obtained through salient object detection clearly marks the location and edges of salient objects in the image.

[0069] The advantage of this salient object detection method is that it can capture the complete outline and details of the target while enhancing the network's sensitivity to edge features, which is crucial for the subsequent generation of pseudo labels.

[0070] The following combination Figure 2 Introduce the NAMLAB superpixel segmentation algorithm.

[0071] A1. Perform asymmetric inverse matrix pattern matching on the input image to obtain segmentation rectangles, that is, obtain rectangles that meet the matching pattern by grid scanning, so as to obtain preliminary superpixels of the image. Figure 2 (b) is the obtained superpixel, and each rectangle represents a superpixel.

[0072] A2. Calculate the color mean and color variance of each rectangle as features, and merge similar rectangles by comparing the features of the four neighborhoods of the superpixel.

[0073] A3. Difference merging: First, set a threshold and merge each superpixel with a pixel count less than the threshold with the most similar superpixel in its four neighborhoods. For region merging, the union-find algorithm can be used to reduce complexity. Figure 2 (c) is the region obtained after differential merging. Compared with the original rectangular superpixel, the number of regions is greatly reduced. By setting the threshold of the number of regions, different output results can be obtained.

[0074] The use of deep learning-based algorithms avoids the need for a large amount of hand-crafted feature extraction, while the use of NAMLAB superpixel segmentation results for supervision can greatly improve the model's ability to perceive edges.

[0075] To understand the algorithm intuitively, Figure 2 The original image on the BSDS500 dataset and the intermediate result of hierarchical image segmentation based on the NAMLab algorithm are shown, and the edges of the segmented area are marked with red lines on the blank image.

[0076] Figure 2 (a) shows the original test image of size 321×481×3 on the BSDS500 dataset. Figure 2 (b) shows the output of the NAMLab-based color image representation algorithm. It is also the input of the NAMLab-based homogeneous block merging algorithm, and the number of NAMLab blocks is 27195. Figure 2 (c) shows the output result of the NAMLab-based homogeneous block merging algorithm; similarly, this is also the input result of the NAMLab-based residual region removal algorithm. After the NAMLab blocks are merged, the number of original NAMLab regions is 9483. These original NAMLab regions usually contain a lot of redundant information that needs to be removed. Figure 2 (d) shows the output of the residual region removal algorithm based on NAMLab. It is also the input of the adjacent region merging algorithm based on NAMLab; the number of intermediate NAMLab regions after removing the redundant information of the original NAMLab region is 829. Figure 2 (e) to (n) show the output results of the NAMLab-based neighboring region merging algorithm, where the number of NAMLab regions is 800, 700, 600, 500, 400, 300, 200, 100, 50 and 30, respectively.

[0077] S3. For the input image, use the classification network to generate a class activation map to estimate the location of the target object in the image.

[0078] The classification network is used to generate a class activation map using image-level labels, and the class activation map is used to mark salient areas related to classification results.

[0079] In the weakly supervised semantic segmentation task, the Class Activation Map (CAM) is a key means to locate the target object. By using a classification network, we can use image-level label information to generate a CAM and preliminarily estimate the location of the target object in the image.

[0080] CAM generation process: Given an input image, first use the pre-trained classification network to forward propagate the image. The output features of the last convolutional layer pass through the global average pooling (GAP) layer to generate a vector, which is used for image classification. By back-propagating the features output by the convolutional layer and applying the ReLU activation function, the activation map of each category in the image can be obtained.

[0081] Locating the target area: The role of CAM is to mark the salient areas related to the classification results, which usually indicate the location of the target object. However, the main disadvantage of CAM is that it can only activate part of the target area, and the activated area is often too coarse to accurately describe the boundary of the target.

[0082] S4. Design a salient object selector module to fuse the salient object detection results and class activation maps to generate high-quality pseudo labels.

[0083] See also Figure 3 and Figure 4 In order to generate high-quality pseudo-labels, this method designs a salient object selector module, which effectively fuses SOD (i.e., saliency map) and CAM and dynamically adjusts their weights in the pseudo-label generation process.

[0084] Complementary Fusion: The salient object selector module is designed to take advantage of the boundary detection advantage of SOD and the object localization ability of CAM. The consistency of SOD and CAM in object localization is measured by the intersection over union (IoU), and the salient object selector determines whether to select the saliency map as a pseudo label based on the set threshold.

[0085] a) If the overlapping area (i.e., intersection-over-union ratio) of SOD and CAM is high, it means that the two have good consistency in the target position. In this case, SOD is used as a pseudo-label to supplement the deficiency of CAM.

[0086] b) If the consistency between the two is low and the edge information provided by SOD is more refined, the weight of SOD is increased in the pseudo-label to improve the pseudo-label's ability to describe the target boundary.

[0087] Dynamic adjustment: The salient object selector generates high-quality pseudo labels by dynamically adjusting the threshold \(\alpha\) and combining the information of SOD and CAM. In this way, the salient pseudo labels can more accurately capture the complete outline of the target object while dealing with the separation problem of co-occurring objects, thereby generating more reliable pseudo labels.

[0088] S5. Use the generated pseudo labels to perform weakly supervised training on the semantic segmentation model to improve the overall performance of the semantic segmentation model.

[0089] The generated high-quality pseudo labels are used to train the weakly supervised semantic segmentation model. The goal of weakly supervised training is to improve the segmentation network's ability to capture targets and enhance the model's generalization performance in complex scenarios through these high-quality pseudo labels.

[0090] Training process: First, the generated pseudo ground truth is input into the segmentation network for training. During the training process, the pseudo ground truth provides the model with supervision information of the target location and boundaries, so that the segmentation network can learn the semantic regions in the image more accurately.

[0091] Loss function design: Multiple loss functions are used to optimize the network, including region loss and boundary loss, to ensure that the network can maintain the fineness of the boundary when learning the target area. Region loss is used to supervise the model to accurately classify the target area, while boundary loss ensures that the model captures the edge of the target more accurately.

[0092] Segmentation model: In this invention, a segmentation model based on a deep convolutional neural network (such as the DeepLab series) is used. This type of model performs well in handling complex semantic segmentation tasks. By using high-quality pseudo-labels, the segmentation model can learn accurate semantic features under weak supervision, thereby achieving higher segmentation accuracy on the validation and test sets.

[0093] like Figure 5 As shown, Figure 5 The figure shows the comparison of various pseudo-label generation methods, showing the differences between different methods in target boundaries and positioning accuracy.

[0094] S6. Obtain an image to be processed, input the image into a trained semantic segmentation model, and output a segmented image.

[0095] In summary, the entire weakly supervised segmentation process effectively improves the quality of pseudo labels and significantly enhances the performance of the segmentation model in complex scenarios through salient object detection, class activation map generation, dynamic fusion of salient object selector modules, and final weakly supervised training.

[0096] Example 2

[0097] This embodiment provides a weakly supervised semantic segmentation device, including:

[0098] An image input module, used to obtain an input image;

[0099] A boundary detection module, used to perform salient object detection on the input image through the salient object detection module, so as to capture the complete outline and edge details of the salient object in the input image;

[0100] The target detection module is used to generate a class activation map using a classification network for an input image to estimate the location of the target object in the image;

[0101] The label generation module is used to design a salient object selector module, fuse the salient object detection results and the class activation map to generate pseudo labels;

[0102] The model training module is used to use the generated pseudo labels to perform weakly supervised training on the semantic segmentation model to improve the overall performance of the semantic segmentation model;

[0103] The image segmentation module is used to obtain the image to be processed, input the image into the trained semantic segmentation model, and output the segmented image.

[0104] Since the device is a weakly supervised semantic segmentation device of an embodiment of the present invention, and the principle of solving the problem by the device is similar to that of the method, the implementation of the device can refer to the implementation process of the above method embodiment, and the repeated parts will not be repeated.

[0105] Example 3

[0106] An embodiment of the present invention further provides an electronic device, the electronic device comprising a processor and a memory, the memory storing at least one instruction, at least one program, a code set or an instruction set, the at least one instruction, the at least one program, the code set or the instruction set being loaded and executed by the processor to implement the following Figure 1 A weakly supervised semantic segmentation method shown in Figure 2.

[0107] It is understood that the memory may include a random access memory (RAM) or a read-only memory (ROM). Optionally, the memory includes a non-transitory computer-readable storage medium. The memory may be used to store instructions, programs, codes, code sets, or instruction sets. The memory may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function, instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data created according to the use of the server, etc.

[0108] The processor may include one or more processing cores. The processor uses various interfaces and lines to connect the various parts of the entire server, and executes various functions of the server and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory, and calling data stored in the memory. Optionally, the processor can be implemented in at least one hardware form of digital signal processing (DSP), field programmable gate array (FPGA), and programmable logic array (PLA). The processor can integrate one or a combination of a central processing unit (CPU) and a modem. Among them, the CPU mainly processes the operating system and application programs; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor, but implemented separately through a chip.

[0109] Since the electronic device is an electronic device corresponding to a weakly supervised semantic segmentation method of an embodiment of the present invention, and the principle of solving the problem by the electronic device is similar to that of the method, the implementation of the electronic device can refer to the implementation process of the above-mentioned method embodiment, and the repeated parts will not be repeated.

[0110] Example 4

[0111] The embodiment of the present invention further provides a computer-readable storage medium, wherein the storage medium stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, the at least one program, the code set or instruction set is loaded and executed by a processor to implement the following Figure 1 A weakly supervised semantic segmentation method shown in Figure 2.

[0112] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable rewritable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0113] Since the storage medium is a storage medium corresponding to a weakly supervised semantic segmentation method of an embodiment of the present invention, and the principle of solving the problem by the storage medium is similar to that of the method, the implementation of the storage medium can refer to the implementation process of the above-mentioned method embodiment, and the repeated parts will not be repeated.

[0114] Example 5

[0115] In some possible implementations, various aspects of the method of the embodiment of the present invention may also be implemented in the form of a program product, which includes a program code, and when the program product is run on a computer device, the program code is used to cause the computer device to perform the steps of a weakly supervised semantic segmentation method according to various exemplary embodiments of the present application described above in this specification. Among them, the executable computer program code or "code" for executing each embodiment can be written in a high-level programming language such as C, C++, C#, Smalltalk, Java, JavaScript, Visual Basic, structured query language (e.g., Transact-SQL), Perl, or in various other programming languages.

[0116] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0117] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0118] The above embodiments are only for illustrating the technical concept and features of the present invention, and their purpose is to enable ordinary technicians in the field to understand the content of the present invention and implement it accordingly, and they cannot be used to limit the protection scope of the present invention. Any equivalent changes or modifications made based on the essence of the content of the present invention should be included in the protection scope of the present invention.

Claims

1. A weakly supervised semantic segmentation method, characterized in that: The following steps are involved: Get the input image; The salient object detection module performs salient object detection on the input image to capture the complete outline and edge details of the salient object in the input image; For the input image, a classification network is used to generate a class activation map to estimate the location of the target object in the image; Design a salient object selector module to fuse salient object detection results and class activation maps to generate pseudo labels; Use the generated pseudo labels to perform weakly supervised training on the semantic segmentation model to improve the overall performance of the semantic segmentation model; Get the image to be processed, input the image into the trained semantic segmentation model, and output the segmented image.

2. A weakly supervised semantic segmentation method according to claim 1, characterized in that: The salient object detection module is implemented using an RGB-SOD network; The RGB-SOD network includes an encoder and a decoder, wherein the encoder is used to extract features of an input RGB image, and the decoder is used to generate a saliency map based on the extracted features to capture the edges and contours of salient objects in the image; wherein, during training, the RGB-SOD network uses the NAMLAB superpixel segmentation algorithm to segment the input RGB image to generate a superpixel segmentation map; and extracts the boundaries between the superpixel segmentation maps to generate a binary map as a boundary guide to optimize the boundary perception ability of the network.

3. A weakly supervised semantic segmentation method according to claim 2, characterized in that: The encoder is implemented using Swin-Transformer.

4. A weakly supervised semantic segmentation method according to claim 1, characterized in that: The classification network is used to generate a class activation map using image-level labels, and the class activation map is used to mark salient areas related to classification results.

5. A weakly supervised semantic segmentation method according to claim 1, characterized in that: The salient object selector module calculates the consistency of the saliency map and the class activation map in object positioning through intersection-over-union ratio, and determines the pseudo label according to the calculated consistency.

6. A weakly supervised semantic segmentation method according to claim 5, characterized in that: The step of determining a pseudo label according to a preset threshold value includes: If the intersection-over-union ratio is greater than the preset threshold, that is, the overlap area between the saliency map and the class activation map is high, indicating that the two have good consistency in the target position. At this time, the saliency map is used as a pseudo-label to supplement the deficiency of the class activation map. If the intersection-over-union ratio is less than the preset threshold, that is, the overlapping area between the saliency map and the class activation map is low, and the edge information provided by the saliency map is more refined, the weight of the saliency map is increased in the pseudo-label to improve the pseudo-label's ability to describe the target boundary.

7. A weakly supervised semantic segmentation method according to claim 1, characterized in that: The weakly supervised training of the semantic segmentation model includes: Input the generated pseudo labels into the semantic segmentation model for training; The model is optimized using the regional loss function and the boundary loss function to ensure that the model can maintain the fineness of the boundary when learning the target area.

8. A weakly supervised semantic segmentation device, characterized in that: include: An image input module, used to obtain an input image; A boundary detection module, used to perform salient object detection on the input image through the salient object detection module, so as to capture the complete outline and edge details of the salient object in the input image; The target detection module is used to generate a class activation map using a classification network for an input image to estimate the location of the target object in the image; The label generation module is used to design a salient object selector module, fuse the salient object detection results and the class activation map to generate pseudo labels; The model training module is used to use the generated pseudo labels to perform weakly supervised training on the semantic segmentation model to improve the overall performance of the semantic segmentation model; The image segmentation module is used to obtain the image to be processed, input the image into the trained semantic segmentation model, and output the segmented image.

9. An electronic device, characterized in that: The electronic device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the method described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the method according to any one of claims 1 to 7.

Images