Method, device, equipment and storage medium for generating semantic segmentation labels

By generating the initial semantic feature map and semantic extension matrix, and expanding the class activation area, the problem of low semantic segmentation label generation efficiency in the prior art is solved, and the training efficiency and accuracy of the semantic segmentation model are improved.

CN114596435BActive Publication Date: 2024-11-26TENCENT TECHNOLOGY (SHENZHEN) CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202210010450.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-01-06
Publication Date
2024-11-26
Estimated Expiration
2042-01-06

AI Technical Summary

Technical Problem

The existing semantic segmentation model requires pixel-level semantic segmentation labels before training, resulting in high data preparation time and labor costs, which affects training efficiency.

Method used

By generating the initial semantic feature map and semantic extension matrix, the class activation area is expanded, and the higher quality semantic segmentation labels are generated, thereby improving the efficiency and quality of label generation.

Benefits of technology

The generation efficiency and quality of semantic segmentation labels are improved, and the training efficiency and accuracy of semantic segmentation models are improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114596435B_ABST
    Figure CN114596435B_ABST
Patent Text Reader

Abstract

The present application discloses a method, device, equipment and storage medium for generating semantic segmentation labels, and relates to the field of artificial intelligence. The method comprises: generating an initial semantic feature map based on a first type of activation map of a target image, the initial semantic feature map is used to characterize the semantic area where a target semantic segmentation object in the target image is located in the feature map; generating a semantic extension matrix based on the initial semantic feature map and the target image, the semantic extension matrix is ​​used to characterize the semantic relationship between different feature positions in the feature map; generating a second type of activation map of the target image based on the initial semantic feature map and the semantic extension matrix, the class activation area corresponding to the target semantic segmentation object in the second type of activation map is larger than the class activation area corresponding to the target semantic segmentation object in the first type of activation map; generating a semantic segmentation label corresponding to the target image based on the second type of activation map. The embodiment of the present application improves the generation efficiency of semantic segmentation labels while improving the generation quality of semantic segmentation labels.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence, and in particular to a method, apparatus, device and storage medium for generating semantic segmentation labels. Background Art

[0002] Semantic Segmentation, as a basic task in computer vision, is used to assign semantic categories to each pixel in an image and is widely used in scenes such as scene recognition and autonomous driving.

[0003] In the related art, the semantic segmentation model used for semantic segmentation is usually trained in a fully supervised manner. Since the semantic segmentation model is used to achieve pixel-level semantic recognition, pixel-level semantic segmentation labels are required as supervision during the model training process. The pixel-level semantic segmentation labels contain the semantic categories corresponding to each pixel point.

[0004] However, pixel-level semantic segmentation labels require manual annotation of pixels in the image one by one, which results in a lot of time and manpower cost for data preparation in the early stage of model training, affecting the training efficiency of the semantic segmentation model. Summary of the invention

[0005] The embodiments of the present application provide a method, apparatus, device and storage medium for generating semantic segmentation labels, which can improve the efficiency of generating semantic segmentation labels and thus improve the training efficiency of semantic segmentation models. The technical solution is as follows:

[0006] On the one hand, an embodiment of the present application provides a method for generating a semantic segmentation label, the method comprising:

[0007] Generate an initial semantic feature map based on the first type activation map of the target image, wherein the initial semantic feature map is used to represent the semantic area where the target semantic segmentation object in the target image is located in the feature map;

[0008] Generate a semantic extension matrix based on the initial semantic feature map and the target image, wherein the semantic extension matrix is ​​used to represent the semantic relationship between different feature positions in the feature map;

[0009] Generate a second class activation map of the target image based on the initial semantic feature map and the semantic extension matrix, wherein the class activation area corresponding to the target semantic segmentation object in the second class activation map is larger than the class activation area corresponding to the target semantic segmentation object in the first class activation map;

[0010] Generate a semantic segmentation label corresponding to the target image based on the second type of activation map.

[0011] On the other hand, an embodiment of the present application provides a device for generating a semantic segmentation label, the device comprising:

[0012] A feature map generation module, used to generate an initial semantic feature map based on the first type activation map of the target image, wherein the initial semantic feature map is used to characterize the semantic area where the target semantic segmentation object in the target image is located in the feature map;

[0013] A matrix generation module, used to generate a semantic extension matrix based on the initial semantic feature map and the target image, wherein the semantic extension matrix is ​​used to represent the semantic relationship between different feature positions in the feature map;

[0014] A semantic extension module, configured to generate a second class activation map of the target image based on the initial semantic feature map and the semantic extension matrix, wherein the class activation area corresponding to the target semantic segmentation object in the second class activation map is larger than the class activation area corresponding to the target semantic segmentation object in the first class activation map;

[0015] A label generation module is used to generate a semantic segmentation label corresponding to the target image based on the second type activation map.

[0016] On the other hand, an embodiment of the present application provides a computer device, which includes a processor and a memory, wherein the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the method for generating semantic segmentation labels as described in the above aspects.

[0017] On the other hand, an embodiment of the present application provides a computer-readable storage medium, in which at least one instruction is stored, and the at least one instruction is loaded and executed by a processor to implement the method for generating semantic segmentation labels as described in the above aspects.

[0018] On the other hand, an embodiment of the present application provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method for generating semantic segmentation labels described in the above aspects.

[0019] In the embodiment of the present application, the computer device generates an initial semantic feature map based on the first class activation map of the target image, and uses the semantic area of ​​the target semantic segmentation object represented by the initial semantic feature map in the feature map as a reference to generate a semantic extension matrix representing the semantic relationship between different feature positions in the feature map, thereby using the semantic extension matrix to perform semantic extension to obtain a second class activation map with a larger class activation area, and then using the second class activation map to generate a semantic segmentation label for the target image. Using the solution provided in the embodiment of the present application, based on the characteristic that the class activation map can accurately locate the target semantic segmentation object in the image within a small range, the class activation area of ​​the class activation map is expanded by semantic extension, so that the class activation map obtained after the semantic extension can cover more areas of the target semantic segmentation object, while improving the efficiency of semantic segmentation label generation, the quality of semantic segmentation label generation is improved, which helps to improve the training efficiency and training quality of subsequent semantic segmentation models. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 A schematic diagram showing an implementation environment provided by an exemplary embodiment of the present application is shown;

[0022] Figure 2 A flowchart of a method for generating a semantic segmentation label provided by an exemplary embodiment of the present application is shown;

[0023] Figure 3 is a schematic diagram of a semantic segmentation label generation process provided by an exemplary embodiment of the present application;

[0024] Figure 4 is a schematic diagram of an implementation of a semantic segmentation label generation process provided by an exemplary embodiment of the present application;

[0025] Figure 5 A flowchart of a method for generating a semantic segmentation label provided by another exemplary embodiment of the present application is shown;

[0026] Figure 6 This is a flow chart of using a semantic extension network to assist semantic segmentation process as shown in an exemplary embodiment of the present application;

[0027] Figure 7 is a schematic diagram of an exemplary embodiment of the present application showing an implementation of a process of assisting semantic segmentation using a semantic extension network;

[0028] Figure 8 It is a schematic diagram of an implementation of a process of generating a class activation graph using a semantic extension network, shown in an exemplary embodiment of the present application;

[0029] Fig. 9 is a comparison diagram of the semantic segmentation effect shown in an exemplary embodiment of the present application;

[0030] Fig.10 A structural block diagram of a device for generating semantic segmentation labels provided by an exemplary embodiment of the present application is shown;

[0031] Fig.11 A schematic diagram of the structure of a computer device provided by an exemplary embodiment of the present application is shown. DETAILED DESCRIPTION

[0032] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.

[0033] For ease of understanding, the nouns involved in the embodiments of the present application are explained below.

[0034] Semantic segmentation label: a label used to characterize the semantic category corresponding to each pixel in the image. The semantic category can be the object category of the semantic segmentation object in the image, such as people, dogs, vehicles, etc. When training a semantic segmentation model, the training image is input into the semantic segmentation model to obtain the predicted semantic segmentation result. The semantic segmentation label serves as supervision for the predicted semantic segmentation result and guides the training of the semantic segmentation model.

[0035] Class Activation Map (CAM): A feature map obtained by overlapping multiple feature maps after feature weighting. The process of generating a class activation map is called class activation mapping. The class activation map is generated based on the image-level label of the image. The image-level label is used to indicate the semantic segmentation object contained in the image, and does not indicate the pixel position of the semantic segmentation object in the image. Therefore, the time and labor cost of annotating an image with an image-level label is much lower than annotating an image with a pixel-level semantic segmentation label.

[0036] The area with larger activation value in the class activation map (i.e., class activation area) corresponds to the area where the semantic segmentation object is located in the image. The visualization effect of the class activation map is similar to that of a heat map, and the class activation area corresponds to the thermal area in the heat map.

[0037] In the embodiment of the present application, the first type of activation map and the second type of activation map correspond to the same image, and the range of the class activation area in the second type of activation map is larger than the range of the class activation area in the first type of activation map, that is, the second type of activation map can cover the semantic segmentation object more comprehensively. Therefore, the label quality of the second semantic segmentation label generated based on the second type of activation map is higher than the label quality of the first semantic segmentation label generated based on the first type of activation map, and accordingly, the semantic segmentation accuracy of the semantic segmentation model trained using the second semantic segmentation label is also higher.

[0038] Semantic Extension Matrix: A matrix used to characterize the semantic relationship between different feature positions in a feature map, wherein the higher the semantic relevance between feature positions, the higher the corresponding weight value in the semantic extension matrix. In an embodiment of the present application, the semantic extension matrix is ​​used to semantically extend the class activation map, thereby expanding the class activation area in the class activation map, and then generating higher quality semantic segmentation labels.

[0039] Self-attention mechanism: A technique that calculates the similarity scores between each feature position and all feature positions in a feature map, and then normalizes these scores into attention weights to effectively aggregate similar features from the entire input. It is widely used in various computer vision tasks, such as video classification, object detection, and instance segmentation. In the embodiment of the present application, the semantic extension matrix is ​​generated by the self-attention mechanism.

[0040] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0041] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0042] Computer vision technology (CV) is a science that studies how to make machines "see". To put it more specifically, it refers to machine vision such as using cameras and computers to replace human eyes to identify and measure objects, and further perform graphic processing to make computer processing into images that are more suitable for human eye observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multidimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and mapping, and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0043] Semantic segmentation is a basic task in computer vision, which is used to assign semantic categories to each pixel in an image, thereby segmenting the locations of different objects in the image. The embodiment of the present application provides a weakly supervised semantic segmentation solution, which can use image-level labels to generate pixel-level semantic segmentation labels, improve the efficiency of generating semantic segmentation labels, and further improve the training efficiency of semantic segmentation models.

[0044] The class activation area of ​​the class activation map usually corresponds to the most recognizable area of ​​the semantic segmentation object. For example, when the semantic segmentation object is a bird, the class activation area usually corresponds to the bird's head area; when the semantic segmentation object is a bicycle, the class activation area usually corresponds to the wheel area. Therefore, the semantic segmentation label generated by the class activation map can only cover a small part of the semantic segmentation object, resulting in poor training effect when the semantic segmentation label is directly applied to the semantic segmentation model training.

[0045] In order to improve the generation quality of semantic segmentation labels, a semantic extension network (SENet) is designed in the embodiment of the present application to generate a class activation map with a larger class activation area, so as to improve the coverage of semantic segmentation labels generated based on the class activation map. Specifically, a semantic extension module for generating a semantic extension matrix is ​​provided in the semantic extension network. After the semantic extension matrix is ​​generated by the semantic extension module, the class activation area in the original class activation map is further extended using the semantic extension matrix to obtain a class activation map with a larger class activation area.

[0046] Figure 1A schematic diagram of an implementation environment provided by an exemplary embodiment of the present application is shown. The implementation environment includes a labeling device 110 and a server 120. The labeling device 110 and the server 120 communicate data via a communication network. Optionally, the communication network can be a wired network or a wireless network, and the communication network can be at least one of a local area network, a metropolitan area network, and a wide area network.

[0047] The labeling device 110 is an electronic device used to set image-level labels for images, and the electronic device includes but is not limited to mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle-mounted terminals, etc., which are not limited in the embodiments of the present application. For example, the labeling device 110 is a computer used by a labeler, and the labeler sets image-level labels for each image according to the objects contained in the image (i.e., indicating which objects are contained in the image, but not marking the specific area of ​​the object in the image).

[0048] Optionally, the implementation environment may include multiple annotation devices 110, and different annotation personnel may use the annotation devices 110 to annotate images in different image sets, thereby improving the annotation efficiency; they may also annotate images in the same image set, thereby determining the final image-level label of the image based on the annotation results of the same image by different annotation personnel, thereby improving the annotation accuracy. This embodiment is not limited to this.

[0049] Server 120 is a device for training semantic segmentation models. It can be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), as well as big data and artificial intelligence platforms.

[0050] Exemplarily, the server 120 includes a processor 122 and a memory 124, and the memory 124 further includes a receiving module 1241, a semantic segmentation label generation module 1242, and a model training module 1243. The receiving module 1241 is used to receive an image containing an image-level label sent by the annotation device 110; the semantic segmentation label generation module 1242 is used to generate a pixel-level semantic segmentation label for the image based on the image-level label; and the model training module 1243 is used to train a semantic segmentation model based on the image containing the semantic segmentation label.

[0051] It should be noted that, in the above implementation environment, the generation of semantic segmentation labels and model training is performed by the server 120 as an example. In other possible implementations, the semantic segmentation labels can also be generated by the annotation device 110, and the server 120 performs model training based on the semantic segmentation labels, or the generation of semantic segmentation labels and model training are both performed by the annotation device 110, which is not limited in the present application embodiment. For the convenience of description, in the following embodiments, the generation method of semantic segmentation labels is performed by a computer device as an example.

[0052] Please refer to Figure 2 , which shows a flowchart of a method for generating semantic segmentation labels provided by an exemplary embodiment of the present application.

[0053] Step 201, generating an initial semantic feature map based on a first type activation map of a target image, wherein the initial semantic feature map is used to characterize a semantic region where a target semantic segmentation object in the target image is located in the feature map.

[0054] In a possible implementation, the computer device first generates a first type of activation map corresponding to the target image based on the image-level label of the target image, and the size of the first type of activation map is consistent with the size of the feature map corresponding to the target image.

[0055] In some embodiments, the first type of activation map is obtained by training a classification network, and thus the first type of activation map can be referred to as a Classification-Class Activation Map (Classification-CAMs).

[0056] After obtaining the first type of activation map of the target image, the computer device further generates an initial semantic feature map based on the first type of activation map as initial seeds in a subsequent semantic expansion process, wherein the initial semantic feature map has the same size as the first type of activation map and is used to indicate the semantic area where each target semantic segmentation object in the feature map is located, and the type of the target semantic segmentation object is at least one.

[0057] Regarding the method of generating an initial semantic feature map, in one possible implementation, a computer device determines the foreground area, background area, and uncertain area in the first type of activation map according to a set threshold to obtain an initial semantic feature map, wherein the foreground area is the semantic area where the target semantic segmentation object is located.

[0058] Since the first type of activation map is sparse and incomplete and cannot cover the complete area of ​​the target semantic segmentation object, the semantic area in the initial semantic feature map generated based on the first type of activation map only covers part of the semantic area of ​​the target semantic segmentation object, which usually corresponds to the most recognizable area of ​​the target semantic segmentation object.

[0059] Step 202: Generate a semantic extension matrix based on the initial semantic feature map and the target image. The semantic extension matrix is ​​used to represent the semantic relationship between different feature positions in the feature map.

[0060] Since the pixels corresponding to the same semantic segmentation object in the target image are semantically related, and the initial semantic feature map can accurately locate the small semantic area where the semantic segmentation object is located, the computer device can use the initial semantic feature map and the target image to train and generate a semantic extension matrix that represents the semantic relationship between different feature positions in the feature map. The weight value in the semantic extension matrix represents the semantic transmission weight from each feature position on the feature map to other feature positions. The greater the semantic correlation between two feature positions in the feature map, the greater the semantic transmission weight corresponding to the two feature positions in the semantic extension matrix.

[0061] In some embodiments, when the size of the initial semantic feature map is H×W, the semantic extension matrix is ​​an N×N matrix, where N=H×W, H is the feature map height, and W is the feature map width.

[0062] Step 203, generating a second type activation map of the target image based on the initial semantic feature map and the semantic extension matrix, wherein the class activation area corresponding to the target semantic segmentation object in the second type activation map is larger than the class activation area corresponding to the target semantic segmentation object in the first type activation map.

[0063] Furthermore, the computer device uses the semantic extension matrix to semantically extend the initial semantic feature map, expand the semantic area where the target semantic segmentation object is located in the feature map, and obtain the second type of activation map. Compared with the first type of activation map, the second type of activation map has a larger class activation area due to the semantic extension, and can more comprehensively cover the target semantic segmentation object.

[0064] In the embodiment of the present application, the second type of activation map is obtained by semantic extension, so the second type of activation map can be called a semantic extension-class activation map (Semantic Extension-Class Activation Map, SE-CAM).

[0065] Step 204: Generate a semantic segmentation label corresponding to the target image based on the second type of activation map.

[0066] Furthermore, the computer device generates a pixel-level semantic segmentation label of the target image based on the second type activation map obtained after semantic extension. The size of the pixel-level semantic segmentation label is consistent with the image size of the target image, that is, it can indicate the semantics of each pixel in the target image (that is, the target semantic segmentation object to which it belongs).

[0067] Optionally, the computer device may use the generated semantic segmentation labels as training supervision for a semantic segmentation model to obtain a semantic segmentation model for segmenting a target semantic segmentation object from an image.

[0068] Indicatively, Figure 3 As shown, during the training process of the semantic segmentation model, the computer device first obtains the first type of activation map 33 of the target image 31 based on the target image 31 and its image-level label 32, and then generates an initial semantic feature map 34 as an initial seed based on the first type of activation map 33. Furthermore, the computer device obtains the semantic extension matrix 35 of the target image 31 through training based on the target image 31 and the initial semantic feature map 34, thereby generating a semantically extended second type of activation map 36 using the semantic extension matrix 35 and the initial semantic feature map 34. Finally, the computer device generates a pixel-level semantic segmentation label 37 based on the second type of activation map 36, and then obtains a semantic segmentation model 38 through training using the target image 31 and the semantic segmentation label 37. It can be seen from the above process that the model training can be completed by only performing image-level annotation on the target image, which can improve the annotation efficiency of the training samples compared to pixel-level annotation, thereby improving the efficiency of model training.

[0069] In summary, in the embodiment of the present application, the computer device generates an initial semantic feature map based on the first class activation map of the target image, and uses the semantic area of ​​the target semantic segmentation object represented by the initial semantic feature map in the feature map as a reference to generate a semantic extension matrix representing the semantic relationship between different feature positions in the feature map, thereby using the semantic extension matrix to perform semantic extension to obtain a second class activation map with a larger class activation area, and then using the second class activation map to generate a semantic segmentation label for the target image. Using the solution provided in the embodiment of the present application, based on the characteristic that the class activation map can accurately locate the target semantic segmentation object in the image within a small range, the class activation area of ​​the class activation map is expanded by semantic extension, so that the class activation map obtained after the semantic extension can cover more areas of the target semantic segmentation object, while improving the efficiency of semantic segmentation label generation, the quality of semantic segmentation label generation is improved, which helps to improve the training efficiency and training quality of subsequent semantic segmentation models.

[0070] Regarding the method of generating the first type of activation map, the computer device can train a classification network based on the target image and the image-level label corresponding to the target image, where the image-level label is used to indicate the target semantic segmentation object contained in the target image.

[0071] Among them, a global average pooling (GAP) layer is set after the last convolution layer of the classification network. During the training process, the computer device uses the image-level label as supervision and trains the classification network through multi-label classification loss.

[0072] In an illustrative example, for a given target image, f k (x, y) is defined as the value at (x, y) in the k-th feature map in the last convolutional layer. The result of global average pooling on the k-th feature map is denoted as F k =∑ x,y f k (x,y). After the classification network is trained, the computer device removes the global average pooling layer and applies the final classification layer (such as softmax) to each column of the feature map. For a given category c, the input to the classification layer is the category score in, is the classification weight associated with category c of the kth feature map.

[0073] F k =∑ x,y f k Substitute (x,y) into the category score S c You can get:

[0074]

[0075] make is the activation of category c at position (x,y). Since S c =∑ x,y M c (x,y), so M c It can represent the importance of the spatial position (x, y) in classifying the target image into category c.

[0076] After the network training is completed, the computer device uses the original feature map (i.e., f k (x, y)) and the classification weight of the category to which the target semantic segmentation object belongs (i.e. ), generate the first type activation map of the target image.

[0077] In one possible implementation, a computer device trains a semantic extension network (SENet) using an initial semantic feature map as ground-truth. After network training is completed, the computer device semantically extends the initial semantic feature map through a semantic extension matrix in the semantic extension network to obtain a class activation map for generating a semantic segmentation label.

[0078] like Figure 4As shown, the computer device uses the semantic extension network 43 to perform semantic segmentation on the target image 41, and uses the initial semantic feature map 42 as the supervision of the semantic segmentation result to train the semantic extension network 43. After the training of the semantic extension network 43 is completed, the computer device performs semantic extension through the semantic extension matrix in the semantic extension network 43 to obtain a semantic extension-class activation map 44. Finally, the computer device generates a semantic segmentation label 45 using the semantic extension-class activation map 44.

[0079] Please refer to Figure 5 , which shows a flowchart of a method for generating semantic segmentation labels provided by another exemplary embodiment of the present application.

[0080] Step 501, generating an initial semantic feature map based on a first type activation map of a target image, wherein the initial semantic feature map is used to characterize a semantic region where a target semantic segmentation object in the target image is located in the feature map.

[0081] In a possible implementation, the computer device may generate an initial semantic feature map by including the following steps.

[0082] 1. Determine the foreground region in the first type of activation map based on the target threshold.

[0083] In a possible implementation, the computer device first normalizes the activation values ​​in the first type of activation map, and determines the foreground area in the first type of activation map based on the target threshold. When the normalized activation value is greater than or equal to the target threshold, the position corresponding to the activation value belongs to the foreground area. For example, the target threshold may be 0.1.

[0084] 2. Use the saliency model to estimate the background of the first type of activation map and determine the background area in the first type of activation map.

[0085] In a possible implementation, the computer device performs saliency detection on the first type of activation map through a saliency model, obtains a saliency score corresponding to each position in the first type of activation map, and performs background estimation on the first type of activation map based on the saliency score. The computer device determines the position whose saliency score is lower than a score threshold as a background area. For example, the score threshold may be 0.06.

[0086] 3. Generate an initial semantic feature map in the feature space based on the foreground area and the background area.

[0087] Furthermore, the computer device generates an initial semantic feature map in the feature space based on the determined foreground area and background area.

[0088] In some embodiments, when a certain position in the first type activation map belongs to both the foreground area and the background area, the computer device determines the position as an unknown area; when a certain position in the first type activation map belongs to neither the foreground area nor the background area, the computer device determines the position as an unknown area. The unknown area will be ignored in subsequent processes.

[0089] Step 502, semantic segmentation is performed on the target image through a semantic extension network to obtain a predicted semantic feature map, and a semantic extension matrix is ​​set in the semantic extension network.

[0090] In one possible design, the semantic extension network is provided with a feature extraction layer and a segmentation head for image feature extraction. The feature extraction layer and the segmentation head act as auxiliary semantic segmentation networks for generating better semantic segmentation labels to train the actual semantic segmentation model.

[0091] Optionally, the feature extraction layer adopts an expanded full convolutional network, and a matrix generation layer is arranged on the top of the expanded full convolutional network, which is used to generate a semantic extension matrix according to the feature map output by the last convolutional layer in the expanded full convolutional network.

[0092] The feature map output by the feature extraction layer is further input into the segmentation head after semantic extension by the semantic extension matrix. The segmentation head performs semantic segmentation to obtain a predicted semantic feature map, which is used to characterize the predicted semantic area of ​​the target semantic segmentation object in the target image in the feature map.

[0093] In a possible implementation, Figure 6 As shown, the process of using a semantic extension network by a computer device to perform semantic segmentation includes the following steps.

[0094] Step 502A, based on the original feature map of the target image, generate a semantic extension matrix through a matrix generation layer.

[0095] Optionally, the matrix generation layer adopts a self-attention mechanism, that is, the computer device performs a self-attention mechanism on the original feature map of the target image (obtained by feature extraction by the feature extraction layer in the semantic extension network) through the matrix generation layer to obtain a semantic extension matrix. The process of performing a self-attention mechanism on the original feature map may include the following steps:

[0096] 1. Reconstruct the original feature map into a query feature map and a key-value feature map.

[0097] In one possible implementation, the computer device uses a 1×1 convolution to reconstruct the original feature map into two different embeddings, namely a query feature map and a key feature map, wherein the query feature map and the key feature map have the same size and are both smaller than the size of the original feature map.

[0098] In an illustrative example, the original feature map F∈R H×W×K , the reconstructed query feature graph Key-value feature graph Where K is the number of channels, Furthermore, if N = H × W, the query feature graph and the key-value feature graph can be reconstructed as In this way, for each position in the spatial dimension of the query feature map and key-value feature map, a shape of Vector.

[0099] 2. Generate a feature similarity matrix based on the query feature graph and the key-value feature graph, where the feature similarity matrix is ​​used to characterize the similarity between feature positions in the query feature graph and the key-value feature graph.

[0100] Further, the computer device calculates the similarity between each feature position in the query feature map and the key-value feature map by matrix multiplication to obtain a feature similarity matrix, wherein the feature similarity matrix can be expressed as:

[0101] E = exp(QK T )∈(0,+∞) N×N

[0102] Where T represents the transpose of the matrix.

[0103] Among them, the similarity between the i-th feature position and the j-th feature position can be indexed as i,j∈{0,1,2,…,N-1}.

[0104] 3. Regularize the feature similarity matrix to obtain the semantic extension matrix.

[0105] The semantic extension matrix can be obtained by regularizing the feature similarity matrix. The obtained semantic extension matrix can be expressed as:

[0106]

[0107] Among them, A ij The larger the (i-th row and j-th column position in the semantic extension matrix) is, the more semantically related the two feature positions in the feature map are.

[0108] It should be noted that the computer device can also generate a semantic extension matrix in other ways. The embodiment of the present application only uses the self-attention mechanism as an example for schematic illustration, but does not constitute a limitation to this.

[0109] Step 502B, performing matrix multiplication on the semantic extension matrix and the original feature map to obtain a semantic aggregation feature map.

[0110] The computer device aggregates the features of each feature position in the original feature map according to the semantic relationship between the feature positions indicated by the semantic extension matrix to obtain semantic aggregated features, wherein the more semantically relevant the feature positions are, the greater the aggregation weight when aggregating the features between the feature positions.

[0111] In a possible implementation, the computer device obtains a semantically aggregated feature map by performing matrix multiplication on the semantic extension matrix and the original feature map. In an illustrative example, when the original feature map is F∈R N×K , the semantic extension matrix is ​​A∈R N×N When the aggregate feature can be expressed as Z = AF, Z ∈ R N×K .

[0112] Indicatively, Figure 7 As shown, the computer device performs matrix multiplication on the original feature map F and the semantic extension matrix A to obtain the semantic aggregation feature map Z.

[0113] Step 502C: Generate a predicted semantic feature map based on the semantic aggregation feature map and the category weight of the category to which the target semantic segmentation object belongs.

[0114] In some embodiments, a semantic segmentation layer is provided in the segmentation head of the semantic extension network, and the computer device performs semantic segmentation through the semantic segmentation layer based on the semantic aggregation feature map to obtain a predicted semantic feature map.

[0115] In one possible implementation, the computer device uses a 1×1 convolution to calculate the output of the segmentation head Where C is the number of categories. For a specific category c, the initial semantic feature map is y c ∈R N×1 , the output predicted semantic feature map is The corresponding category weight is w c ∈R K×1 .

[0116] Indicatively, Figure 7 As shown, the computer device calculates the category weight w corresponding to the category c in the semantic segmentation layer c Perform matrix multiplication with the semantic aggregation feature map Z and perform softmax processing on the result to obtain the predicted semantic feature map corresponding to category c:

[0117] Step 503: Use the initial semantic feature map as supervision for predicting the semantic feature map to train the semantic extension network.

[0118] Since the initial semantic feature map can accurately locate the semantic segmentation object in a small area, in an embodiment of the present application, the computer device uses the initial semantic feature map as supervision to train the semantic extension network, wherein the process of training the semantic extension network is the process of optimizing the semantic extension matrix and the category weights.

[0119] Although it seems very difficult to use rough labels (i.e., initial semantic feature maps) as supervision to generate a better class activation map, since the initial semantic feature maps can be accurately located in a small area, this high-precision but low-recall label is sufficient to provide clues for semantic extension.

[0120] In a possible implementation, the computer device determines the semantic segmentation loss between the initial semantic feature map and the predicted semantic feature map, thereby training the semantic extension network based on the semantic segmentation loss. The semantic segmentation loss may be a cross entropy loss, which is not limited in the present embodiment.

[0121] Step 504: semantically extend the initial semantic feature graph through the semantic extension matrix in the semantic extension network to obtain a semantic extension feature graph.

[0122] In a possible design, the semantic extension network is provided with a semantic extension head in addition to a segmentation head. After the segmentation head training is completed, the computer device further performs semantic extension on the initial semantic feature map through the semantic extension head to obtain a semantic extension feature map.

[0123] In a possible implementation, the computer device performs matrix multiplication on the semantic extension matrix and the initial semantic feature map to obtain the semantic extension feature map. When the semantic extension matrix obtained through training is A and the initial semantic feature map is y c When , the semantic extension feature map can be expressed as:

[0124] A c =A T y c ∈R N×1

[0125] Step 505: Generate a second type of activation map based on the semantic extension feature map, the original feature map of the target image, and the category weight of the category to which the target semantic segmentation object belongs.

[0126] For the i-th feature position in the k-th feature map, its feature value Z after semantic extension i,k It can be expressed as:

[0127] Z i,k =A i,. F .,k =∑ x,y a i (x,y)f k (x,y)(Formula 2)

[0128] Among them, i∈{0,1,2,…,N-1} and k∈{0,1,2,…,K-1}, f k (x,y) and a i (x,y) is F .,k ∈R N×1 and A i,. ∈R 1×N The value at any spatial position (x, y) of . For a given category c and the i-th position in the k-th feature map, the softmax input for semantic segmentation is Substitute formula (2) into the category score S c You can get:

[0129]

[0130] Add up the activation maps generated by all points belonging to category c, and get the activation value of category c at each position on the plane:

[0131]

[0132] Among them, Ω c Represents the set of pixels belonging to category c.

[0133] Based on the above principle, the computer device can quickly calculate the matrix based on the semantic extension feature map, the original feature map of the target image, and the category weight of the category to which the target semantic segmentation object belongs, and obtain the second type of activation map. Among them, the second type of activation map can be expressed as:

[0134] M c =Fw c ⊙A c ∈R N×1 (Formula 5)

[0135] Among them, F is the original feature map, w c is the category weight of category c, and ⊙ represents the Hadamard product of the matrix.

[0136] Indicatively, Figure 8 As shown, the computer device is based on the original feature map F, the category weight w of category c c And the semantic extension feature map A c , calculate the second type of activation map Mc .

[0137] Step 506: Generate a semantic segmentation label corresponding to the target image based on the second type of activation map.

[0138] In a possible implementation, the computer device may generate a semantic segmentation label based on the second type of activation map, including the following steps.

[0139] 1. Determine the foreground region in the second type of activation map based on the target threshold.

[0140] Similar to the process of generating the initial seed, in a possible implementation, the computer device first normalizes the activation value in the second type of activation map, and determines the foreground area in the second type of activation map based on the target threshold. When the normalized activation value is greater than or equal to the target threshold, the position corresponding to the activation value belongs to the foreground area. For example, the target threshold may be 0.1.

[0141] 2. Perform background estimation on the second type of activation map through a saliency model to determine the background area in the second type of activation map.

[0142] Similar to the process of generating the initial seed, in a possible implementation, the computer device performs a saliency detection on the second type of activation map through a saliency model, obtains a saliency score corresponding to each position in the second type of activation map, and performs background estimation on the second type of activation map based on the saliency score. The computer device determines the position whose saliency score is lower than a score threshold as a background area. For example, the score threshold may be 0.06.

[0143] 3. Map the foreground area and background area to the pixel position in the image space to obtain the semantic segmentation label corresponding to the target image.

[0144] Since the size of the class activation map is different from that of the target image (the class activation map is a feature space, while the target image is an image space), the computer device needs to map the determined foreground area and background area from the feature space to the image space, that is, determine whether each pixel position in the target image belongs to the background area or the foreground area, and finally generate a pixel-level semantic segmentation label.

[0145] In a possible implementation manner, the computer device performs pixel mapping on the foreground area and the background area by upsampling, which is not limited in this embodiment.

[0146] It should be noted that when a position in the second type of activation map belongs to both the foreground area and the background area, the computer device determines the position as an unknown area; when a position in the second type of activation map belongs to neither the foreground area nor the background area, the computer device determines the position as an unknown area. The unknown area is ignored in the mapping and subsequent semantic segmentation process.

[0147] To verify the effectiveness of SENet, we use the initial seeds generated by Classification-CAMs as supervision to train the semantic extension network, and use the semantic extension network to generate semantic extension-class activation maps (SE-CAMs). As shown in Table 1, the pseudo labels (i.e., semantic segmentation labels) generated based on SE-CAMs have higher quality and an accuracy of 9.8% higher than the pseudo labels generated by Classification-CAMs (the accuracy mIoU here is measured on the PASCAL VOC2012 training set).

[0148] In addition, we compared the segmentation results on the PASCAL VOC dataset. Directly using pseudo labels generated based on the classification-class activation map achieved a segmentation mIoU of 50.9%, while using SENet was able to improve the segmentation performance by 4.5%. It can be seen that using SENet can improve the segmentation performance of semantic segmentation.

[0149] Table 1

[0150]

[0151] To demonstrate the applicability of SENet, we also use three different initial seeds, generated by OAA, BES, and GWSM, respectively. Fig. 9 Qualitative comparisons between different segmentation labels generated by OAA, OAA+SENet, BES, BES+SENet, GESM, and GWSM+SENet are shown. It can be clearly seen that when SENet is trained with different initial seeds, the generated pseudo labels are always of higher quality than the initial seeds. Among them, GT represents the true value of semantic segmentation.

[0152] Please refer to Fig.10 , which shows a structural block diagram of a device for generating semantic segmentation labels provided by an exemplary embodiment of the present application, the device comprising:

[0153] A feature map generating module 1001 is used to generate an initial semantic feature map based on the first type activation map of the target image, wherein the initial semantic feature map is used to characterize the semantic region where the target semantic segmentation object in the target image is located in the feature map;

[0154] A matrix generation module 1002, configured to generate a semantic extension matrix based on the initial semantic feature map and the target image, wherein the semantic extension matrix is ​​used to represent the semantic relationship between different feature positions in the feature map;

[0155] A semantic extension module 1003 is used to generate a second class activation map of the target image based on the initial semantic feature map and the semantic extension matrix, wherein the class activation area corresponding to the target semantic segmentation object in the second class activation map is larger than the class activation area corresponding to the target semantic segmentation object in the first class activation map;

[0156] The label generation module 1004 is used to generate a semantic segmentation label corresponding to the target image based on the second type activation map.

[0157] Optionally, the matrix generation module 1002 includes:

[0158] A segmentation unit, configured to perform semantic segmentation on the target image through a semantic extension network to obtain a predicted semantic feature map, wherein the semantic extension network is provided with the semantic extension matrix;

[0159] A training unit is used to train the semantic extension network by using the initial semantic feature map as supervision of the predicted semantic feature map.

[0160] Optionally, the semantic extension network includes a matrix generation layer;

[0161] The segmentation unit is used for:

[0162] Based on the original feature map of the target image, generating the semantic extension matrix through the matrix generation layer;

[0163] Performing matrix multiplication on the semantic extension matrix and the original feature map to obtain a semantic aggregation feature map;

[0164] The predicted semantic feature map is generated based on the semantic aggregation feature map and the category weight of the category to which the target semantic segmentation object belongs.

[0165] Optionally, when the semantic extension matrix is ​​generated by the matrix generation layer based on the original feature map of the target image, the segmentation unit is used to:

[0166] The original feature map is processed by the self-attention mechanism through the matrix generation layer to obtain the semantic extension matrix.

[0167] Optionally, when the matrix generation layer performs self-attention mechanism processing on the original feature map to obtain the semantic extension matrix, the segmentation unit is used to:

[0168] Reconstructing the original feature map into a query feature map and a key-value feature map;

[0169] Generate a feature similarity matrix based on the query feature graph and the key-value feature graph, wherein the feature similarity matrix is ​​used to represent the similarity between feature positions in the query feature graph and the key-value feature graph;

[0170] Regularization is performed on the feature similarity matrix to obtain the semantic extension matrix.

[0171] Optionally, the training unit is used to:

[0172] Determining a semantic segmentation loss between the initial semantic feature map and the predicted semantic feature map;

[0173] The semantic extension network is trained based on the semantic segmentation loss.

[0174] Optionally, the semantic extension module 1003 includes:

[0175] A semantic extension unit, configured to perform semantic extension on the initial semantic feature graph through the semantic extension matrix in the semantic extension network to obtain a semantic extension feature graph;

[0176] An activation map generating unit is used to generate the second type activation map based on the semantic extension feature map, the original feature map of the target image, and the category weight of the category to which the target semantic segmentation object belongs.

[0177] Optionally, the semantic extension unit is used to:

[0178] The semantic extension matrix and the initial semantic feature map are matrix multiplied to obtain the semantic extension feature map.

[0179] Optionally, the label generating module 1004 includes:

[0180] A first determining unit, configured to determine a foreground area in the second type of activation map based on a target threshold;

[0181] A second determining unit is used to perform background estimation on the second type of activation map by using a saliency model to determine a background area in the second type of activation map;

[0182] A mapping unit is used to map the foreground area and the background area to pixel positions in an image space to obtain the semantic segmentation label corresponding to the target image.

[0183] Optionally, the feature map generation module 1001 includes:

[0184] A third determining unit, configured to determine a foreground area in the first type of activation map based on a target threshold;

[0185] a fourth determining unit, configured to perform background estimation on the first type of activation map by using a saliency model, and determine a background area in the first type of activation map;

[0186] A feature map generating unit is used to generate the initial semantic feature map in a feature space based on the foreground area and the background area.

[0187] Optionally, the device further comprises:

[0188] A classification network training module, used for training a classification network based on the target image and an image-level label corresponding to the target image, wherein the image-level label is used to indicate the target semantic segmentation object contained in the target image;

[0189] A class activation map generation module is used to generate the first class activation map of the target image based on the original feature map corresponding to the target image in the classification network and the classification weight of the category to which the target semantic segmentation object belongs.

[0190] In summary, in the embodiment of the present application, the computer device generates an initial semantic feature map based on the first class activation map of the target image, and uses the semantic area of ​​the target semantic segmentation object represented by the initial semantic feature map in the feature map as a reference to generate a semantic extension matrix representing the semantic relationship between different feature positions in the feature map, thereby using the semantic extension matrix to perform semantic extension to obtain a second class activation map with a larger class activation area, and then using the second class activation map to generate a semantic segmentation label for the target image. Using the solution provided in the embodiment of the present application, based on the characteristic that the class activation map can accurately locate the target semantic segmentation object in the image within a small range, the class activation area of ​​the class activation map is expanded by semantic extension, so that the class activation map obtained after the semantic extension can cover more areas of the target semantic segmentation object, while improving the efficiency of semantic segmentation label generation, the quality of semantic segmentation label generation is improved, which helps to improve the training efficiency and training quality of subsequent semantic segmentation models.

[0191] Please refer to Fig.11, which shows a schematic diagram of the structure of a computer device provided by an exemplary embodiment of the present application. Specifically, the computer device 1100 includes a central processing unit (CPU) 1101, a system memory 1104 including a random access memory 1102 and a read-only memory 1103, and a system bus 1105 connecting the system memory 1104 and the central processing unit 1101. The computer device 1100 may also include a basic input / output system (I / O system) 1106 that helps transmit information between various devices in the computer, and a large-capacity storage device 1107 for storing an operating system 1113, application programs 1114 and other program modules 1115.

[0192] In some embodiments, the basic input / output system 1106 may include a display 1108 for displaying information and an input device 1109 such as a mouse and a keyboard for user inputting information. The display 1108 and the input device 1109 are connected to the central processing unit 1101 through an input / output controller 1110 connected to the system bus 1105. The basic input / output system 1106 may also include an input / output controller 1110 for receiving and processing inputs from a plurality of other devices such as a keyboard, a mouse, or an electronic stylus. Similarly, the input / output controller 1110 also provides output to a display screen, a printer, or other types of output devices.

[0193] The mass storage device 1107 is connected to the central processing unit 1101 through a mass storage controller (not shown) connected to the system bus 1105. The mass storage device 1107 and its associated computer readable media provide non-volatile storage for the computer device 1100. That is, the mass storage device 1107 may include a computer readable medium (not shown) such as a hard disk or drive.

[0194] Without loss of generality, the computer-readable medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Computer storage media include random access memory (RAM), read-only memory (ROM), flash memory or other solid-state storage technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, cassette, magnetic tape, disk storage or other magnetic storage devices. Of course, those skilled in the art will know that the computer storage medium is not limited to the above. The above-mentioned system memory 1104 and mass storage device 1107 can be collectively referred to as memory.

[0195] The memory stores one or more programs, and the one or more programs are configured to be executed by one or more central processing units 1101. The one or more programs contain instructions for implementing the above-mentioned methods. The central processing unit 1101 executes the one or more programs to implement the methods provided by the above-mentioned various method embodiments.

[0196] According to various embodiments of the present application, the computer device 1100 can also be connected to a remote computer on the network through a network such as the Internet. That is, the computer device 1100 can be connected to the network 1112 through the network interface unit 1111 connected to the system bus 1105, or the network interface unit 1111 can be used to connect to other types of networks or remote computer systems (not shown).

[0197] The memory also includes one or more programs, which are stored in the memory and include steps executed by a computer device in the method provided in the embodiment of the present application.

[0198] An embodiment of the present application also provides a computer-readable storage medium, in which at least one instruction is stored. The at least one instruction is loaded and executed by a processor to implement the method for generating semantic segmentation labels described in any of the above embodiments.

[0199] The embodiment of the present application provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method for generating semantic segmentation labels described in the above embodiment.

[0200] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0201] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for generating semantic segmentation labels, characterized in that: The method comprises: Generate an initial semantic feature map based on the first type activation map of the target image, wherein the initial semantic feature map is used to represent the semantic area where the target semantic segmentation object in the target image is located in the feature map; The target image is semantically segmented by a semantic extension network to obtain a predicted semantic feature map, wherein a semantic extension matrix is ​​provided in the semantic extension network, and the semantic extension matrix is ​​used to characterize the semantic relationship between different feature positions in the feature map, and the predicted semantic feature map is used to characterize the predicted semantic region where the target semantic segmentation object in the target image is located in the feature map; Using the initial semantic feature map as supervision of the predicted semantic feature map, training the semantic extension network; Performing semantic extension on the initial semantic feature graph through the semantic extension matrix in the semantic extension network to obtain a semantic extension feature graph; Based on the semantic extension feature map, the original feature map of the target image, and the category weight of the category to which the target semantic segmentation object belongs, generating a second class activation map of the target image, wherein the class activation area corresponding to the target semantic segmentation object in the second class activation map is larger than the class activation area corresponding to the target semantic segmentation object in the first class activation map; Generate a semantic segmentation label corresponding to the target image based on the second type of activation map.

2. The method according to claim 1, characterized in that: The semantic extension network includes a matrix generation layer; The semantic segmentation of the target image is performed by using a semantic extension network to obtain a predicted semantic feature map, including: Based on the original feature map of the target image, generating the semantic extension matrix through the matrix generation layer; Performing matrix multiplication on the semantic extension matrix and the original feature map to obtain a semantic aggregation feature map; The predicted semantic feature map is generated based on the semantic aggregation feature map and the category weight of the category to which the target semantic segmentation object belongs.

3. The method according to claim 2, characterized in that The generating the semantic extension matrix through the matrix generating layer based on the original feature map of the target image includes: The original feature map is processed by the self-attention mechanism through the matrix generation layer to obtain the semantic extension matrix.

4. The method according to claim 3, characterized in that The self-attention mechanism processing is performed on the original feature map through the matrix generation layer to obtain the semantic extension matrix, including: Reconstructing the original feature map into a query feature map and a key-value feature map; Generate a feature similarity matrix based on the query feature graph and the key-value feature graph, wherein the feature similarity matrix is ​​used to represent the similarity between feature positions in the query feature graph and the key-value feature graph; Regularization is performed on the feature similarity matrix to obtain the semantic extension matrix.

5. The method according to claim 1, characterized in that The method of using the initial semantic feature map as supervision of the predicted semantic feature map and training the semantic extension network includes: Determining a semantic segmentation loss between the initial semantic feature map and the predicted semantic feature map; The semantic extension network is trained based on the semantic segmentation loss.

6. The method according to claim 1, characterized in that The semantic extension of the initial semantic feature graph by the semantic extension matrix in the semantic extension network to obtain a semantic extension feature graph includes: The semantic extension matrix and the initial semantic feature map are matrix multiplied to obtain the semantic extension feature map.

7. The method according to any one of claims 1 to 6, characterized in that: The generating a semantic segmentation label corresponding to the target image based on the second type activation map includes: Determine a foreground region in the second type of activation map based on a target threshold; Performing background estimation on the second type of activation map by using a saliency model to determine a background area in the second type of activation map; The foreground area and the background area are mapped to pixel positions in an image space to obtain the semantic segmentation label corresponding to the target image.

8. The method according to any one of claims 1 to 6, characterized in that: The generating an initial semantic feature map based on the first type activation map of the target image comprises: Determine a foreground region in the first type of activation map based on a target threshold; Performing background estimation on the first type of activation map by using a saliency model to determine a background area in the first type of activation map; The initial semantic feature map in a feature space is generated based on the foreground area and the background area.

9. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: Training a classification network based on the target image and an image-level label corresponding to the target image, wherein the image-level label is used to indicate the target semantic segmentation object contained in the target image; Based on the original feature map corresponding to the target image in the classification network and the classification weight of the category to which the target semantic segmentation object belongs, the first category activation map of the target image is generated.

10. A device for generating semantic segmentation labels, characterized in that: The device comprises: A feature map generation module, used to generate an initial semantic feature map based on the first type activation map of the target image, wherein the initial semantic feature map is used to characterize the semantic area where the target semantic segmentation object in the target image is located in the feature map; Matrix generation module, including segmentation unit and training unit; The segmentation unit is used to perform semantic segmentation on the target image through a semantic extension network to obtain a predicted semantic feature map, wherein a semantic extension matrix is ​​provided in the semantic extension network, and the semantic extension matrix is ​​used to characterize the semantic relationship between different feature positions in the feature map, and the predicted semantic feature map is used to characterize the predicted semantic region where the target semantic segmentation object in the target image is located in the feature map; The training unit is used to train the semantic extension network by taking the initial semantic feature map as the supervision of the predicted semantic feature map; A semantic extension module, including a semantic extension unit and an activation map generation unit; The semantic extension unit is used to semantically extend the initial semantic feature map through the semantic extension matrix in the semantic extension network to obtain a semantic extension feature map; The activation map generating unit is used to generate a second class activation map of the target image based on the semantic extension feature map, the original feature map of the target image, and the class weight of the class to which the target semantic segmentation object belongs, wherein the class activation area corresponding to the target semantic segmentation object in the second class activation map is larger than the class activation area corresponding to the target semantic segmentation object in the first class activation map; A label generation module is used to generate a semantic segmentation label corresponding to the target image based on the second type activation map.

11. The device according to claim 10, characterized in that The semantic extension network includes a matrix generation layer; The segmentation unit is used for: Based on the original feature map of the target image, generating the semantic extension matrix through the matrix generation layer; Performing matrix multiplication on the semantic extension matrix and the original feature map to obtain a semantic aggregation feature map; The predicted semantic feature map is generated based on the semantic aggregation feature map and the category weight of the category to which the target semantic segmentation object belongs.

12. The device according to claim 11, characterized in that When the semantic extension matrix is ​​generated by the matrix generation layer based on the original feature map of the target image, the segmentation unit is used to: The original feature map is processed by the self-attention mechanism through the matrix generation layer to obtain the semantic extension matrix.

13. The device according to claim 12, characterized in that When the original feature map is processed by the matrix generation layer through the self-attention mechanism to obtain the semantic extension matrix, the segmentation unit is used to: Reconstructing the original feature map into a query feature map and a key-value feature map; Generate a feature similarity matrix based on the query feature graph and the key-value feature graph, wherein the feature similarity matrix is ​​used to represent the similarity between feature positions in the query feature graph and the key-value feature graph; Regularization is performed on the feature similarity matrix to obtain the semantic extension matrix.

14. The device according to claim 10, characterized in that The training unit is used to: Determining a semantic segmentation loss between the initial semantic feature map and the predicted semantic feature map; The semantic extension network is trained based on the semantic segmentation loss.

15. The device according to claim 10, characterized in that The semantic extension unit is used to: The semantic extension matrix and the initial semantic feature map are matrix multiplied to obtain the semantic extension feature map.

16. The device according to any one of claims 10 to 15, characterized in that The label generation module includes: A first determining unit, configured to determine a foreground area in the second type of activation map based on a target threshold; A second determining unit is used to perform background estimation on the second type of activation map by using a saliency model to determine a background area in the second type of activation map; A mapping unit is used to map the foreground area and the background area to pixel positions in an image space to obtain the semantic segmentation label corresponding to the target image.

17. The device according to any one of claims 10 to 15, characterized in that The feature map generation module comprises: A third determining unit, configured to determine a foreground area in the first type of activation map based on a target threshold; a fourth determining unit, configured to perform background estimation on the first type of activation map by using a saliency model, and determine a background area in the first type of activation map; A feature map generating unit is used to generate the initial semantic feature map in a feature space based on the foreground area and the background area.

18. The device according to any one of claims 10 to 15, characterized in that The device also includes: A classification network training module, used for training a classification network based on the target image and an image-level label corresponding to the target image, wherein the image-level label is used to indicate the target semantic segmentation object contained in the target image; A class activation map generation module is used to generate the first class activation map of the target image based on the original feature map corresponding to the target image in the classification network and the classification weight of the category to which the target semantic segmentation object belongs.

19. A computer device, characterized in that: The computer device includes a processor and a memory, wherein the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the method for generating a semantic segmentation label as described in any one of claims 1 to 9.

20. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the method for generating semantic segmentation labels as described in any one of claims 1 to 9.

21. A computer program product, characterized in that The computer program product includes computer instructions, which are stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method for generating semantic segmentation labels as described in any one of claims 1 to 9.

Citation Information

Patent Citations

  • Weak supervision semantic segmentation method based on vehicle image, and related equipment thereof

    CN112329659A

  • Weak supervision image semantic segmentation method and system based on node classification

    CN113657560A

  • Dense reconstruction method, system and device based on semantic segmentation and medium

    CN113808251A

  • Image semantic segmentation method and device, computer equipment and storage medium

    CN113822282A

  • Training method for image semantic segmentation model and server

    WO2019196633A1