Semantic segmentation model training method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a semantic segmentation model training method and device, electronic equipment and a storage medium, and can relate to the fields of artificial intelligence, semantic segmentation and the like, the method comprises the steps that a plurality of first sample images are acquired, labels of the first sample images comprise text labels of first object categories in a foreground, and for each first sample image, the first object categories of the first object categories in the foreground are classified; obtaining a category activation graph of each first object category in the first sample image through a first neural network model, obtaining a category representation graph of each first object category according to each category activation graph of each first object category, and obtaining a category representation graph of each first object category according to a text feature of a text tag of each first object category; and determining a first training loss according to the first similarity between the image features of the category representation graph and the image features of the category representation graph, so as to perform model training based on the first training loss. Based on the method, the semantic segmentation accuracy is improved.

Description

Technical Field

[0001] This application belongs to the field of computer technology and may relate to fields such as artificial intelligence and natural language processing. Specifically, this application relates to training methods, devices, electronic devices, and storage media for semantic segmentation models. Background Technology

[0002] Semantic segmentation, a popular task in computer vision, is widely used in various fields such as autonomous driving and intelligent transportation. However, training semantic segmentation models requires a large number of pixel-level segmentation labels, i.e., labeling each pixel in the image, which consumes a lot of manpower and time.

[0003] To reduce annotation costs, related technologies use training samples with image-level labels to train an image classification network. The trained image classification network is used to obtain the category activation maps of the sample images, which are then repaired into pseudo-segmentation labels. Finally, the pseudo-segmentation labels are used to train a semantic segmentation model.

[0004] However, category activation maps typically only activate the most salient regions of the target object, and some background is falsely activated. As a result, there are also a large number of incorrect annotations in the generated pseudo-segmentation labels, which affects the performance of the semantic segmentation model. Summary of the Invention

[0005] The purpose of this application is to provide a training method, apparatus, electronic device, and storage medium for a semantic segmentation model that can effectively improve the accuracy of semantic segmentation. To achieve this purpose, the technical solutions provided by this application are as follows: On the one hand, embodiments of this application provide a method for training a semantic segmentation model, the method comprising: Acquire multiple first sample images with labels, wherein the label of each first sample image includes text labels of each first object category included in the foreground of the first sample image; For each of the first sample images, a category activation map of each first object category in the first sample image is obtained through a first neural network model; For each of the first sample images, a category representation map of each first object category in the first sample image is obtained based on the category activation map of each first object category in the first sample image; The text features of the text labels of each first object category and the image features of the category representation graph are extracted respectively, and the first similarity between the text features of the text labels of each first object category and the image features of the category representation graph is determined. Based on the first similarity corresponding to each first object category in each of the first sample images, a first training loss is determined, and based on the total training loss, the first neural network model is continuously trained to obtain a first semantic segmentation model; wherein, the total training loss includes the first training loss.

[0006] On the other hand, embodiments of this application also provide a training apparatus for a semantic segmentation model, the apparatus comprising: The first sample acquisition module is used to acquire multiple first sample images with labels, wherein the label of each first sample image includes the text label of each first object category included in the foreground of the first sample image; The category activation map generation module is used to obtain the category activation map of each first object category in each first sample image through a first neural network model for each first sample image; The category representation map generation module is used to obtain a category representation map of each first object category in the first sample image for each first sample image based on the category activation map of each first object category in the first sample image; The first similarity calculation module is used to extract the text features of the text labels of each first object category and the image features of the category representation graph, and determine the first similarity between the text features of the text labels of each first object category and the image features of the category representation graph. The training module is used to determine a first training loss based on the first similarity corresponding to each first object category in each of the first sample images, and to continuously train the first neural network model based on the total training loss to obtain a first semantic segmentation model; wherein the total training loss includes the first training loss.

[0007] Optionally, the training device further includes a second similarity calculation module, which can be used for: For each first object category in each first sample image, a background activation map of the first object category is determined based on the category activation map of the first object category; and a background representation map of the first object category is obtained based on the background activation map of the first object category. Extract image features from the background representation map of each first object category, and determine the second similarity between the text features of the text label of each first object category and the image features of the background representation map; The training module can also be used for: The second training loss is determined based on the second similarity corresponding to each first object category in each of the first sample images; The total training loss also includes the second training loss.

[0008] Optionally, the label for each first sample image further includes: a text label for at least one second object category corresponding to each first object category, wherein the second object category is an object category that co-occurs with the first object category in the background of the image; The training device further includes a third similarity calculation module, which can be used for: Extract the text features of the text labels for each second object category; For each second object category corresponding to each first object category, a third similarity is determined between the image features of the category representation map of the first object category and the text features of the text label of the second object category; The training module can also be used for: The third training loss is determined based on the third similarity corresponding to each first object category in each of the first sample images; The total training loss also includes the third training loss.

[0009] Optionally, at least one second object category corresponding to a first object category is determined in the following way: For each of the first sample images, an image description text is generated based on the first sample image using a trained image description model; Identify each object appearing in the image description text of each of the first sample images; From each of the objects in the image description text, determine each third object that does not belong to each of the first candidate object categories; Determine the category of each of the aforementioned third objects; From each of the third object categories, determine the third object category that co-occurs with the first object category in each first sample image, and use it as at least one second object category corresponding to the first object category.

[0010] Optionally, the third similarity calculation module can also be used for: The number of times each of the first object category and each of the third object categories co-occurred in each of the first sample images was determined. Based on the order of co-occurrence frequency of each of the aforementioned third object categories from highest to lowest, at least one third object category with the highest ranking is selected as at least one second object category corresponding to the first object category.

[0011] Optionally, the training module can also be used for: Determine the third similarity corresponding to each first object category in each of the first sample images; The third training loss is determined based on each third similarity score that is greater than a preset similarity threshold.

[0012] Optionally, the training device further includes a regularization loss calculation module, which can be used for: Based on the category activation map of each first object category in each first sample image, determine the average probability that the sample pixel corresponding to each pixel belongs to the first object category; The regularization loss is determined based on the average probability that each pixel belongs to the first object category; the regularization loss is used to constrain the area size of the first object category in the foreground of the first sample image. The total training loss also includes the regularization loss.

[0013] On the other hand, embodiments of this application also provide a method for training a semantic segmentation model, the method comprising: Acquire multiple labeled second sample images; the label of each second sample image is used to indicate the object category to which each pixel in the second sample image belongs; Based on multiple second sample images, the second neural network model is continuously trained to obtain a well-trained second semantic segmentation model. Specifically, the label for each of the second sample images is obtained in the following way: Based on the second sample image, the category activation map of the second sample image corresponding to each first candidate object category is obtained by using the trained first semantic segmentation model; wherein, the first semantic segmentation model is trained using the method described in any one of claims 1 to 7; the first candidate object category is the object category that can be identified in the foreground; The label of the second sample image is obtained based on the category activation map corresponding to each first candidate object category of the second sample image.

[0014] On the other hand, embodiments of this application also provide a training apparatus for a semantic segmentation model, the apparatus comprising: The second sample acquisition module is used to acquire multiple labeled second sample images; the label of each second sample image is used to indicate the object category to which each pixel in the second sample image belongs; The model training module is used to continuously perform training operations on the second neural network model based on multiple second sample images to obtain a trained second semantic segmentation model. Specifically, the label for each of the second sample images is obtained in the following way: Based on the second sample image, the category activation map of the second sample image corresponding to each first candidate object category is obtained by using the trained first semantic segmentation model; wherein, the first semantic segmentation model is trained using the method described in any one of claims 1 to 7; the first candidate object category is the object category that can be identified in the foreground; The label of the second sample image is obtained based on the category activation map corresponding to each first candidate object category of the second sample image.

[0015] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method provided in any optional embodiment of this application.

[0016] On the other hand, embodiments of this application also provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method provided in any optional embodiment of this application.

[0017] On the other hand, embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the methods provided in any optional embodiment of this application.

[0018] The beneficial effects of the technical solution provided in this application are as follows: The semantic segmentation model training method provided in this application, during the model training process, determines the training loss based on the first similarity between the text features of the text labels of each first object category in the foreground of each first sample image and the image features of the category representation map (the activated image region in the category activation map), and trains the model so that the regions in the category activation map extracted by the model that are related to each first object category are activated, suppressing the erroneous activation of irrelevant regions, improving the accuracy of pixel classification in the category activation map, and at the same time providing a large number of automatically generated pixel-level semantic segmentation labels for supervised semantic segmentation tasks at a low cost. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below.

[0020] Figure 1 A schematic diagram of the implementation environment for a training method of a semantic segmentation model provided in this application embodiment; Figure 2 A flowchart illustrating a training method for a semantic segmentation model provided in an embodiment of this application; Figure 3A schematic diagram illustrating the training process of the semantic segmentation model provided in this application embodiment; Figure 4 This application provides a schematic diagram of a process for automatically generating obfuscated backgrounds. Figure 5 A flowchart illustrating another training method for a semantic segmentation model provided in this application embodiment; Figure 6 This is a schematic diagram illustrating the semantic segmentation result obtained by segmenting an input image according to an embodiment of this application; Figure 7 A schematic diagram comparing the category activation map generated for the embodiments of this application with category activation maps generated by other schemes; Figure 8 A schematic diagram illustrating the effects of ablation experiments on various loss functions provided in this application embodiment; Figure 9 A schematic diagram of the structure of a training device for a semantic segmentation model provided in an embodiment of this application; Figure 10 A schematic diagram of the structure of a training device for another semantic segmentation model provided in an embodiment of this application; Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0021] The embodiments of this application are described below with reference to the accompanying drawings. It should be understood that the embodiments described below with reference to the accompanying drawings are exemplary descriptions for explaining the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions of the embodiments of this application.

[0022] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the terms “comprising” and “including” as used in embodiments of this application mean that the corresponding feature can be implemented as the presented feature, information, data, step, operation, element, and / or component, but do not exclude implementation as other features, information, data, step, operation, element, component, and / or combinations thereof supported by the art. It should be understood that when we say that an element is “connected” or “coupled” to another element, the one element can be directly connected or coupled to the other element, or it can mean that the one element and the other element establish a connection relationship through an intermediate element. Furthermore, “connected” or “coupled” as used herein can include wireless connection or wireless coupling. The term “and / or” as used herein indicates at least one of the items defined by the term; for example, “A and / or B” can be implemented as “A,” or as “B,” or as “A and B.” When describing multiple (two or more) items, if the relationship between the multiple items is not explicitly defined, the multiple items can refer to one, several or all of the multiple items. For example, the description of "parameter A includes A1, A2, A3" can be implemented as parameter A includes A1 or A2 or A3, or it can be implemented as parameter A includes at least two of the three items A1, A2 and A3.

[0023] To better understand and explain the methods provided in the embodiments of this application, some technical terms involved in the embodiments of this application will be explained and described below.

[0024] Image foreground and background: The area of ​​interest to the user in the image is the foreground, and the area unrelated to the foreground is the background; for example, for an image of a person, the person is the foreground, and the surrounding environment is the background.

[0025] Class Activation Map (CAM): A spatial attention map extracted from a classification network that indicates the location of a target.

[0026] Weakly supervised semantic segmentation (WSSS): uses a large amount of unlabeled data, along with some data with pixel-level annotations, to obtain a semantic segmentation network.

[0027] Pseudo-segmentation labels: Segmentation labels extracted from classification networks that contain many errors compared to true semantic segmentation labels.

[0028] Cross-Language Image Pretraining (CLIP): A multimodal model used for jointly processing image and text data. Its design aims to enable computers to understand the semantic relationships between images and text.

[0029] Intersection over Union (IoU): A metric used to evaluate the performance of semantic segmentation models.

[0030] Mean Intersection over Union (mIoU): The average IoU calculated for all categories.

[0031] Training a semantic segmentation model requires a large number of pixel-level segmentation labels. To reduce the labeling cost, related technologies use image-level labeled training samples to first train an image classification network. The trained image classification network model extracts the class activation maps (CAMs) of the sample images. Based on the class activation maps (CAMs) of the sample images, pseudo-segmentation labels are generated for the sample images. The pseudo-segmentation labels are then used to train the semantic segmentation model.

[0032] However, the highlighted parts (activation parts) in the category activation map (CAM) of the sample images are often only the most distinguishable parts of the object. For example, for the category "vehicle", the most distinguishable parts are the wheels, and for the category "person", the most distinguishable parts are the head and limbs. In addition, some background is also falsely activated. Therefore, there are a lot of mislabeled pseudo-segmentation labels generated based on the category activation map (CAM) of the sample images, resulting in low accuracy of the trained semantic segmentation model.

[0033] Based on the aforementioned problems, embodiments of this application provide a training method, apparatus, electronic device, and storage medium for a semantic segmentation model. The method aims to minimize the first similarity between the text features of the text labels of each first object category in the foreground of each first sample image and the image features of the category representation map (the activated image region in the category activation map). This trains the model so that the regions in the category activation map extracted by the model that are related to each first object category are activated, while erroneous activation of irrelevant regions is suppressed, thereby improving the accuracy of the category activation map. This, in turn, improves the accuracy and efficiency of weakly supervised image semantic segmentation, and at the same time, provides a large number of automatically generated semantic segmentation labels for supervised semantic segmentation tasks at a low cost.

[0034] It should be noted that, in the optional embodiments of this application, the data related to object information (e.g., image data collected during driving) requires the permission or consent of the object when the embodiments of this application are applied to specific products or technologies. Furthermore, the collection, use, and processing of the relevant data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. In other words, if the embodiments of this application involve data related to the object, it must be obtained with the object's authorization and consent, the authorization and consent of relevant departments, and in accordance with the relevant laws, regulations, and standards of the country and region. If the embodiments involve personal information, the acquisition of all personal information requires the individual's consent. If sensitive information is involved, the separate consent of the information subject is required. The embodiments also need to be implemented with the object's authorization and consent.

[0035] The semantic segmentation model training method provided in this application embodiment can theoretically be applied to any scenario requiring semantic segmentation, including but not limited to navigation scenarios in autonomous driving. As an example, Figure 1 A schematic diagram of an implementation environment to which this application embodiment applies is shown, such as... Figure 1 As shown, the implementation environment includes a first training server 10, an annotation server 20, a second training server 30, a navigation server 40, and a user terminal 50. The first training server 10 trains a first semantic segmentation model based on a first training dataset (including multiple first sample images) and its image-level labels (text labels of each first object category in the foreground of the first sample images). The trained first semantic segmentation model is then deployed to the annotation server 20. The annotation server 20 performs semantic segmentation on multiple second sample images in the second training dataset based on the trained first semantic segmentation model, obtaining class activation maps for each second sample image, and obtaining pixel-level labels for each second sample image based on the class activation maps. However, since the first semantic segmentation model is trained based on image-level labels, its accuracy is still lower compared to semantic segmentation models trained using pixel-level labels.

[0036] The second training server 30 continues to train the second semantic segmentation model based on the second training dataset and its pixel-level labels, and deploys the trained second semantic segmentation model to the navigation server 40. When the navigation server 40 receives the road image sent by the user terminal 50 in the navigation request, it can perform semantic segmentation on the road image through the second semantic segmentation model to identify road condition information such as pedestrians, vehicles and road signs. Based on the identified road condition information, it plans a navigation route and returns it to the user terminal 50 so that the user terminal 50 can drive along the planned navigation route.

[0037] The aforementioned servers can be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The aforementioned user terminals (also referred to as user devices) can be smartphones, tablets, laptops, desktop computers, intelligent voice interaction devices (e.g., smart speakers), wearable electronic devices (e.g., smartwatches), in-vehicle terminals, smart home appliances (e.g., smart TVs), AR / VR devices, etc., but are not limited to these. Terminals and servers can be directly or indirectly connected via wired or wireless network communication methods, which is not limited herein.

[0038] The technical solutions of this application and their effects are described below through several embodiments. It should be noted that the following embodiments can be referenced, learned from, or combined with each other. Identical terms, similar features, and similar implementation steps in different embodiments will not be repeated.

[0039] Figure 2 This illustration shows a flowchart of a training method based on a semantic segmentation model provided in an embodiment of this application. This method can be executed by any electronic device, such as a user terminal or a server. Figure 1 The first training server 10 shown is executing.

[0040] like Figure 2 As shown, the training method for the semantic segmentation model provided in this application embodiment may include the following steps S110 to S150.

[0041] S110: Obtain multiple first sample images with labels.

[0042] When training the model, a first training dataset can be obtained. This first training dataset includes multiple first sample images, each containing one or more categories of objects, such as animals, plants, vehicles, etc. This application does not limit the method or source of obtaining the first training dataset; for example, existing image datasets such as the PASCAL VOC 2012 dataset or the MS COCO dataset can be used.

[0043] It is understandable that using a trained semantic segmentation model to segment an image involves classifying / segmenting the image into multiple predefined candidate object categories, thereby determining which target category the image contains. If there are M (M≥2) predefined candidate object categories, they can typically be divided into M-1 foreground categories (i.e., first candidate object categories) and background (object) categories. For example, when using the PASCAL VOC 2012 dataset as the first training dataset, the corresponding candidate object categories include: people, birds, cats, cows, dogs, horses, sheep, airplanes, bicycles, ships, buses, cars, motorcycles, trains, bottles, chairs, dining tables, potted plants, sofas, and televisions / monitors, totaling 20 foreground first candidate object categories, as well as background (object) categories. For an image containing cars, motorcycles, and trees, since cars and motorcycles belong to the 20 first candidate object categories, they are considered foreground objects. The other objects besides cars and motorcycles are considered background objects, i.e., trees belong to the background category.

[0044] For each first sample image, the object category to which each object in the foreground of the first sample image belongs is determined from a set of preset first candidate object categories, and is taken as the first object category. Based on the text labels of each first object category included in the foreground of the first sample image, the label of the first sample image is determined.

[0045] For example, suppose the preset multiple first candidate object categories include 5 categories: cat, dog, cow, sheep, and pig. For an image of "a cat and a dog playing on the grass", the foreground of the image includes two first object categories, "cat" and "dog". The image can be labeled as "an image of a cat" or "an image of a dog".

[0046] It should be noted that this application does not restrict the way the object categories are divided. It can be a coarse-grained category division, such as animals and plants, or a fine-grained category division, such as cats, dogs, cows and sheep. The specific classification can be set as needed.

[0047] S120: For each first sample image, obtain the category activation map of each first object category in the first sample image through the first neural network model.

[0048] The first neural network model includes multiple convolutional layers. The first sample image is input into the first neural network model, and the high-dimensional feature map of the first sample image is output through the last convolutional layer. Based on the high-dimensional feature map of the first sample image and the weight parameters corresponding to each first object category, a category activation map of the first sample image corresponding to each first object category is generated.

[0049] It should be noted that the first neural network model includes weight parameters corresponding to each first candidate object category. For each first sample image, the first neural network model can generate a category activation map corresponding to each first candidate object category. Since the training objective of the semantic segmentation model in this embodiment is to segment more accurately the category activation maps of each first object category in the image, the category activation maps of each first object category can be determined from the category activation maps of each first candidate object output by the model. Thus, the training loss is calculated through subsequent processes based on the category activation maps of each first object category.

[0050] For each pixel in the category activation map of each object category, the pixel value represents the probability (or confidence) that the sample pixel belongs to that object category. The sample pixel is the pixel corresponding to that pixel in the first sample image. For example, assuming that each pixel in the category activation map of a certain object category corresponds to 4 pixels in the first sample image, then the pixel value of each pixel in the category activation map of that object category represents the probability that the corresponding 4 pixels in the first sample image belong to that object category.

[0051] S130: For each first sample image, obtain the category representation map of each first object category in the first sample image based on the category activation map of each first object category in the first sample image.

[0052] Specifically, for each first object category in each first sample image, a category representation map of the first object category is obtained based on the category activation map of the first object category and the first sample image. The category representation map of the first object category represents the activation image region of the first object category in the first sample image.

[0053] S140: Extract the text features of the text labels of each first object category and the image features of the category representation graph, and determine the first similarity between the text features of the text labels of each first object category and the image features of the category representation graph.

[0054] Optionally, a pre-trained visual-language model can be used to achieve cross-modal understanding between images and text. After obtaining the category representation map of each first object category and the text label of each first object category, for each first object category, the text features of the text label of the first object category are extracted by the text encoder in the pre-trained visual-language model; the image features of the category representation map of the first object category are extracted by the image encoder in the pre-trained visual-language model; based on the text features of the text label of the first object category and the image features of the category representation map, the first similarity between the text features of the text label of the first object category and the image features of the category representation map is obtained by the decoder in the pre-trained visual-language model.

[0055] This application does not impose a unique limitation on the specific network architecture of the visual-language model; it can be selected according to actual application requirements. Theoretically, the visual-language model can adopt any neural network model capable of achieving cross-modal understanding of text and images. As an optional solution, the visual-language model can adopt a pre-trained cross-language image matching model (CLIP).

[0056] Optionally, the visual-language model in this embodiment can be trained in the following ways: Obtain a third training dataset; this dataset includes multiple positive and negative sample pairs. Positive sample pairs are matching image-text pairs where the object in the image matches the object described in the text. Negative sample pairs are mismatched image-text pairs where the object in the image differs from the object described in the text. For example, the text description "a picture of a cat" and the corresponding cat picture can be considered a positive sample pair, while the text description "a picture of a cat" and the picture of a dog can be considered a negative sample pair.

[0057] For each sample pair in the third training dataset, the text features of the text in the sample pair are extracted by the text encoder of the third neural network model, and the image features of the image in the sample pair are extracted by the image encoder of the third neural network model. Based on the text features of the Chinese text and the image features of the image in the sample pair, the similarity between the text features of the Chinese text and the image features of the image in the sample pair is obtained through the decoder of the third neural network model. Since the image features and text features are in different feature spaces, when calculating the similarity between the text features and the image features, it is necessary to first map the text features and the image features to the same feature space and calculate the similarity between the text features and the image features that are in the same feature space.

[0058] With the goal of maximizing the similarity between image features and text features in each positive sample pair and minimizing the similarity between image features and text features in each negative sample pair, the third neural network model is iteratively trained to obtain a well-trained visual-language model.

[0059] S150: Determine the first training loss based on the first similarity corresponding to each first object category in each first sample image, and continuously train the first neural network model based on the total training loss to obtain the first semantic segmentation model.

[0060] The total training loss includes a first training loss, which represents the text features of the text labels of each first object category in the foreground of each first sample image and the degree of difference between the image features of the category representation map.

[0061] In this embodiment of the application, the category representation map of the first object category represents the activated image region of the first object category in the first sample image. By determining the first similarity between the text features of the text label of the first object category and the image features of the activated image region of the first object category, and training the first semantic segmentation model with the goal of maximizing the first similarity, the integrity of the activated regions of each first object category in the category activation map can be improved, and the activation of the background region in the category activation map can be reduced, thereby improving the accuracy of the category activation map and thus improving the accuracy of the image pixel-level annotation.

[0062] The training termination condition and the model's loss function can be configured according to requirements. For example, the training termination condition may include, but is not limited to, reaching a preset number of training iterations, loss function convergence (e.g., the model's training loss is less than a preset value, or the training loss is less than a preset value for multiple consecutive iterations), and the model's test metrics meeting preset metrics. The model's training loss represents the feature deviation between the text labels of each first object category in the foreground of the first sample image and the category representation map predicted by the model. By using labeled training samples and the model's training loss function, the model is trained in a supervised manner using the gradient descent algorithm. This allows the image features based on the category representation map predicted by the model to continuously approximate the text features of the true text labels of the first object categories, thereby obtaining a well-trained first semantic segmentation model that meets the needs of practical applications. Based on the semantic segmentation results of the first semantic segmentation, the image is labeled at the pixel level, that is, the object category to which each pixel in the image belongs is labeled.

[0063] based on Figure 2The training method of the semantic segmentation model shown aims to minimize the first similarity between the text features of the text labels of each first object category in the foreground of each first sample image and the image features of the category representation map (the activated image region in the category activation map). This method trains the model so that the regions in the category activation map extracted by the model that are related to each first object category are activated, while the false activation of irrelevant regions is suppressed. This improves the accuracy of pixel classification in the category activation map and can also provide a large number of automatically generated pixel-level semantic segmentation labels for supervised semantic segmentation tasks at a low cost.

[0064] Optionally, the total training loss may also include a second training loss, which characterizes the degree of difference between the text features of the text labels of each first object category in the foreground of each first sample image and the image features of the background representation map (the inactive image regions of each first object category in each first sample image).

[0065] The second training loss is obtained in the following way: For each first object category in each first sample image, a background activation map for the first object category is determined based on the category activation map of the first object category. For each pixel in the background activation map of the first object category, the pixel value represents the probability that the sample pixel does not belong to the first object category, and the sample pixel is the pixel corresponding to the pixel in the first sample image.

[0066] Secondly, based on the background activation map of the first object category, a background representation map of the first object category is obtained from the first sample image, wherein the background representation map of the first object category represents the inactive image region of the first object category in the first sample image. Then, image features of the background representation maps of each first sub-object category are extracted, and a second similarity is determined between the text features of the text labels of each first object category and the image features of the background representation maps. Alternatively, the second similarity can be obtained based on the text labels and background representation maps of the first object category using a visual-language model.

[0067] Finally, the second training loss is determined based on the second similarity corresponding to each first object category in each first sample image.

[0068] In this embodiment, by introducing a second training loss, the model is able to explore non-discriminative object regions (also known as image background regions, inactive image regions, or background representation maps) and suppress background regions activated in the category activation map. Furthermore, by minimizing the second similarity corresponding to each first object category, fewer pixels of the first object category are retained in the background representation map (non-discriminative object regions), while more pixels of the first object category are recovered in the category representation map of the first object category, thus improving the integrity of the activated object regions in the category activation map.

[0069] Optionally, the total training loss may also include a third training loss. This third training loss characterizes the difference between the image features of the category representation map of each first object category in the foreground of each first sample image and the text features of the text labels of each second object category corresponding to that first object category. Here, a second object category corresponding to a first object category refers to an object category that co-occurs (co-appears) with that first object category in the same image and belongs to the image background. Continuing with the example of an image from the PASCAL VOC 2012 dataset containing cars, motorcycles, and trees, in this image, "trees" is the second object category corresponding to "cars," and "trees" is also the second object category corresponding to "motorcycles."

[0070] The labels for the first sample images further include text labels for at least one second object category corresponding to each first object category in the first sample image. The second object category can be an object category in the background that appears only in the first sample image and shares the same background with the first object category, or it can be an object category in the background that appears in all first sample images of the first training dataset and shares the same background with the first object category. This embodiment of the application does not impose any limitations on this.

[0071] The third training loss is obtained in the following way: Extract the text features of the text labels for each second object category. For each second object category corresponding to each first object category, determine the third similarity between the image features of the category representation map of the first object category and the text features of the text labels of the second object category. Optionally, the third similarity can be obtained based on the category representation map of the first object category and the text labels of the second object category using a trained vision-language model.

[0072] The third training loss is determined based on the third similarity corresponding to each first object category in each first sample image.

[0073] Optionally, to suppress false activations in confusing background regions, when calculating the third training loss, the image features of the category representation map of each first object category in each first sample image are determined, and the third similarity is calculated between them and the text features of the text label of each second object category. From these, each third similarity greater than a preset similarity threshold is selected, and the third training loss is determined based on these third similarities. Specifically, if the third similarity between the image features of the category representation map of the first object category and the text features of the text label of the second object category is greater than the preset similarity threshold, it indicates that the second object category and the first object category are prone to confusion.

[0074] Optionally, at least one second object category corresponding to a first object category can be other object categories that are easily confused with the first object category in a context set based on human experience, or at least one second object category corresponding to a first object category can also be determined in the following ways: For each first sample image, an image description text is generated based on that first sample image using a trained image description model; Identify each object appearing in the image description text of each first sample image; From each object in the image description text, identify each third object that does not belong to each of the first candidate object categories; Determine the category of each third object; From each third object category, determine the third object category that co-occurs with the first object category in each first sample image, and use it as at least one second object category corresponding to the first object category.

[0075] In this alternative approach, the image description text is used to describe information such as the objects appearing in the image, their behavior, or the emotions they express. By parsing the image description text, the entities included in the image description text are obtained, i.e., the objects in the first sample image. For example, for an image whose image description text is "a white train is traveling on the tracks," the objects appearing in the image description text are: train and tracks.

[0076] In one alternative implementation, when classifying third objects from the various objects in each image description text, a mapping relationship between each first candidate object category and the objects within that category can be pre-constructed. For example, the "airplane" category may include helicopters, gliders, etc. For each object in each image description text, the first candidate object category to which the object belongs is determined based on the object category mapping relationship. If the object does not belong to any first candidate object category, then the object is treated as a third object.

[0077] In another alternative implementation, when dividing the third object from each object in each image description text, each object in each image description text and each first candidate object category (category label) can be input into a text encoder to extract the text features of each object and the text features of each first candidate object category respectively; then, for each first candidate object category, the similarity between the text features of each object and the text features of the first candidate object category is determined, and each object with a similarity greater than or equal to a first threshold is regarded as a synonym / synonymous object of the first candidate object category, and each object with a similarity less than the first threshold is regarded as the third object (i.e., the object in the background).

[0078] Optionally, for each first object category, when selecting the corresponding second object categories from the third object categories, the co-occurrence frequency of the first object category and each third object category in each first sample image can be counted. The first third object category is then selected as the at least one second object category corresponding to the first object category, sorted from highest to lowest co-occurrence frequency. For example, if "airplane" co-occurs frequently with "airport" and "sky" in the images, "airport" and "sky" can be used as the second object categories corresponding to "airplane".

[0079] It is understandable that the second object category corresponding to each first object category can also be set based on human experience, using object categories in the context that are easily confused with the first object category.

[0080] This application does not limit the model structure of the image description model. As an optional approach, the OFA model can be used to generate the title of each first sample image, which is the image description text of each first sample image.

[0081] To avoid overly dispersed activation regions in the category activation map, this embodiment of the application also introduces a regularization loss into the total training loss to limit the size of the activation regions in the category activation map. Based on the category activation map of each first object category in each first sample image, the average probability that each pixel belongs to the first object category is determined. Based on the average probability that each pixel belongs to the first object category, a regularization loss is determined. This regularization loss is used to constrain the area size of objects of the first object category in the foreground of the first sample image.

[0082] To facilitate understanding, the training method of the above neural network model is described below using a complete embodiment, such as... Figure 3 As shown, where, Figure 3 (a) represents the extraction of the category activation map (CAM). Figure 3(b) represents the calculation of the matching loss between the object region and the text label, and between the background region and the text label, i.e., the calculation of the first training loss and the second training loss. Figure 3 (c) represents the calculation of background suppression loss, i.e. the calculation of the third training loss.

[0083] Suppose the first sample image is X, an image of "a white train traveling on the tracks", and the text labels for each first candidate object category are: The text label for the l-th second object category corresponding to each first candidate object category is: .in, K represents the total number of first candidate object categories. L represents the total number of second object categories corresponding to the first candidate object category.

[0084] (a) Extracting Category Activation Map (CAM) The first sample image X is input into the backbone network. Features of the first sample image X are extracted through multiple convolutional layers in the backbone network, resulting in a high-dimensional feature map Z output by the last convolutional layer. C represents the number of channels, H×W represents the spatial dimension of the feature, and the high-dimensional feature map Z includes feature maps corresponding to multiple channels.

[0085] For each first candidate object category in the foreground, taking the k-th first candidate object category as an example, based on the high-dimensional feature map Z of the first sample image X and the weight matrix corresponding to the k-th first candidate object category... Calculate the category activation map of the first sample image X corresponding to the k-th first candidate object category. .

[0086] This category's activation graph The pixel value of each pixel in the image can be represented as:

[0087] in, This represents the coordinates of any pixel in the category activation graph. This represents the feature representation of the pixel in the high-dimensional feature map Z. This represents the transpose of the weight matrix for the k-th first candidate object category. , This represents the probability that the pixel belongs to the k-th first candidate object category.

[0088] (b) Calculate the first training loss and the second training loss Based on the category activation graph of the k-th first candidate object category And the first sample image X, to obtain the category representation map of the k-th first candidate object category. The class representation map is extracted using the CLIP model image encoder. Image features The text encoder of the CLIP model extracts the text label of the k-th first candidate object category. Text features Calculate the image features of the category representation graph of the k-th first candidate object category. Text features of text labels First similarity between Based on each first object category in the foreground of the first sample image X, a first similarity between the image features and text features of each first object category is determined, and a first training loss is determined based on the first similarity between the image features and text features of each first object category. .

[0089] It is understandable that when the kth first candidate object category is train, the train category represents an image area in the image that almost only shows trains. The image encoder of the CLIP model extracts the image features of the train category in the image. The text label of the train, "a picture of a train", is input into the text encoder of the CLIP model to obtain the text features of the train.

[0090] When the first training loss is maximized, it can be assumed that the activation region in the train category activation map (CAM) extracted by the model is close to the image region of the train in the first sample image, thus ensuring the integrity of the train activation.

[0091] Among them, the first training loss It can be represented as:

[0092]

[0093] in, This is used to indicate whether the first sample image X contains an object of the first candidate object category; if it does, then... =1, if not included =0; Representing image features Text features Cosine similarity between them.

[0094] Based on the category activation graph of the k-th first candidate object category Determine the background activation of the k-th first candidate object category. Figure 1 - Background activation based on the k-th first candidate object category Figure 1 - And the first sample image X, to obtain the background representation map of the kth first candidate object category. The background representation map is extracted using the CLIP model image encoder. Image features The text encoder of the CLIP model extracts the text label of the k-th first candidate object category. Text features Calculate the image features of the background representation map of the k-th first candidate object category. Text features of text labels Second similarity between Based on each first object category in the foreground of the first sample image X, a second similarity is determined between the image features of the background representation map of each first object category and the text features of the text label. Based on this second similarity, a second training loss is determined. .

[0095] It is understandable that when the kth first candidate object category is train, the background representation image of the train shows other background image areas besides the train. The image encoder of the CLIP model is used to extract the image features of the background in the background representation image of the train. The text label of the train, "a picture of a train", is input into the text encoder of the CLIP model to obtain the text features of the train.

[0096] When the second training loss is minimized, it can be assumed that the background region in the train category activation map CAM extracted by the model does not exist in the first sample image, thus avoiding the background region in the first sample image being incorrectly activated.

[0097] Among them, the second training loss It can be represented as:

[0098]

[0099] in, Representing image features Text features Cosine similarity between them.

[0100] (c) Calculate the third training loss Based on the category representation diagram of the k-th first candidate object category The class representation map is extracted using the CLIP model image encoder. Image features By using the CLIP model's text encoder, text features of the text labels for each of the second object categories corresponding to the k-th first candidate object category are extracted, and image features of the category representation map of the k-th first candidate object category are calculated. The third similarity between the text features of the text labels of their respective second object categories.

[0101] Among them, each second object category corresponding to each first candidate object category can be a background object category extracted from multiple first sample images that is easily confused with each first candidate object category.

[0102] Text label of the l-th second object category For example, the text label of the l-th second object category The text features are represented as Calculate the category representation diagram Image features , with text labels Text features Third similarity between .

[0103] Based on the categories of each first object in the foreground of the first sample image X, determine the category representation map of each first object category. Image features The third training loss is determined based on the third similarity between the text features of the text labels of each second object category and the third similarity between each first object category and the third similarity of each third similarity that is greater than a preset similarity threshold. .

[0104] It is understandable that when the kth first candidate object category is train, the train category represents an image area in the image that almost only shows trains. The second object category that is easily confused with trains in the background is track. The image encoder of the CLIP model extracts the image features of trains in the image representing the category of trains. The text label of track, "a photo of a track", is input into the text encoder of the CLIP model to obtain the text features of track.

[0105] When the third training loss is minimized, it can be assumed that the activation regions in the train category activation map (CAM) extracted by the model do not contain tracks, thus avoiding the incorrect activation of tracks.

[0106] Among them, the third training loss It can be represented as:

[0107]

[0108]

[0109] in, express and Cosine similarity between them This indicates a preset similarity threshold. hour, =1, otherwise =0, It is to ensure third similarity Masks with similarity greater than a preset threshold.

[0110] In addition, in this embodiment, the total training loss also includes regularization loss. (Not shown in the figure), this regularization loss It can be represented as:

[0111]

[0112] Where H and W represent the height and width of the category activation map, This represents the average probability that a pixel belongs to the k-th first candidate object category.

[0113] The total training loss of the first semantic segmentation model can be expressed as:

[0114] in, , , and These are the hyperparameter weights corresponding to each loss term.

[0115] Figure 4 This is a flowchart illustrating an automatic generation of obfuscated backgrounds provided in an embodiment of this application. It involves generating image description information for each first sample image using an image description model, and then parsing the generated image description information to obtain the objects within each image description. For example... Figure 4 As shown in (a), for the first image, the generated image description information is "an airplane is taking off from the airport runway". The objects included in the image description information are: airplane, airport, and runway.

[0116] Then, based on the objects in each image description and each first candidate object category, the text features of each object and each first candidate object category are extracted using a text encoder. The similarity between each object and each first candidate object category is calculated. Using each first candidate object category as an anchor point, and based on the calculated similarities, foreground objects belonging to each first candidate object category and background objects not belonging to any first candidate object category are determined. For example... Figure 4 As shown in (b), the objects extracted from the image description information include: airplane, airport, runway, jet, etc. The preset first candidate object categories include: airplane, etc. Based on the similarity between text features, each object is divided into foreground objects such as airplane and jet, and background objects such as airport and runway.

[0117] Finally, based on the image description information, the number of times each first candidate object category and each background object co-occurs in each first sample image is counted (the number of times they appear together). These are then sorted from highest to lowest co-occurrence count to filter background objects that are easily confused with each first candidate object category. For example... Figure 4 As shown in (c), the image description information of each generated first sample image includes: "Air Force planes on the airport runway", "Fighter jets flying in the air", and "A group of planes parked on the runway". Among them, the runway, airport, sky and so on appear more often with "planes". The runway, airport and sky that appear first in the order are used as the obfuscated background objects corresponding to the planes.

[0118] Figure 5 This illustration shows a flowchart of a training method based on a semantic segmentation model provided in an embodiment of this application. This method can be executed by any electronic device, such as a user terminal or a server. Figure 1 The second training server 30 shown is executed.

[0119] like Figure 5 As shown, the training method for the semantic segmentation model provided in this application embodiment may include the following steps S310 to S320.

[0120] Step S310: Obtain multiple second sample images with labels.

[0121] Step S320: Based on multiple second sample images, continuously perform training operations on the second neural network model to obtain a trained second semantic segmentation model.

[0122] Specifically, a second training dataset is obtained, and the second semantic segmentation model is trained using the second training dataset. The second training dataset includes multiple second sample images, and the label of each second sample image is used to indicate the object category to which each pixel in the second sample image belongs.

[0123] Wherein, for each second sample image, the label of the second sample image can be obtained by the method shown in steps S410 to S420: Step S410: Based on the second sample image, the category activation map of the second sample image corresponding to each first candidate object category is obtained through the trained first semantic segmentation model.

[0124] Step S420: Based on the category activation map corresponding to each first candidate object category of the second sample image, obtain the label of the second sample image.

[0125] The first semantic segmentation model used in this embodiment can be trained using any of the training methods shown in the above embodiments, as detailed in steps S110-S120, which will not be repeated here. Since the first semantic segmentation model is trained based on image-level labels, its accuracy is still lower than that of a semantic segmentation model trained based on pixel-level annotations. Image-level annotations indicate the object categories included in the image, while pixel-level annotations indicate the object category to which each pixel in the image belongs.

[0126] Therefore, in this embodiment of the application, a first semantic segmentation model is used to perform semantic segmentation on each second sample image in the second training dataset to obtain semantic segmentation results. Based on the semantic segmentation results, pixel-level labels of each second sample image are obtained, so as to train a second semantic segmentation model with higher accuracy based on each second sample image and its pixel-level labels.

[0127] It should be noted that the first sample image used to train the first semantic segmentation model and the second sample image used to train the second semantic segmentation model can use the same sample image data or different sample image data. This application embodiment does not limit this, and it can be set as needed.

[0128] Optionally, the size of the category activation map obtained by the first semantic segmentation model is smaller than the image size of the original second sample image. Therefore, the category activation map can be restored to the size of the original image through post-processing methods such as upsampling. At this time, each pixel in the category activation map corresponds to each pixel in the second sample image.

[0129] In this diagram, the pixel value of each pixel in the category activation map of the first candidate object category represents the probability that the pixel belongs to that first candidate object category. Based on the category activation map of the second sample image corresponding to each first candidate object category, the probability that each pixel in the second sample image belongs to each first candidate object category is determined. Based on the probability that each pixel belongs to each first candidate object category, the object category to which each pixel belongs is determined. For example, assuming the first candidate object categories include train, motorcycle, and truck, for a pixel in the second sample image, the probability that the pixel belongs to a train is determined from the category activation map corresponding to a train, the probability that the pixel belongs to a motorcycle is determined from the category activation map corresponding to a motorcycle, and the probability that the pixel belongs to a truck is determined from the category activation map corresponding to a truck. If the probability that the pixel belongs to a train is 0.8, and the probabilities that it belongs to a motorcycle and a truck are both 0.1, then the pixel belongs to a train.

[0130] It should be noted that if the probability of a pixel belonging to any of the first candidate object categories is less than a preset value, then the pixel can be considered to belong to the background category.

[0131] This application does not impose a unique limitation on the specific network architecture of the semantic segmentation model; it can be selected according to actual application requirements. Theoretically, any neural network model capable of recognizing the category and location of objects in an image can be used for semantic segmentation.

[0132] based on Figure 5 The semantic segmentation model trained using this method can be applied to any application scenario requiring semantic segmentation of images to identify the category of each pixel, including but not limited to navigation and medical scenarios. When applied to navigation, semantic segmentation of acquired road images can identify pedestrians, vehicles, and road signs, aiding in route planning. In medical settings, it can perform semantic segmentation on patient medical images to accurately locate and identify organs or lesions, assisting doctors in precise diagnosis and treatment. For example, in diagnosing lung nodules, semantic segmentation of lung CT images identifies the size and boundaries of nodules, helping doctors determine their benign or malignant nature. During liver resection surgery, semantic segmentation of chest CT scans identifies the boundaries of organs such as the heart and liver, helping doctors plan surgical paths, determine the surgical scope, and improve surgical accuracy and safety.

[0133] based on Figure 5The training method of the semantic segmentation model shown herein employs a first semantic segmentation model to perform pixel-level annotation on sample images. Since the first semantic segmentation model is trained based on image-level labels, and during the training phase, the training loss is determined based on the first similarity between the text features of the text labels of each object category in the foreground of the sample image and the image features of the activated image regions in the category activation map. The model is then trained based on the training loss, which activates the regions in the category activation map that are related to each first object category, suppresses the false activation of irrelevant regions, and improves the accuracy of pixel classification in the category activation map. This, in turn, improves the accuracy and efficiency of weakly supervised image semantic segmentation, while also providing a large number of automatically generated semantic segmentation labels for supervised semantic segmentation tasks at a low cost.

[0134] To evaluate the annotation performance of the first semantic segmentation model trained using the training method of this application, experiments were conducted on the first semantic segmentation model using the PASCAL VOC 2012 and MS COCO 2014 datasets. PASCAL VOC 2012 is a semantic segmentation dataset containing 20 object categories, including 1464 images for training, 1449 images for validation, and 1456 images for testing. The COCO dataset contains 80 object categories, including 82783 training images and 40504 validation images. In the experiments, the mean Intersection over Union (mIoU) percentage was used as the evaluation metric.

[0135] Ablation experiments were conducted on each training loss term, and the experimental results are shown in Table 1 below. The existing scheme uses training loss... During model training, the mIoU between the activation regions in the extracted class activation maps and the real object regions was only 28.6%. This was further investigated after adding the first training loss. Afterwards, mIoU significantly improved to 51.3%, and a second training loss was added. Third training loss Regularization loss All of these bring a certain degree of improvement. The training loss terms constructed in the embodiments of this application enable the model to generate more accurate class activation maps.

[0136]

[0137] Table 1 Table 2 compares the quality of the class activation maps generated by our proposed method with several other existing methods. The backbone networks used in each method are WideResNet38 (WR38), Vision Transformer-B (ViT-B), and ResNet50 (RN50), respectively. Mask represents the mask label obtained after post-processing the class activation map CAM. The comparison shows that the Mask generated by our proposed method achieves an accuracy of 75.1%, which is a significant improvement over other existing methods.

[0138]

[0139] Table 2 Table 3 shows the evaluation results of semantic segmentation models trained using our proposed method and several other existing methods on the MS COCO 2014 dataset. The segmentation network architectures used in each method include DeepLabV2 (V2), DeepLabV1 (V1), and WR38, while the backbone network architectures include ViT, WR38, V16, and RN50. Specifically, methods 16-19 use image-level labels plus saliency maps for supervision, methods 1, 8, 10, and 20-23 use only image-level labels for supervision, and our proposed method uses a combination of image-level labels and language text labels for supervision. As shown in Table 3, when validating the trained segmentation network using a validation set, the semantic segmentation model trained using our proposed method achieves significantly better results than methods using only image-level labels for supervision, with an mIoU of 42.5% with the ground truth labels.

[0140]

[0141] Table 3 Table 4 shows the evaluation results of the semantic segmentation models trained by our scheme and several other existing schemes on the PASCAL VOC 2012 dataset. The network structures of the segmentation networks used by each scheme are as follows: , The backbone network architectures used in V2, V1, and WR38 include ViT-B, WR38, V16, and RN50. Among them, and These represent the pre-trained V1 and V2, respectively.

[0142] In Table 4, schemes 24-26 use pixel-level labels for supervision, schemes 17-19 and 27-32 use image-level labels + saliency maps for supervision, schemes 3, 4, 6-12, 22, and 33-38 use only image-level labels for supervision, and scheme 39 and embodiments 1-5 of this scheme use a combination of image-level labels and language text labels for supervision.

[0143] In this scheme, Embodiments 1 and 3 represent the evaluation results of semantic segmentation models trained based on setting confusing background categories (second object categories) using human experience. Embodiments 2, 4, and 5 represent the evaluation results of semantic segmentation models trained based on generating confusing background categories using image description models. The segmentation network in Embodiment 1 uses a V2 network structure, while the segmentation network in Embodiment 3 uses a pre-trained... The network structure shows that pre-training is used. A semantic segmentation model trained with a network structure can achieve more accurate semantic segmentation results.

[0144] The segmentation network in Embodiment 2 of this scheme adopts a V2 network structure, while the segmentation network in Embodiment 4 adopts a pre-trained network structure. Regarding the network structure, the segmentation network in Example 5 adopts the WR38 network structure. As can be seen from the comparison, the segmentation network using the WR38 network structure can bring better segmentation results, achieving 73.8% mIoU in both the validation and test sets, which surpasses other existing schemes that use image-level labels or additional language text labels for supervised training.

[0145]

[0146] Table 4 Figure 6 This diagram illustrates the semantic segmentation results obtained by segmenting an input image using the scheme described in this application. It shows that the first semantic segmentation model trained using this scheme generates pseudo-segmentation labels for the image. The semantic segmentation model trained based on these pseudo-segmentation labels can then accurately segment objects in the image. For example, the airplane in the first column is accurately segmented, and the bird in the fifth column is accurately segmented, avoiding the misactivation of background elements such as tree trunks.

[0147] exist Figure 7In this paper, the category activation map (CAM) generated using the proposed method is compared with the initial CAM and the existing CAM generated solely based on image-level labels. As shown in the figures, the CAM generated using the proposed method activates more complete object regions and fewer class-related background regions. As illustrated in columns 2 and 5, the initial CAM underestimates the regions of ships and airplanes, while the CAM generated based on image-level labels incorrectly activates the river region. In comparison, the category activation map generated by the proposed method is more complete and compact. In columns 3 and the last two columns, the initial CAM and the CAM generated solely based on image-level labels show a large number of background object categories related to the object category being identified being incorrectly activated, namely docks, airport runways, and city roads. The proposed method calculates a third training loss based on each second object category (a background category prone to confusion) corresponding to each first object category. Under the supervision of this third training loss, the incorrect activation of spuriously related regions can be effectively reduced, resulting in a complete and compact CAM.

[0148] The total training loss provided in this embodiment includes a first training loss, a second training loss, a third training loss, and a regularization loss. These different loss functions play different roles in constraining the CAM generated by the backbone network. Ablation experiments can further verify the effectiveness of each loss function, such as... Figure 8 As shown, Figure 8 The second column shows the results of model training using only the first training loss. It can be observed that only salient regions of the object are activated in the CAM, such as the head and tail of the aircraft in the first row, but the fuselage is not activated. Figure 8 The third column shows the result after adding a second training loss on top of the first training loss. It can be seen that the activated regions of objects are significantly increased in the generated CAM, such as the front end of the train being activated in the second column.

[0149] However, based solely on the first and second training losses, the generated CAM incorrectly activates object-related background elements; for example, the runway in the first row, the railway in the second row, and the dock in the third row are all incorrectly activated. After adding a third training loss and a regularization loss, as shown in columns 4 and 5 of the figure, the generated CAM reduces some background misidentification. Furthermore, after adding the third training loss, the generated CAM (such as...) Figure 8 The fourth column of the CAM can significantly exclude relevant backgrounds (such as railways, runways, and docks) from the CAM, effectively constraining the size of the activation area. The activation area in the constrained CAM is very close to the real label.

[0150] based on Figure 2The semantic segmentation model provided follows the same training method. This application embodiment provides a training apparatus for a semantic segmentation model, such as... Figure 9 As shown, the training device 500 of the semantic segmentation model may include: a first sample acquisition module 510, a category activation map generation module 520, a category representation map generation module 530, a first similarity calculation module 540, and a training module 550. Wherein: The first sample acquisition module 510 is used to acquire multiple first sample images with labels, wherein the label of each first sample image includes the text label of each first object category included in the foreground of the first sample image; The category activation map generation module 520 is used to obtain the category activation map of each first object category in the first sample image through a first neural network model for each first sample image; The category representation map generation module 530 is used to obtain a category representation map of each first object category in the first sample image for each first sample image based on the category activation map of each first object category in the first sample image; The first similarity calculation module 540 is used to extract the text features of the text labels of each first object category and the image features of the category representation graph, and determine the first similarity between the text features of the text labels of each first object category and the image features of the category representation graph. The training module 550 is used to determine a first training loss based on the first similarity corresponding to each first object category in each first sample image, and to continuously train the first neural network model based on the total training loss to obtain a first semantic segmentation model; wherein the total training loss includes the first training loss.

[0151] Optionally, the training device 500 further includes a second similarity calculation module, which can be used for: For each first object category in each first sample image, a background activation map of the first object category is determined based on the category activation map of the first object category; and a background representation map of the first object category is obtained based on the background activation map of the first object category. Extract image features from the background representation map of each first object category, and determine the second similarity between the text features of the text label of each first object category and the image features of the background representation map; The training module 550 can also be used for: The second training loss is determined based on the second similarity corresponding to each first object category in each of the first sample images; The total training loss also includes the second training loss.

[0152] Optionally, the label for each first sample image further includes: a text label for at least one second object category corresponding to each first object category, wherein the second object category is an object category that co-occurs with the first object category in the background of the image; The training device 500 further includes a third similarity calculation module, which can be used for: Extract the text features of the text labels for each second object category; For each second object category corresponding to each first object category, a third similarity is determined between the image features of the category representation map of the first object category and the text features of the text label of the second object category; The training module 550 can also be used for: The third training loss is determined based on the third similarity corresponding to each first object category in each of the first sample images; The total training loss also includes the third training loss.

[0153] Optionally, at least one second object category corresponding to a first object category is determined in the following way: For each of the first sample images, an image description text is generated based on the first sample image using a trained image description model; Identify each object appearing in the image description text of each of the first sample images; From each of the objects in the image description text, determine each third object that does not belong to each of the first candidate object categories; Determine the category of each of the aforementioned third objects; From each of the third object categories, determine the third object category that co-occurs with the first object category in each first sample image, and use it as at least one second object category corresponding to the first object category.

[0154] Optionally, the third similarity calculation module can also be used for: The number of times each of the first object category and each of the third object categories co-occurred in each of the first sample images was determined. Based on the order of co-occurrence frequency of each of the aforementioned third object categories from highest to lowest, at least one third object category with the highest ranking is selected as at least one second object category corresponding to the first object category.

[0155] Optionally, the training module 550 can also be used for: Determine the third similarity corresponding to each first object category in each of the first sample images; The third training loss is determined based on each third similarity score that is greater than a preset similarity threshold.

[0156] Optionally, the training device 500 further includes a regularization loss calculation module, which can be used for: Based on the category activation map of each first object category in each first sample image, determine the average probability that the sample pixel corresponding to each pixel belongs to the first object category; The regularization loss is determined based on the average probability that each pixel belongs to the first object category; the regularization loss is used to constrain the area size of the first object category in the foreground of the first sample image. The total training loss also includes the regularization loss.

[0157] based on Figure 9 The training device for the semantic segmentation model shown determines the training loss based on the first similarity between the text features of the text labels of each first object category in each first sample image and the image features of the category representation map during the model training process. This trains the model so that the regions related to each first object category in the category activation map extracted by the model are activated, while erroneous activation of irrelevant regions is suppressed. This improves the accuracy of pixel classification in the category activation map and can also provide a large number of automatically generated pixel-level semantic segmentation labels for supervised semantic segmentation tasks at a low cost.

[0158] based on Figure 5 The semantic segmentation model provided follows the same training method. This application embodiment provides a training apparatus for a semantic segmentation model, such as... Figure 10 As shown, the training device 600 for the semantic segmentation model may include: a second sample acquisition module 610 and a model training module 620. Wherein: The second sample acquisition module 610 is used to acquire multiple labeled second sample images; the label of each second sample image is used to indicate the object category to which each pixel in the second sample image belongs; The model training module 620 is used to continuously perform training operations on the second neural network model based on multiple second sample images to obtain a trained second semantic segmentation model. Specifically, the label for each of the second sample images is obtained in the following way: Based on the second sample image, the category activation map of the second sample image corresponding to each first candidate object category is obtained by using the trained first semantic segmentation model. The first semantic segmentation model is trained using the training method of the semantic segmentation model provided in any of the above embodiments. The first candidate object category is the object category that can be identified in the foreground. The label of the second sample image is obtained based on the category activation map corresponding to each first candidate object category of the second sample image.

[0159] based on Figure 10 The training device for the semantic segmentation model shown uses a first semantic segmentation model to perform pixel-level annotation on sample images. Since the first semantic segmentation model is trained based on image-level labels, during the training phase, the training loss is determined based on the first similarity between the text features of the text labels of each object category in the foreground of the sample image and the image features of the activated image regions in the category activation map. This allows the model to be trained so that the regions in the category activation map extracted by the model that are related to each first object category are activated, while the false activation of irrelevant regions is suppressed. This improves the accuracy of pixel classification in the category activation map, thereby improving the accuracy and efficiency of weakly supervised image semantic segmentation. At the same time, it can also provide a large number of automatically generated semantic segmentation labels for supervised semantic segmentation tasks at a low cost.

[0160] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0161] This application provides an electronic device, including a memory, a processor, and a computer program stored in the memory. When the processor executes the computer program stored in the memory, it can implement the method in any optional embodiment of this application.

[0162] Figure 11 A schematic diagram of the structure of an electronic device to which an embodiment of the present invention applies is shown, such as... Figure 11 As shown, the electronic device can be a server or a user terminal, and it can be used to implement the methods provided in any embodiment of the present invention.

[0163] like Figure 11 As shown, the electronic device 2000 may primarily include at least one processor 2001. Figure 11 The diagram shows components such as a memory 2002, a communication module 2003, and an input / output interface 2004. Optionally, these components can be connected and communicate with each other via a bus 2005. It should be noted that... Figure 11The structure of the electronic device 2000 shown is merely illustrative and does not constitute a limitation on the electronic devices to which the methods provided in the embodiments of this application are applicable.

[0164] The memory 2002 can be used to store operating systems and applications, etc. The applications can include computer programs that implement the methods shown in the embodiments of the present invention when invoked by the processor 2001, and can also include programs for implementing other functions or services. The memory 2002 can be ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices that can store information and computer programs, or it can be EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disk storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer, but is not limited thereto.

[0165] Processor 2001 is connected to memory 2002 via bus 2005, and implements corresponding functions by calling application programs stored in memory 2002. Processor 2001 can be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this invention. Processor 2001 can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0166] Electronic device 2000 can connect to a network via communication module 2003 (which may include, but is not limited to, components such as a network interface) to communicate with other devices (such as user terminals or servers) through the network and achieve data interaction, such as sending data to or receiving data from other devices. Communication module 2003 may include wired network interfaces and / or wireless network interfaces, meaning the communication module may include at least one of wired or wireless communication modules.

[0167] Electronic device 2000 can connect to required input / output devices, such as keyboards and display devices, via input / output interface 2004. Electronic device 2000 itself may have a display device, and other display devices can also be connected externally via interface 2004. Optionally, storage devices, such as hard drives, can also be connected via interface 2004 to store data from electronic device 2000, retrieve data from storage device, or store data from storage device into memory 2002. It is understood that input / output interface 2004 can be a wired interface or a wireless interface. Depending on the actual application scenario, the device connected to input / output interface 2004 can be a component of electronic device 2000 or an external device connected to electronic device 2000 when needed.

[0168] The bus 2005 used to connect the components may include a pathway for transmitting information between the components. The bus 2005 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Depending on its function, the bus 2005 can be divided into address bus, data bus, control bus, etc.

[0169] Optionally, for the solution provided in the embodiments of the present invention, the memory 2002 can be used to store a computer program that executes the solution of the present invention, and the processor 2001 runs the computer program. When the processor 2001 runs the computer program, it implements the operation of the method or apparatus provided in the embodiments of the present invention.

[0170] Based on the same principle as the method provided in the embodiments of this application, the embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, can implement the corresponding content of the aforementioned method embodiments.

[0171] This application also provides a computer program product, which includes a computer program that, when executed by a processor, can implement the corresponding content of the aforementioned method embodiments.

[0172] It should be noted that the terms "first," "second," "third," "fourth," "1," "2," etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in a sequence other than that shown in the figures or text.

[0173] It should be understood that although arrows indicate various operation steps in the flowcharts of this application's embodiments, the order in which these steps are implemented is not limited to the order indicated by the arrows. Unless explicitly stated herein, in some implementation scenarios of this application's embodiments, the implementation steps in each flowchart can be executed in other orders as required. Furthermore, some or all steps in each flowchart may include multiple sub-steps or multiple stages based on the actual implementation scenario. Some or all of these sub-steps or stages can be executed at the same time, and each sub-step or stage can also be executed at different times. In scenarios where execution times differ, the execution order of these sub-steps or stages can be flexibly configured according to requirements, and this application's embodiments do not limit this.

[0174] The above description is only an optional implementation method for some implementation scenarios of this application. It should be noted that for those skilled in the art, other similar implementation methods based on the technical concept of this application without departing from the technical concept of this application also fall within the protection scope of the embodiments of this application.

Claims

1. A training method for a semantic segmentation model, characterized in that, include: Acquire multiple first sample images with labels, wherein the label of each first sample image includes the text label of each first object category included in the foreground of the first sample image; For each of the first sample images, a category activation map of each first object category in the first sample image is obtained through a first neural network model; For each of the first sample images, a category representation map of each first object category in the first sample image is obtained based on the category activation map of each first object category in the first sample image; The text features of the text labels of each first object category and the image features of the category representation graph are extracted respectively, and the first similarity between the text features of the text labels of each first object category and the image features of the category representation graph is determined. Based on the first similarity corresponding to each first object category in each of the first sample images, a first training loss is determined, and based on the total training loss, the first neural network model is continuously trained to obtain a first semantic segmentation model; wherein, the total training loss includes the first training loss.

2. The method according to claim 1, characterized in that, The method further includes: For each first object category in each first sample image, a background activation map of the first object category is determined based on the category activation map of the first object category; and a background representation map of the first object category is obtained based on the background activation map of the first object category. Extract image features from the background representation map of each first object category, and determine the second similarity between the text features of the text label of each first object category and the image features of the background representation map; The second training loss is determined based on the second similarity corresponding to each first object category in each of the first sample images; The total training loss also includes the second training loss.

3. The method according to claim 1 or 2, characterized in that, The label for each of the first sample images further includes: a text label for at least one second object category corresponding to each of the first object categories, wherein the second object category is an object category that co-occurs with the first object category in the background of the image; The method further includes: Extract the text features of the text labels for each second object category; For each second object category corresponding to each first object category, a third similarity is determined between the image features of the category representation map of the first object category and the text features of the text label of the second object category; The third training loss is determined based on the third similarity corresponding to each first object category in each of the first sample images; The total training loss also includes the third training loss.

4. The method according to claim 3, characterized in that, At least one second object category corresponding to a first object category is determined in the following way: For each of the first sample images, an image description text is generated based on the first sample image using a trained image description model; Identify each object appearing in the image description text of each of the first sample images; From each of the objects in the image description text, determine each third object that does not belong to each of the first candidate object categories; Determine the category of each of the aforementioned third objects; From each of the third object categories, determine the third object category that co-occurs with the first object category in each first sample image, and use it as at least one second object category corresponding to the first object category.

5. The method according to claim 4, characterized in that, The step of determining, from each of the third object categories, the object category co-occurring with the first object category in each of the first sample images, as at least one second object category corresponding to the first object category, includes: The number of times each of the first object category and each of the third object categories co-occurred in each of the first sample images was determined. Based on the order of co-occurrence frequency of each of the aforementioned third object categories from highest to lowest, at least one third object category with the highest ranking is selected as at least one second object category corresponding to the first object category.

6. The method according to claim 3, characterized in that, The step of determining the third training loss based on the third similarity corresponding to each first object category in each of the first sample images includes: Determine the third similarity corresponding to each first object category in each of the first sample images; The third training loss is determined based on each third similarity score that is greater than a preset similarity threshold.

7. The method according to claim 1, characterized in that, The method further includes: Based on the category activation map of each first object category in each first sample image, determine the average probability that the sample pixel corresponding to each pixel belongs to the first object category; The regularization loss is determined based on the average probability that each pixel belongs to the first object category; the regularization loss is used to constrain the area size of the first object category in the foreground of the first sample image. The total training loss also includes the regularization loss.

8. A training method for a semantic segmentation model, characterized in that, include: Acquire multiple labeled second sample images; the label of each second sample image is used to indicate the object category to which each pixel in the second sample image belongs; Based on multiple second sample images, the second neural network model is continuously trained to obtain a well-trained second semantic segmentation model. Specifically, the label for each of the second sample images is obtained in the following way: Based on the second sample image, the category activation map of the second sample image corresponding to each first candidate object category is obtained by using the trained first semantic segmentation model; wherein, the first semantic segmentation model is trained using the method described in any one of claims 1 to 7; the first candidate object category is the object category that can be identified in the foreground; The label of the second sample image is obtained based on the category activation map corresponding to each first candidate object category of the second sample image.

9. A training device for a semantic segmentation model, characterized in that, include: The first sample acquisition module is used to acquire multiple first sample images with labels, wherein the label of each first sample image includes the text label of each first object category included in the foreground of the first sample image; The category activation map generation module is used to obtain the category activation map of each first object category in each first sample image through a first neural network model for each first sample image; The category representation map generation module is used to obtain a category representation map of each first object category in the first sample image for each first sample image based on the category activation map of each first object category in the first sample image; The first similarity calculation module is used to extract the text features of the text labels of each first object category and the image features of the category representation graph, and determine the first similarity between the text features of the text labels of each first object category and the image features of the category representation graph. The training module is used to determine a first training loss based on the first similarity corresponding to each first object category in each of the first sample images, and to continuously train the first neural network model based on the total training loss to obtain a first semantic segmentation model; wherein the total training loss includes the first training loss.

10. A training device for a semantic segmentation model, characterized in that, include: The second sample acquisition module is used to acquire multiple labeled second sample images; The label of each second sample image is used to indicate the object category to which each pixel in the second sample image belongs; The model training module is used to continuously perform training operations on the second neural network model based on multiple second sample images to obtain a trained second semantic segmentation model. Specifically, the label for each of the second sample images is obtained in the following way: Based on the second sample image, the category activation map of the second sample image corresponding to each first candidate object category is obtained by using the trained first semantic segmentation model; wherein, the first semantic segmentation model is trained using the method described in any one of claims 1 to 7; the first candidate object category is the object category that can be identified in the foreground; The label of the second sample image is obtained based on the category activation map corresponding to each first candidate object category of the second sample image.

11. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method of any one of claims 1 to 7 or 8.

12. A computer-readable storage medium, characterized in that, The storage medium stores a computer program that, when executed by a processor, implements the method described in any one of claims 1 to 7 or 8.

13. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 7 or 8.

Images