A training method and related device for unsupervised semantic segmentation model

Abstract

The present application discloses a training method for an unsupervised semantic segmentation model, which is applied to the field of artificial intelligence technology. In this solution, the feature map of the image is clustered based on the category set to which the objects in the image belong, and the image region corresponding to each object in the image is obtained. By clustering the image features based on the category set to which the objects in the image belong, semantic information can be introduced in the clustering process, and the accuracy of clustering can be improved, thereby ensuring the segmentation accuracy of multiple image regions obtained after clustering. Then, the image region and the text of the category set are matched through a multimodal model to obtain the category corresponding to each image region, thereby introducing pseudo-annotations containing semantic information for the image. Finally, the semantic segmentation model is trained based on the pseudo-annotations containing semantic information, which enables the semantic segmentation model to learn semantic information during the training process and improves the accuracy of the trained semantic segmentation model.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a training method and related devices for an unsupervised semantic segmentation model. Background Art

[0002] Semantic segmentation is a recognition task in computer vision with a wide range of applications, such as autonomous driving, augmented reality, and smart healthcare. By inputting an image into a semantic segmentation model, the model labels each pixel in the image with a semantic category, thereby segmenting the image into different objects. Existing semantic segmentation models often rely on large numbers of annotated images during training. However, acquiring large amounts of annotated images is extremely time-consuming and expensive. Therefore, it is crucial to train semantic segmentation models by extracting information from unlabeled image data with minimal annotation effort.

[0003] To reduce the cost of annotation, unsupervised semantic segmentation model training methods have emerged. These methods leverage various forms of prior knowledge to generate pseudo-annotations for unlabeled image data and then train the semantic segmentation model.

[0004] In related technologies, due to the lack of annotation information in images, unsupervised semantic segmentation model training methods mainly rely on mining the similarity of low-level visual information in images, such as color, contour, and local context, to generate pseudo-annotations for training. However, the pseudo-annotations generated based on low-level visual information lack high-level semantic information, resulting in poor semantic segmentation accuracy of semantic segmentation models trained with pseudo-annotations. Summary of the Invention

[0005] The present application provides a training method for an unsupervised semantic segmentation model, which enables the semantic segmentation model to learn semantic information during the training process and improve the accuracy of the trained semantic segmentation model.

[0006] A first aspect of the present application provides a training method for an unsupervised semantic segmentation model, comprising: obtaining a feature map of a first image in a training data set, and clustering the feature map according to a category set of the first image to obtain multiple feature regions in the feature map, wherein the category set is used to indicate the categories of multiple objects in the first image, and each feature region of the multiple feature regions corresponds to a category in the category set.

[0007] Then, multiple image areas corresponding to the multiple feature areas in the first image are determined, and the multiple image areas and multiple texts describing the category set are input into a multimodal model to obtain an output result, which is used to indicate the category corresponding to each image area in the multiple image areas, wherein the multimodal model is used to perform image and text matching.

[0008] Secondly, the first image is input into the semantic segmentation model to be trained to obtain a first semantic segmentation result.

[0009] Finally, the semantic segmentation model is trained according to a first loss function to update the semantic segmentation model, wherein the first loss function is constructed based on the output result and the first semantic segmentation result.

[0010] In this solution, the feature map of the image is clustered based on the category set to which the objects in the image belong, and the image regions corresponding to each object in the image are obtained. By clustering the image features based on the category set to which the objects in the image belong, semantic information can be introduced into the clustering process, improving the accuracy of clustering, thereby ensuring the segmentation accuracy of the multiple image regions obtained after clustering. Then, the image regions and the text of the category set are matched through a multimodal model to obtain the category corresponding to each image region, thereby introducing pseudo-annotations containing semantic information for the image. Finally, the semantic segmentation model is trained based on the pseudo-annotations containing semantic information, which enables the semantic segmentation model to learn semantic information during the training process, thereby improving the accuracy of the trained semantic segmentation model.

[0011] In one possible implementation, the multimodal model is used to extract multiple image features corresponding to the multiple image regions and multiple text features corresponding to the multiple texts, and calculate the similarity between each of the multiple image features and the multiple text features to obtain a category corresponding to each of the multiple image regions;

[0012] Among them, the category corresponding to the first image area is the category corresponding to the text feature with the highest similarity to the first image feature, the first image area is any one of the multiple image areas, and the first image feature is the image feature corresponding to the first image area among the multiple image features.

[0013] In one possible implementation, the training dataset includes multiple images and multiple category labels, the multiple images including the first image, and the multiple category labels are used to indicate the categories of objects in the multiple images. The method further includes: inputting the first image and the multiple category labels into the multimodal model to obtain multiple similarities between features of the first image and features of each of the multiple category labels; and determining a category set for the first image based on the multiple similarities, wherein the category set includes multiple target categories, and the similarity between the features of the multiple target categories and the features of the first image is greater than or equal to a preset threshold.

[0014] In this solution, the first image and multiple category labels of the training data set are processed based on the multimodal model, which can determine the category set of the first image, so that subsequent clustering can be performed based on the category set to which the objects in the image belong, thereby introducing semantic information in the clustering process and improving the accuracy of clustering.

[0015] In one possible implementation, the method further includes: based on the first semantic segmentation result, obtaining a first mask image and a second mask image, wherein the first mask image only includes the image of the target object indicated in the first semantic segmentation result, and the second mask image only includes the image other than the target object in the first semantic segmentation result, and the target object is an object of any category in the first semantic segmentation result; inputting the first mask image, the second mask image, and the category text corresponding to the target object into the multimodal model to obtain a first similarity between the features of the first mask image and the features of the category text, and a second similarity between the features of the first mask image and the features of the second mask image; and training the semantic segmentation model according to the first loss function and the second loss function, wherein the second loss function is obtained based on the first similarity and the second similarity.

[0016] This solution introduces a second loss function that characterizes the differences between the target object and the categorized text, as well as the differences between the target object and other objects, in the semantic segmentation results obtained by the semantic segmentation model. Further training the semantic segmentation model based on the second loss function, building on the first loss function, can guide the semantic segmentation model to learn more precise region segmentation, thereby improving the segmentation accuracy of the semantic segmentation model.

[0017] In a possible implementation, the first similarity is negatively correlated with the second loss function, and the second similarity is positively correlated with the second loss function.

[0018] In one possible implementation, the method further includes: inputting the first image into a denoising model to obtain a second semantic segmentation result, the structure of the denoising model is the same as the structure of the semantic segmentation model, and the network parameters of the denoising model are different from the network parameters of the semantic segmentation model; training the semantic segmentation model according to the first loss function includes: training the semantic segmentation model according to the first loss function and a third loss function, the third loss function being obtained based on the first semantic segmentation result and the second semantic segmentation result.

[0019] In this solution, by introducing a denoising model with the same structure as the semantic segmentation model during the training process of the semantic segmentation model, the training oscillation that occurs during the training of the semantic segmentation model can be effectively suppressed, and the training stability of the semantic segmentation model can be improved, so as to efficiently realize the training of the semantic segmentation model.

[0020] In one possible implementation, the method further includes: inputting the first image into a denoising model to obtain a second semantic segmentation result, the structure of the denoising model is the same as the structure of the semantic segmentation model, and the network parameters of the denoising model are different from the network parameters of the semantic segmentation model; training the semantic segmentation model according to the first loss function and the second loss function includes: training the semantic segmentation model according to the first loss function, the second loss function and the third loss function, the third loss function being obtained based on the first semantic segmentation result and the second semantic segmentation result.

[0021] In a possible implementation, the network parameters of the denoising model are obtained based on the network parameters of the semantic segmentation model and the network parameters of the denoising model in the previous round of iterative training.

[0022] The second aspect of the present application provides a semantic segmentation method, including: obtaining an image to be processed; inputting the image to be processed into a semantic segmentation model to obtain a first semantic segmentation result; wherein the semantic segmentation model is trained based on a first loss function, the first loss function is obtained based on an output result and a second semantic segmentation result, the second semantic segmentation result is obtained by inputting a first image in a training data set into the semantic segmentation result model, the output result is obtained by inputting multiple image regions and multiple texts describing a category set into a multimodal model, the output result is used to indicate the category corresponding to each of the multiple image regions, the multimodal model is used to perform image and text matching, the multiple image regions are image regions in the first image corresponding to multiple feature regions, the multiple feature regions are obtained by clustering the feature map of the first image according to the category set of the first image, the category set is used to indicate the categories of multiple objects in the first image, and each feature region in the multiple feature regions corresponds to a category in the category set.

[0023] In one possible implementation, the multimodal model is used to extract multiple image features corresponding to the multiple image regions and multiple text features corresponding to the multiple texts, and calculate the similarity between each of the multiple image features and the multiple text features to obtain the category corresponding to each of the multiple image regions; wherein the category corresponding to the first image region is the category corresponding to the text feature with the highest similarity to the first image feature, the first image region is any one of the multiple image regions, and the first image feature is the image feature among the multiple image features corresponding to the first image region.

[0024] In one possible implementation, the training data set includes multiple images and multiple category labels, the multiple images include the first image, and the multiple category labels are used to indicate the categories of objects in the multiple images; the category set of the first image is obtained based on multiple similarities, the category set includes multiple target categories, and the similarities between the features of the multiple target categories and the features of the first image are all greater than or equal to a preset threshold, the multiple similarities are multiple similarities between the features of the first image and the features of each category label in the multiple category labels, and the multiple similarities are obtained after the first image and the multiple category labels are input into the multimodal model.

[0025] In one possible implementation, the semantic segmentation model is trained based on the first loss function and the second loss function, the second loss function is obtained based on the first similarity and the second similarity, the first similarity is the similarity between the features of the first mask image and the features of the category text, the second similarity is the similarity between the features of the first mask image and the features of the second mask image, the first similarity and the second similarity are obtained by inputting the first mask image, the second mask image and the category text corresponding to the target object into the multimodal model, the first mask image and the second mask image are obtained based on the second semantic segmentation result, the first mask image only includes the image of the target object indicated in the second semantic segmentation result, the second mask image only includes the image other than the target object in the second semantic segmentation result, and the target object is an object of any category in the first semantic segmentation result.

[0026] In a possible implementation, the first similarity is negatively correlated with the second loss function, and the second similarity is positively correlated with the second loss function.

[0027] In one possible implementation, the semantic segmentation model is trained based on the first loss function and the third loss function, the third loss function is obtained based on the second semantic segmentation result and the third semantic segmentation result, the third semantic segmentation result is obtained after inputting the first image into the denoising model, the structure of the denoising model is the same as that of the semantic segmentation model, and the network parameters of the denoising model are different from the network parameters of the semantic segmentation model.

[0028] In one possible implementation, the semantic segmentation model is trained based on the first loss function, the second loss function and the third loss function, the third loss function is obtained based on the second semantic segmentation result and the third semantic segmentation result, the third semantic segmentation result is obtained after inputting the first image into the denoising model, the structure of the denoising model is the same as that of the semantic segmentation model, and the network parameters of the denoising model are different from the network parameters of the semantic segmentation model.

[0029] In a possible implementation, the network parameters of the denoising model are obtained based on the network parameters of the semantic segmentation model and the network parameters of the denoising model in the previous round of iterative training.

[0030] In a third aspect, the present application provides a training device for an unsupervised semantic segmentation model, comprising: an acquisition module for acquiring a feature map of a first image in a training data set, and clustering the feature map according to a category set of the first image to obtain multiple feature regions in the feature map, wherein the category set is used to indicate the categories of multiple objects in the first image, and each feature region of the multiple feature regions corresponds to a category in the category set; a processing module for determining multiple image regions in the first image corresponding to the multiple feature regions; the processing module is further used to input the multiple image regions and multiple texts describing the category set into a multimodal model to obtain an output result, wherein the output result is used to indicate the category corresponding to each image region in the multiple image regions, wherein the multimodal model is used to perform image and text matching; the processing module is further used to input the first image into a semantic segmentation model to obtain a first semantic segmentation result; a training module for training the semantic segmentation model according to a first loss function to update the semantic segmentation model, wherein the first loss function is constructed based on the output result and the first semantic segmentation result.

[0031] In one possible implementation, the multimodal model is used to extract multiple image features corresponding to the multiple image regions and multiple text features corresponding to the multiple texts, and calculate the similarity between each of the multiple image features and the multiple text features to obtain the category corresponding to each of the multiple image regions; wherein the category corresponding to the first image region is the category corresponding to the text feature with the highest similarity to the first image feature, the first image region is any one of the multiple image regions, and the first image feature is the image feature among the multiple image features corresponding to the first image region.

[0032] In one possible implementation, the training data set includes multiple images and multiple category labels, the multiple images include the first image, and the multiple category labels are used to indicate the categories of objects in the multiple images; the processing module is further used to: input the first image and the multiple category labels into the multimodal model to obtain multiple similarities between the features of the first image and the features of each category label in the multiple category labels; determine the category set of the first image based on the multiple similarities, wherein the category set includes multiple target categories, and the similarities between the features of the multiple target categories and the features of the first image are all greater than or equal to a preset threshold.

[0033] In one possible implementation, the processing module is further used to obtain a first mask image and a second mask image based on the first semantic segmentation result, wherein the first mask image only includes the image of the target object indicated in the first semantic segmentation result, and the second mask image only includes the image other than the target object in the first semantic segmentation result, and the target object is an object of any category in the first semantic segmentation result; the processing module is further used to input the first mask image, the second mask image and the category text corresponding to the target object into the multimodal model to obtain a first similarity between the features of the first mask image and the features of the category text, and a second similarity between the features of the first mask image and the features of the second mask image; the training module is specifically used to train the semantic segmentation model according to the first loss function and the second loss function, wherein the second loss function is obtained based on the first similarity and the second similarity.

[0034] In a possible implementation, the first similarity is negatively correlated with the second loss function, and the second similarity is positively correlated with the second loss function.

[0035] In one possible implementation, the processing module is also used to input the first image into a denoising model to obtain a second semantic segmentation result. The structure of the denoising model is the same as that of the semantic segmentation model, and the network parameters of the denoising model are different from the network parameters of the semantic segmentation model. The training module is specifically used to train the semantic segmentation model according to the first loss function and the third loss function. The third loss function is obtained based on the first semantic segmentation result and the second semantic segmentation result.

[0036] In one possible implementation, the processing module is also used to input the first image into a denoising model to obtain a second semantic segmentation result. The structure of the denoising model is the same as that of the semantic segmentation model, and the network parameters of the denoising model are different from the network parameters of the semantic segmentation model. The training module is specifically used to train the semantic segmentation model according to the first loss function, the second loss function and the third loss function, and the third loss function is obtained based on the first semantic segmentation result and the second semantic segmentation result.

[0037] In a possible implementation, the network parameters of the denoising model are obtained based on the network parameters of the semantic segmentation model and the network parameters of the denoising model in the previous round of iterative training.

[0038] The fourth aspect of the present application provides a semantic segmentation device, including: an acquisition module for acquiring an image to be processed; a processing module for inputting the image to be processed into a semantic segmentation model to obtain a first semantic segmentation result; wherein the semantic segmentation model is trained based on a first loss function, the first loss function is obtained based on an output result and a second semantic segmentation result, the second semantic segmentation result is obtained by inputting a first image in a training data set into the semantic segmentation result model, the output result is obtained by inputting multiple image regions and multiple texts describing a category set into a multimodal model, the output result is used to indicate the category corresponding to each of the multiple image regions, the multimodal model is used to perform image and text matching, the multiple image regions are image regions in the first image corresponding to multiple feature regions, the multiple feature regions are obtained by clustering the feature map of the first image according to the category set of the first image, the category set is used to indicate the categories of multiple objects in the first image, and each feature region in the multiple feature regions corresponds to a category in the category set.

[0039] In one possible implementation, the multimodal model is used to extract multiple image features corresponding to the multiple image regions and multiple text features corresponding to the multiple texts, and calculate the similarity between each of the multiple image features and the multiple text features to obtain the category corresponding to each of the multiple image regions; wherein the category corresponding to the first image region is the category corresponding to the text feature with the highest similarity to the first image feature, the first image region is any one of the multiple image regions, and the first image feature is the image feature among the multiple image features corresponding to the first image region.

[0040] In one possible implementation, the training data set includes multiple images and multiple category labels, the multiple images include the first image, and the multiple category labels are used to indicate the categories of objects in the multiple images; the category set of the first image is obtained based on multiple similarities, the category set includes multiple target categories, and the similarities between the features of the multiple target categories and the features of the first image are all greater than or equal to a preset threshold, the multiple similarities are multiple similarities between the features of the first image and the features of each category label in the multiple category labels, and the multiple similarities are obtained after the first image and the multiple category labels are input into the multimodal model.

[0041] In one possible implementation, the semantic segmentation model is trained based on the first loss function and the second loss function, the second loss function is obtained based on the first similarity and the second similarity, the first similarity is the similarity between the features of the first mask image and the features of the category text, the second similarity is the similarity between the features of the first mask image and the features of the second mask image, the first similarity and the second similarity are obtained by inputting the first mask image, the second mask image and the category text corresponding to the target object into the multimodal model, the first mask image and the second mask image are obtained based on the second semantic segmentation result, the first mask image only includes the image of the target object indicated in the second semantic segmentation result, the second mask image only includes the image other than the target object in the second semantic segmentation result, and the target object is an object of any category in the first semantic segmentation result.

[0042] In a possible implementation, the first similarity is negatively correlated with the second loss function, and the second similarity is positively correlated with the second loss function.

[0043] In one possible implementation, the semantic segmentation model is trained based on the first loss function and the third loss function, the third loss function is obtained based on the second semantic segmentation result and the third semantic segmentation result, the third semantic segmentation result is obtained after inputting the first image into the denoising model, the structure of the denoising model is the same as that of the semantic segmentation model, and the network parameters of the denoising model are different from the network parameters of the semantic segmentation model.

[0044] In one possible implementation, the semantic segmentation model is trained based on the first loss function, the second loss function and the third loss function, the third loss function is obtained based on the second semantic segmentation result and the third semantic segmentation result, the third semantic segmentation result is obtained after inputting the first image into the denoising model, the structure of the denoising model is the same as that of the semantic segmentation model, and the network parameters of the denoising model are different from the network parameters of the semantic segmentation model.

[0045] In a possible implementation, the network parameters of the denoising model are obtained based on the network parameters of the semantic segmentation model and the network parameters of the denoising model in the previous round of iterative training.

[0046] In a fifth aspect, the present application provides a training device, which may include a processor coupled to a memory, wherein the memory stores program instructions, and when the program instructions stored in the memory are executed by the processor, the method described in the first aspect above is implemented.

[0047] In a sixth aspect, the present application provides an execution device, which may include a processor, the processor and a memory coupled, the memory storing program instructions, and the method described in the second aspect above is implemented when the program instructions stored in the memory are executed by the processor.

[0048] In a seventh aspect, the present application provides a computer program product, which, when executed on a computer, enables the computer to execute the method described in the first or second aspect above.

[0049] In an eighth aspect, the present application provides a computer-readable storage medium, in which a computer program is stored. When the program is run on a computer, the computer executes the method described in the first or second aspect above.

[0050] In a ninth aspect, the present application provides a circuit system, which includes a processing circuit, and the processing circuit is configured to execute the method described in the first or second aspect above.

[0051] In a tenth aspect, the present application provides a chip system, which includes a processor for implementing the functions involved in the above-mentioned various aspects, for example, sending or processing the data and / or information involved in the above-mentioned method. In one possible design, the chip system also includes a memory, which is used to store program instructions and data necessary for the server or communication device. The chip system can be composed of a chip or can include a chip and other discrete devices.

[0052] For the specific implementation steps of various possible implementation methods of the fifth to tenth aspects of the embodiments of the present application, as well as the beneficial effects brought about by each possible implementation method, you can refer to the description of the various possible implementation methods in the first to second aspects, and will not be repeated here one by one. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 A structural diagram of the main framework of artificial intelligence;

[0054] Figure 2 A schematic diagram of images and annotations required for the training process of a semantic segmentation model in related technologies;

[0055] Figure 3 A schematic diagram showing a time-consuming comparison of image annotation provided in an embodiment of the present application;

[0056] Figure 4 A flowchart of a method for training an unsupervised semantic segmentation model provided in an embodiment of the present application;

[0057] Figure 5A schematic diagram of a mask image provided in an embodiment of the present application;

[0058] Figure 6 A schematic diagram of a training framework provided in an embodiment of the present application;

[0059] Figure 7 A schematic diagram of the training process of an unsupervised semantic segmentation model provided in an embodiment of the present application;

[0060] Figure 8 A schematic diagram of a workflow of a semantic model provided in an embodiment of the present application;

[0061] Figure 9 A schematic diagram of a mask model workflow provided in an embodiment of the present application;

[0062] Figure 10 A schematic diagram of an open set-based semantic segmentation model training method provided in an embodiment of the present application;

[0063] Figure 11 A schematic diagram of optimization of a semantic segmentation model during training provided in an embodiment of the present application;

[0064] Figure 12 A schematic diagram of the semantic segmentation effect of a semantic segmentation model provided in an embodiment of the present application in different scenarios;

[0065] Figure 13 A schematic diagram of the structure of a training device for an unsupervised semantic segmentation model provided in an embodiment of the present application;

[0066] Figure 14 A schematic diagram of the structure of a semantic segmentation device provided in an embodiment of the present application;

[0067] Figure 15 A schematic diagram of the structure of an execution device provided in an embodiment of the present application;

[0068] Figure 16 A schematic diagram of the structure of the training device provided in an embodiment of the present application;

[0069] Figure 17 A schematic diagram of the structure of the chip provided in an embodiment of the present application. DETAILED DESCRIPTION

[0070] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, and this is merely a way of distinguishing the objects of the same attributes when describing them in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0071] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0072] First, the overall workflow of the artificial intelligence system is described. Figure 1 , Figure 1 This is a structural diagram of the main framework of artificial intelligence. The following explains this framework from two dimensions: the "intelligent information chain" (horizontal axis) and the "IT value chain" (vertical axis). The "intelligent information chain" reflects the entire process from data acquisition to processing. For example, it could be the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. In this process, data undergoes a condensed process of "data-information-knowledge-wisdom." The "IT value chain" reflects the value that artificial intelligence brings to the information technology industry, from the underlying infrastructure of human intelligence, information (provided and processed by technology), to the system's industrial ecosystem.

[0073] (1) Infrastructure

[0074] The infrastructure provides computing power support for artificial intelligence systems, enabling communication with the outside world and providing support through the basic platform. Communication with the outside world is achieved through sensors; computing power is provided by intelligent chips, which can specifically adopt hardware acceleration chips such as central processing units (CPUs), embedded neural network processing units (NPUs), graphics processing units (GPUs), application specific integrated circuits (ASICs), or field programmable gate arrays (FPGAs); the basic platform includes related platform guarantees and support such as distributed computing frameworks and networks, and can include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to obtain data, and this data is provided to the intelligent chips in the distributed computing system provided by the basic platform for calculation.

[0075] (2) Data

[0076] Data above the infrastructure layer represents data sources for AI. This data includes graphics, images, voice, and text, as well as IoT data from traditional devices. This includes business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.

[0077] (3) Data processing

[0078] Data processing generally includes data training, machine learning, deep learning, search, reasoning, decision-making, etc.

[0079] Among them, machine learning and deep learning can symbolize and formalize data for intelligent information modeling, extraction, preprocessing, and training.

[0080] Reasoning refers to the process of simulating human intelligent reasoning in computers or intelligent systems, using formalized information to perform machine thinking and solve problems based on reasoning control strategies. Typical functions are search and matching.

[0081] Decision-making refers to the process of making decisions after intelligent information is reasoned, and usually provides functions such as classification, sorting, and prediction.

[0082] (4) General ability

[0083] After the data has undergone the data processing mentioned above, some general capabilities can be further formed based on the results of the data processing, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0084] (5) Smart products and industry applications

[0085] Smart products and industry applications refer to the products and applications of artificial intelligence systems in various fields. They are the encapsulation of the overall artificial intelligence solution, which productizes intelligent information decision-making and realizes practical application. Its application areas mainly include: smart terminals, smart manufacturing, smart transportation, smart homes, smart medical care, smart security, autonomous driving, smart cities, etc.

[0086] Currently, semantic segmentation is a research hotspot in the field of artificial intelligence and has a wide range of applications. To obtain a semantic segmentation model with high segmentation accuracy, related technologies typically require training the semantic segmentation model using a large number of manually annotated images.

[0087] See Figure 2 , Figure 2 Schematic diagram of images and annotations required for the training process of semantic segmentation models in related technologies. Figure 2 As mentioned above, obtaining accurate training data typically requires manual pixel-level annotation of images, meaning each pixel in the image is manually labeled with its category. Furthermore, because semantic segmentation models require a large number of annotated images as training data during training, obtaining large-scale pixel-level annotated images as training data often consumes significant time and financial costs. In this context, it is crucial to learn how to train semantic segmentation models by extracting information from unlabeled image data with minimal annotation effort.

[0088] A related art provides a training method for an unsupervised semantic segmentation model, and the specific steps are as follows.

[0089] Step 1: First, a saliency detection model is pre-trained based on a pre-prepared image dataset and its saliency mask annotations. This saliency detection model can output a mask of the target region in an input image, but does not distinguish the category of the target region.

[0090] Step 2: For an image used to train a semantic segmentation model, extract the image mask based on the pre-trained saliency detection model to obtain an image mask. In the image mask, pixels with a value of 1 represent pixels of the target object, and pixels with a value of 0 represent pixels of non-target objects.

[0091] In step 3, the image mask generated in step 2 is used as a pseudo-annotation to train the semantic segmentation model so that the semantic segmentation model learns the pixel features of the input image during the training process.

[0092] Step 4: Extract pixel features from the segmented image output by the semantic segmentation model and cluster these pixel features. After clustering, pixels belonging to the same cluster are assigned the same semantic class label.

[0093] Step 5: In order to make a one-to-one correspondence between clustered pixels and semantic labels, we need to use an image set with semantic segmentation annotations (hereinafter referred to as the annotated image set). Specifically, first cluster the images in the image set with semantic segmentation annotations, and calculate the similarity between the clustering features of the segmented images and the clustering features of the images in the annotated image set, so as to associate the clustering features with high similarity. Since the images in the annotated image set have pixel-level semantic labels, the associated clustering features can all correspond to a semantic category, thereby obtaining the annotation results of the segmented images output by the semantic segmentation model. In this way, based on the above steps 1-5, the semantic segmentation results of the image can be predicted in an unsupervised manner.

[0094] Due to the lack of annotation information in the image, related technologies cluster image pixels based on the similarity of underlying visual information (such as color, contour, local context, etc.), and then use auxiliary annotation sets to predict the semantic categories of the clustered image pixels.

[0095] However, the underlying visual information has certain limitations. The areas obtained by similarity clustering based on the underlying visual information are difficult to accurately cover the target areas corresponding to the various objects in the image. Therefore, the semantic segmentation effect of the related technology is poor. In addition, due to the gap between the underlying visual information and the high-level semantic information, training the semantic segmentation model based on the pseudo-annotations generated by the underlying visual information will hinder the semantic segmentation model's learning of semantic information, resulting in the semantic segmentation model being unable to perform semantic segmentation from the perspective of semantic information, further affecting the semantic segmentation effect. Moreover, since the related technology relies on a labeled data set to generate semantic labels for the clustering results generated by the semantic segmentation model, the semantic segmentation model cannot directly and effectively learn the semantic segmentation information; and the introduced labeled data set and the training set in the training phase include different image data, which easily introduces new errors into the semantic segmentation results.

[0096] In general, the semantic segmentation effect of the semantic segmentation model obtained by the unsupervised semantic segmentation model training method in related technologies is poor and it is difficult to meet the current semantic segmentation requirements.

[0097] Based on this, an embodiment of the present application provides a training method for an unsupervised semantic segmentation model, which clusters the feature graph of the image based on the category set to which the objects in the image belong, and obtains the image region corresponding to each object in the image. By clustering the image features based on the category set to which the objects in the image belong, semantic information can be introduced in the clustering process, and the accuracy of clustering can be improved, thereby ensuring the segmentation accuracy of multiple image regions obtained after clustering. Then, the image region and the text of the category set are matched through a multimodal model to obtain the category corresponding to each image region, thereby introducing pseudo-annotations containing semantic information for the image. Finally, the semantic segmentation model is trained based on the pseudo-annotations containing semantic information, which enables the semantic segmentation model to learn semantic information during the training process, thereby improving the accuracy of the trained semantic segmentation model.

[0098] To facilitate understanding, the technical terms involved in the embodiments of the present application are explained below.

[0099] (1) Unsupervised learning

[0100] Unsupervised learning is a machine learning method that learns from unlabeled data, discovering commonalities in the data and automatically classifying or clustering the data. Rather than responding to feedback, unsupervised learning identifies and reacts to commonalities in each new piece of data based on whether certain commonalities exist.

[0101] (2) Cross-modal model

[0102] A multimodal model is one that can simultaneously process data from multiple modalities (e.g., images, text, and speech) and leverage these modalities and the interactions between them to perform processing tasks. For example, by learning from large amounts of data, a multimodal model can discover semantic correspondences between data from different modalities, such as connecting the word "dog" in text with the appearance of a "dog" in an image.

[0103] (3) Knowledge Distillation

[0104] Knowledge distillation is a method used in transfer learning. It uses the output of a pre-trained model as a supervisory signal to train another network. Simply put, knowledge distillation uses a pre-trained network as the teacher network and the network to be trained as the student network. Under the guidance of the teacher network, the student network is trained to achieve the same or similar data processing capabilities as the teacher network. Typically, the teacher network can be a complex model, while the student network can be a simpler one. Through knowledge distillation, the feature representation knowledge learned by the complex network can be transferred to the simpler model.

[0105] (4) Pre-trained model

[0106] A pre-trained model is a saved network that has been trained on a large dataset.

[0107] (5) Contrastive Language–Image Pre-training (CLIP) Model

[0108] The CLIP model was trained using a large number of pre-collected image-text pairs (approximately 400 million pairs), resulting in a very powerful image-text pairing capability. The CLIP model works by encoding the image and text using an image encoder and a text encoder, respectively, and calculating the similarity between the image and text encodings to determine the text corresponding to the image.

[0109] The following describes the application scenarios of the training method of the graph unsupervised semantic segmentation model provided in the embodiments of the present application.

[0110] The unsupervised semantic segmentation model training method provided in the embodiments of this application can be applied to the field of image processing in the field of artificial intelligence. Specifically, the unsupervised semantic segmentation model training method of the embodiments of this application can be applied to application scenarios such as image annotation, smart home, road traffic, and smart security. The following examples illustrate specific application scenarios of the embodiments of this application.

[0111] Application scenario 1: image annotation.

[0112] Based on the training method of the unsupervised semantic segmentation model provided in the embodiment of the present application, a semantic segmentation model with higher accuracy can be trained, thereby building an interactive automatic annotation platform to assist manual image annotation, thereby improving image annotation efficiency.

[0113] In the traditional image annotation process, it is necessary to manually mark the corner points around the target object in the image, that is, the turning points on the outer bounding polygon of the target object, thereby forming a closed irregular shape. The pixels within the closed irregular shape are the pixels of the target object. The semantic segmentation model obtained by the training method of the unsupervised semantic segmentation model in the embodiment of the application can replace manual object recognition and segmentation in the image, thereby realizing image annotation.

[0114] For example, see Figure 3 , Figure 3 This is a time-consuming comparison diagram of an image annotation provided in an embodiment of the present application. Figure 3 The semantic segmentation model obtained by the training method of the unsupervised semantic segmentation model in the embodiment of the present application performs unsupervised semantic segmentation on the image, obtains the annotation result of the aircraft in the image, and the time consuming of the unsupervised semantic segmentation process is 0.13 seconds. In addition, after obtaining the annotation result of the aircraft in the image, the annotation result of the aircraft is manually corrected in the details, which takes 2 seconds. However, in the traditional image annotation process, it takes about 70 seconds to manually annotate the aircraft in the image. Obviously, compared to the traditional image annotation process, the semantic segmentation model obtained by the training method of the unsupervised semantic segmentation model in the embodiment of the present application can replace the manual implementation of image annotation, effectively improving the efficiency of image annotation.

[0115] Application scenario two: smart home.

[0116] In the smart home sector, various smart furniture often need to identify different objects in the home during operation. For example, during the operation of a sweeping robot, the sweeping robot needs to identify different objects in the home to perform operations such as cleaning and obstacle avoidance. By deploying a semantic segmentation model obtained by the training method based on the unsupervised semantic segmentation model in the embodiments of this application in the sweeping robot, the sweeping robot can easily identify various objects that may appear in the home, thereby effectively performing cleaning and obstacle avoidance operations.

[0117] Application scenario two: road traffic.

[0118] In the field of road traffic, autonomous vehicles or urban traffic cameras often collect images on the road and perform recognition of pedestrians, vehicles, and other objects in the collected images to facilitate the execution of subsequent tasks. By deploying a semantic segmentation model obtained by the training method of the unsupervised semantic segmentation model in the embodiment of the present application in an autonomous vehicle or urban traffic camera, the autonomous vehicle or urban traffic camera can identify pedestrians or vehicles that may appear on the road, thereby effectively performing autonomous driving or road traffic planning.

[0119] The unsupervised semantic segmentation model training method provided in the embodiments of the present application can be applied to electronic devices. For example, the electronic devices may be servers, smartphones, personal computers, laptops, tablet computers, smart TVs, mobile internet devices (MIDs), wearable devices, virtual reality (VR) devices, augmented reality (AR) devices, wireless terminals in smart cities, wireless terminals in smart homes, etc.

[0120] The above describes the scenarios and devices used in the training method of the unsupervised semantic segmentation model provided by the embodiment of the present application. The following details the execution process of the training method of the unsupervised semantic segmentation model provided by the embodiment of the present application. Figure 4 , Figure 4 A flowchart of a method for training an unsupervised semantic segmentation model provided in an embodiment of the present application. Figure 4 As shown, the training method of the unsupervised semantic segmentation model is applied to a training device, and the method includes the following steps 401-405.

[0121] Step 401: Obtain a feature map of a first image in a training data set, and cluster the feature map according to a category set of the first image to obtain multiple feature regions in the feature map, wherein the category set is used to indicate categories of multiple objects in the first image, and each feature region corresponds to a category in the category set.

[0122] In this embodiment, in the process of training the semantic segmentation model based on the first image in the training data set, the training device obtains the feature map of the first image through the feature extraction model. Among them, the feature extraction model can be, for example, an image processing model with a structure such as a convolutional neural network structure, a recurrent neural network structure, or a residual neural network structure, which can process the first image to obtain the feature map of the first image. Exemplarily, the feature extraction model can be, for example, a self-distillation with no labels (DINO) model. The DINO model is a model obtained by self-supervised learning training on the ImageNet dataset. The characteristic of the DINO model is that it can output high-resolution feature maps, which is particularly suitable for segmentation tasks.

[0123] After obtaining a feature map of the first image, the training device clusters the feature map according to the category set of the first image, i.e., classifying features in the feature map and grouping similar features into the same category, thereby obtaining multiple feature regions in the feature map. For example, the training device may cluster the feature map based on a K-means clustering algorithm.

[0124] In the multiple feature areas of the feature map, the features in the same feature area are similar to each other, and the features in different feature areas are different from each other. In addition, the category set of the first image indicates the category of each object in the first image. For example, the category set of the first image may be {bird, branch, grass}, which indicates that the categories of objects in the first image include: bird, branch, grass. Since the multiple feature areas of the feature map are obtained by clustering according to the category set of the first image, each feature area in the multiple feature areas obtained after clustering corresponds to a category in the category set. Moreover, the number of categories of the multiple feature areas obtained after clustering is the same as the number of categories in the category set. For example, assuming that the category set includes four categories, the training device sets the feature map to be divided into feature areas of four categories, so that the feature map is clustered based on the clustering algorithm to obtain multiple feature areas corresponding to the four categories.

[0125] It should be noted that, for the multiple feature regions obtained after clustering, each feature region may include a continuous region in the feature map, or multiple discontinuous regions in the feature map. This embodiment does not limit the representation of the feature regions.

[0126] Optionally, the category set of the first image may be pre-specified manually, or may be obtained in other ways, such as by identifying the first image to obtain the category set of the first image.

[0127] Step 402: Determine a plurality of image regions corresponding to the plurality of feature regions in the first image.

[0128] In the embodiment of the present application, since the feature map of the first image is obtained by performing feature extraction on the first image, a conversion relationship exists between the first image and the feature map of the first image. Therefore, based on the conversion relationship between the first image and the feature map of the first image, it is possible to determine the image region in the first image to which each feature region in the feature map of the first image corresponds. In this way, based on the multiple feature regions in the feature map, the training device can determine multiple image regions in the first image that correspond one-to-one to the multiple feature regions.

[0129] Step 403: Input the multiple image regions and the multiple texts describing the category set into a multimodal model to obtain an output result, wherein the output result is used to indicate the category corresponding to each image region in the multiple image regions, wherein the multimodal model is used to perform image and text matching.

[0130] In this embodiment, the multimodal model inputs are multiple image regions and multiple texts describing a set of categories for a first image. During operation, the multimodal model matches each image region with each text, thereby determining the text that best matches each image region. Furthermore, the category corresponding to each image region is determined to be the category described by the text that best matches it. For example, the multimodal model may be a CLIP model.

[0131] For example, assume that the multiple image regions in the first image are image region 1, image region 2, and image region 3, and the multiple texts describing the category set of the first image are text 1, text 2, and text 3. Then, the multimodal model calculates the degree of matching between image region 1 and the three texts, the degree of matching between image region 2 and the three texts, and the degree of matching between image region 3 and the three texts, and ultimately obtains the text that best matches the three image regions. For example, if the text that best matches image region 1 is text 2, the category corresponding to image region 1 is the category described by text 2; similarly, if the text that best matches image region 2 is text 3, the category corresponding to image region 2 is the category described by text 3.

[0132] Specifically, the multimodal model is used to extract multiple image features corresponding to the multiple image regions and multiple text features corresponding to the multiple texts, and calculate the similarity between each of the multiple image features and the multiple text features to obtain a category corresponding to each of the multiple image regions. The category corresponding to the first image region is the category corresponding to the text feature with the highest similarity to the first image feature, the first image region is any one of the multiple image regions, and the first image feature is the image feature corresponding to the first image region among the multiple image features.

[0133] In other words, after inputting multiple image regions and multiple texts into the multimodal model, the model extracts the image features corresponding to the multiple image regions and the text features corresponding to the multiple texts. It then calculates the similarity between each image feature and each text feature to determine the similarity between the image regions and the texts. The higher the similarity between the image region and the text, the closer the match between the image region and the text. For a particular image region, the text that best matches the image region can be determined by selecting the text with the highest similarity to the image region.

[0134] Optionally, the multiple texts describing the category set of the first image may be, for example, "a photo of a bird," "a photo of a branch," or "a photo of a leaf." That is, the text describing the category set of the first image may be composed of keywords (bird, branch, or leaf) and other words related to the keywords. The purpose of describing other words is mainly to make the text a complete sentence, which helps the multimodal model obtain features corresponding to the keywords.

[0135] Step 404: input the first image into a semantic segmentation model to obtain a first semantic segmentation result.

[0136] In this embodiment, the semantic segmentation model is a model to be trained, which can perform semantic segmentation on the input image to obtain the corresponding semantic segmentation results. For example, the semantic segmentation model can be a Deeplabv3 model based on a convolutional neural network structure, or a Segmenter model based on a Transformer structure. The embodiments of this application do not limit the specific structure of the semantic segmentation model.

[0137] Step 405: Training the semantic segmentation model according to a first loss function to update the semantic segmentation model, wherein the first loss function is constructed based on the output result and the first semantic segmentation result.

[0138] In this embodiment, since the output result of the multimodal model can indicate the category corresponding to each of the multiple image regions of the first image, and the first semantic segmentation result can also indicate the category corresponding to each image region divided by the semantic segmentation model in the first image. Therefore, the training device can construct a first loss function based on the output result of the multimodal model and the first semantic segmentation result obtained by the semantic segmentation model, and the first loss function indicates the difference between the output result and the first semantic segmentation result. The greater the difference between the output result and the first semantic segmentation result, the greater the value of the first loss function; the smaller the difference between the output result and the first semantic segmentation result, the smaller the value of the first loss function.

[0139] It is understood that steps 401-405 above describe the process of the training device training the semantic segmentation model based on the first image in the training dataset. In actual applications, the training dataset includes a large number of images, and the training device may perform multiple rounds of iterative training on the semantic segmentation model based on the training dataset until convergence conditions are reached, thereby obtaining a trained semantic segmentation model.

[0140] In this embodiment, the training device clusters the feature map of the first image based on the category set to which the objects in the first image belong, and obtains the image regions corresponding to the various objects in the first image. By clustering the image features based on the category set to which the objects in the image belong, semantic information can be introduced into the clustering process, thereby improving the accuracy of clustering and ensuring the segmentation accuracy of the multiple image regions obtained after clustering. In addition, the training device also matches the image region and the text of the category set through a multimodal model to obtain the category corresponding to each image region, thereby introducing pseudo-annotations containing semantic information into the image. Finally, training the semantic segmentation model based on the pseudo-annotations containing semantic information enables the semantic segmentation model to learn semantic information during the training process, thereby improving the accuracy of the trained semantic segmentation model.

[0141] For ease of understanding, the following describes a process of obtaining a category set of the first image based on the multimodal model.

[0142] Optionally, the training dataset for training the semantic segmentation model includes multiple images and multiple category labels. The multiple images include the first image, and the multiple category labels are used to indicate the categories of objects in the multiple images. In other words, the multiple category labels can indicate the categories that all images in the training dataset may include. For any image in the training dataset, the categories of all objects in the image can be found from the multiple category labels. For example, assuming that the training dataset is a set of images related to a farm, the multiple category labels in the training dataset may include, for example, category labels such as chicken, duck, cow, sheep, pig, dog, horse, and fence.

[0143] Before the above-mentioned step 401, the training device can input the first image and multiple category labels in the training set into the multimodal model to obtain multiple similarities between the features of the first image and the features of each category label in the multiple category labels. Then, the training device determines the category set of the first image based on the multiple similarities output by the multimodal model. The category set of the first image includes multiple target categories, and the similarities between the features of the multiple target categories and the features of the first image are all greater than or equal to a preset threshold. That is, after obtaining the similarities between the features of the first image and the features of each category label through the multimodal model, the training device selects the similarities that are greater than or equal to the preset threshold, and determines the category labels corresponding to these similarities as the target categories in the category set of the first image. In addition, the preset threshold can be set or adjusted according to the actual application and the selected similarity calculation method, which is not limited in this embodiment.

[0144] For example, assuming the preset threshold is 0.5, the training dataset corresponds to multiple category labels including chicken, duck, cow, sheep, pig, and dog, and the similarities between the features of the first image and the features of each of the multiple category labels are 0.7, 0.8, 0.2, 0.15, 0.3, and 0.35, respectively. The training device can then determine that the similarities greater than the preset threshold are 0.7 and 0.8, meaning that the category set corresponding to the first image is chicken and duck.

[0145] In this solution, the first image and multiple category labels of the training data set are processed based on the multimodal model, which can determine the category set of the first image, so that subsequent clustering can be performed based on the category set to which the objects in the image belong, thereby introducing semantic information in the clustering process and improving the accuracy of clustering.

[0146] The above introduces the process of training a semantic segmentation model based on the first loss function between the pseudo-annotations obtained by the multimodal model and the semantic segmentation results of the semantic segmentation model. In some possible implementations, the training device can also train the semantic segmentation model based on the first loss function and other loss functions to improve the semantic segmentation accuracy of the trained semantic segmentation model.

[0147] Implementation method 1: The training device trains the semantic segmentation model based on the first loss function and the second loss function.

[0148] First, the training device obtains a first mask image and a second mask image based on the first semantic segmentation result, wherein the first mask image only includes the image of the target object indicated in the first semantic segmentation result, and the second mask image only includes the image other than the target object in the first semantic segmentation result, and the target object is an object of any category in the first semantic segmentation result.

[0149] Since the first semantic segmentation result output by the semantic segmentation model marks the image area where each object in the first image is located and the category corresponding to each object, the training device can determine the first mask image and the second mask image based on the image area where the target object indicated in the first semantic segmentation result is located. The first mask image can, for example, be an image of the same size as the first image, and the first mask image only includes the image of the target object, and does not include images of other objects other than the target object. Conversely, the second mask image can also, for example, be an image of the same size as the first image, and the second mask image only includes images of other objects other than the target object, and does not include an image of the target object.

[0150] For example, see Figure 5 , Figure 5 This is a schematic diagram of a mask image provided in an embodiment of the present application. Figure 5 The objects in the first image include birds, branches, and tree trunks. The first semantic segmentation result divides the image into the areas where the various objects in the first image reside. Assuming the target object is a bird, the first mask image obtained based on the first semantic segmentation result only includes the image of the bird in the first image, and all areas in the first mask image except the area where the bird resides are represented by pixels of a specific value (i.e., black pixels). The second mask image only includes the images of the objects in the first image except the bird, and all areas in the first mask image where the bird resides are represented by pixels of a specific value.

[0151] Then, the training device inputs the first mask image, the second mask image, and the category text corresponding to the target object into the multimodal model to obtain a first similarity between the features of the first mask image and the features of the category text, and a second similarity between the features of the first mask image and the features of the second mask image.

[0152] Since the first mask image only includes the target object, a higher first similarity between the features of the first mask image and the features of the category text indicates that the content in the first mask image is closer to the category of the target object, i.e., the first mask image contains less content other than the target object. Since the second mask image only includes objects other than the target object, a lower second similarity between the features of the first mask image and the features of the second mask image indicates that the difference between the first and second mask images is greater, i.e., the second mask image contains less content related to the target object.

[0153] Finally, the training device calculates a second loss function based on the first similarity and the second similarity, and trains the semantic segmentation model based on the first loss function and the second loss function.

[0154] In this embodiment, a higher first similarity indicates less content other than the target object in the first mask image, which means the semantic segmentation model has higher segmentation accuracy. A lower second similarity indicates less content related to the target object in the second mask image, which means the semantic segmentation model has higher segmentation accuracy. Therefore, based on the content represented by the first and second similarities and the training objectives of the semantic segmentation model, a second loss function can be designed.

[0155] Optionally, the first similarity is negatively correlated with the second loss function, and the second similarity is positively correlated with the second loss function. That is, the first similarity and the second loss function have different directions of change in magnitude, the value of the second loss function decreases as the first similarity increases, and the value of the second loss function increases as the first similarity decreases. The second similarity and the second loss function have different directions of change in magnitude, the value of the second loss function increases as the first similarity increases, and the value of the second loss function decreases as the first similarity decreases. That is to say, when the training goal is to make the second loss function as small as possible, the value of the first similarity needs to be as large as possible, and the value of the second similarity needs to be as small as possible.

[0156] In general, a higher first similarity indicates a higher segmentation accuracy of the semantic segmentation model and a lower second loss function derived from the first similarity. A lower second similarity indicates a higher segmentation accuracy of the semantic segmentation model and a lower second loss function derived from the second similarity. Since the training goal of the semantic segmentation model is to keep the second loss function as low as possible, the constraints of the second loss function allow the semantic segmentation model to more accurately segment objects in the image.

[0157] It should be noted that the above describes the calculation of two similarities associated with two mask images corresponding to a certain object (i.e., the target object) in the first semantic segmentation result. In practical applications, the training device may also obtain two mask images corresponding to each object in the first semantic segmentation result and calculate two similarities associated with the two mask images corresponding to each object, that is, calculate the similarities associated with each object in the first semantic segmentation result. Then, the training device calculates the second loss function based on the similarities associated with each object in the first semantic segmentation result.

[0158] This solution introduces a second loss function that characterizes the differences between the target object and the categorized text, as well as the differences between the target object and other objects, in the semantic segmentation results obtained by the semantic segmentation model. Further training the semantic segmentation model based on the second loss function, building on the first loss function, can guide the semantic segmentation model to learn more precise region segmentation, thereby improving the segmentation accuracy of the semantic segmentation model.

[0159] Implementation method two: The training device trains the semantic segmentation model based on the first loss function and the third loss function.

[0160] Specifically, the training device inputs the first image into a denoising model to obtain a second semantic segmentation result. The denoising model has the same structure as the semantic segmentation model, and the network parameters of the denoising model are different from the network parameters of the semantic segmentation model. For example, the denoising model and the semantic segmentation model both have convolutional neural network structures, or both have Transformer structures.

[0161] Then, the training device obtains a third loss function according to the first semantic segmentation result and the second semantic segmentation result, and trains the semantic segmentation model based on the first loss function and the third loss function.

[0162] Optionally, the network parameters of the denoising model are derived based on the network parameters of the semantic segmentation model and the network parameters of the denoising model during the previous round of iterative training. That is, during training, compared to the rapid update of the network parameters in the semantic segmentation model, the update of the network parameters in the denoising model has a certain lag, and the update of the network parameters in the denoising model will be slower and closer to the network parameters in the previous training process.

[0163] In this solution, by introducing a denoising model with the same structure as the semantic segmentation model during the training process of the semantic segmentation model, the training oscillation that occurs during the training of the semantic segmentation model can be effectively suppressed, and the training stability of the semantic segmentation model can be improved, so as to efficiently realize the training of the semantic segmentation model.

[0164] Implementation method three: The training device trains the semantic segmentation model based on the first loss function, the second loss function and the third loss function.

[0165] Specifically, after the training device obtains the second loss function based on the above-mentioned implementation method one and obtains the third loss function based on the above-mentioned implementation method two, the training device trains the semantic segmentation model based on the first loss function, the second loss function and the third loss function.

[0166] The above introduces the implementation process of a training method for an unsupervised semantic segmentation model provided in an embodiment of the present application. The following will introduce the training method for the unsupervised semantic segmentation model in detail with reference to specific examples.

[0167] In this embodiment, three models are used as teacher networks, and the semantic segmentation model to be trained as the student network is trained by knowledge distillation, thereby obtaining a trained semantic segmentation model.

[0168] For example, see Figure 6 , Figure 6 This is a schematic diagram of a training framework provided in an embodiment of the present application. Figure 6As shown in Figure 2, in this training framework, the semantic model, mask model, and denoising model are the teacher network, and the semantic segmentation model to be trained is the student network. The semantic model and mask model are based on the multimodal pre-training model, while the denoising model is based on the semantic segmentation model.

[0169] Specifically, the input of the semantic model includes an input image and its possible categories. The output of the semantic model is pseudo-annotations generated based on the input image, that is, the result of performing semantic segmentation on the input image. Based on the output of the semantic model and the output of the semantic segmentation model, the first loss function described in the above embodiment can be constructed.

[0170] The input of the mask model is the prediction result of the student network (i.e., the semantic segmentation result predicted by the student network), and the output of the mask model is multiple similarities, including the similarity between the target area and the non-target area in the student network's prediction result, and the similarity between the target area and the category text corresponding to the target area. Based on the output of the mask model and the output of the semantic segmentation model, a second loss function as described in the above embodiment can be constructed.

[0171] The input of the denoising model includes the input image, and the output of the denoising model is the prediction result generated based on the input image (i.e., the semantic segmentation result predicted by the denoising model). Based on the output of the denoising model and the output of the semantic segmentation model, the third loss function described in the above embodiment can be constructed.

[0172] See Figure 7 , Figure 7 A schematic diagram of the training process of an unsupervised semantic segmentation model provided in an embodiment of the present application. The semantic segmentation model, serving as the student network, uses the Vision Transformer model, and the backbone network of the semantic segmentation model is a ViT-B / 16 structure. The semantic model uses the CLIP model and the DINO model.

[0173] The following will be combined Figure 7 The working process of each model is introduced in detail.

[0174] First, we introduce a semantic model for generating pseudo-annotations. This semantic model is used to extract, aggregate, and classify pixel-level features based on the generalized recognition capabilities of pre-trained large models (such as the CLIP model and the DINO model), thereby acquiring pixel-level semantic knowledge and generating pseudo-annotations.

[0175] Specifically, see Figure 8 , Figure 8 A workflow diagram of a semantic model provided in an embodiment of the present application.

[0176] like Figure 8 As shown in FIG, for a certain input image in the training dataset, the input image may include objects of multiple different categories. Therefore, the semantic model predicts the category set of objects in the input image based on the CLIP model.

[0177] Specifically, the training device first obtains the label set of the training data set, which indicates the categories that any input image in the training data set may include. For example, the category labels in the label set include: people, dogs, cats, bottles and other labels. Then, the training device inputs the label set and the input image into the CLIP model, and the CLIP model extracts the features of the category labels and the features of the input image, and calculates the similarity between the features of each category label and the features of the input image. Based on a pre-set threshold τ, the features of the category labels whose similarity with the features of the input image is greater than the threshold τ are determined, so that the category labels corresponding to these features are determined as the categories of the input image, and the category set corresponding to the input image is obtained. Figure 8 As shown, the category set C corresponding to the input image is C={person, dog}, that is, the category set corresponding to the input image includes people and dogs.

[0178] In addition, the training device also performs feature extraction on the input image through the DINO model in the semantic model to obtain a feature map of the input image. Due to the characteristics of the DINO model, a feature map with higher resolution can be extracted based on the DINO model, thereby extracting pixel-level features of the input image. Secondly, the number of cluster centers is determined based on the category set corresponding to the input image, and the feature map of the input image is clustered based on the k-means clustering method to obtain multiple feature regions (such as Figure 8 In the case where multiple feature regions are obtained through clustering, multiple image regions corresponding to the multiple feature regions are extracted from the first image based on the positions of the multiple feature regions in the image.

[0179] After obtaining multiple image regions, multiple category texts corresponding to the multiple image regions and the category set of the input image are input into the CLIP model. The CLIP model extracts the features of each image region and the features of each category text, and calculates the similarity between the features of each image region and the features of each category text, thereby matching the image region with the category text and obtaining the category corresponding to each image region, that is, obtaining the pseudo-label corresponding to the input image. The first loss function L is used based on the relationship between the pseudo-label P and the prediction result S of the semantic segmentation model to be trained. ST , the semantic segmentation model is able to learn semantic segmentation in an unsupervised manner.

[0180] Exemplarily, the first loss function may be as shown in Formula 1.

[0181] L ST (P, S) = -E P [logS] Formula 1

[0182] Among them, L ST Represents the first loss function, P represents pseudo-labeling, and S represents the prediction result of the semantic segmentation model.

[0183] The previous section described how to generate pseudo-annotations based on semantic models, including the CLIP model and the DINO model, to assist the semantic segmentation model in achieving unsupervised learning of semantic segmentation. The following section describes how to use mask models to guide the semantic segmentation model in learning to segment regions more accurately.

[0184] See Figure 9 , Figure 9 This is a schematic diagram of the workflow of a mask model provided in an embodiment of the present application. Figure 9 As shown in Figure 1, for the target object in the image - a motorcycle, due to the lack of accurate pseudo-labeling, the mask learned by the semantic segmentation model cannot accurately cover the target area of ​​the motorcycle. For the prediction result S output by the semantic segmentation model, each channel of the prediction result S corresponds to the segmentation result of a category. For a possible category c, the prediction mask corresponding to the prediction result S is S c Thus, for the predicted mask S c , we can divide the positive and negative sample areas, the positive sample area (i.e., the first mask image in the above embodiment) corresponds to the foreground of the current category, the negative sample area (i.e., the second mask image in the above embodiment) corresponds to the background. For example, the positive sample area and negative sample areas This can be shown as the following formula 2 and formula 3.

[0185]

[0186]

[0187] Where σ(·) represents the sigmoid activation function, ∈ p ,∈ n is the preset parameter, ∈ p ,∈ n The value of can be, for example, 0.65 and 0.4, η is a preset parameter, and the value of η can be, for example, 0.03.

[0188] Then, based on the CLIP model in the mask model, the positive sample area and negative sample areas Extract image features separately and image features Then extract the text feature l for the label of the current category c c In the design process of the second loss function, in order to make the foreground area contain only the target area of ​​the corresponding category, the image features of the positive sample area Need to be as close as possible to the text feature l of category c c , to suppress the non-current class area (false positive) in the foreground. In order to prevent the background area from containing the target area of ​​category c, the image features of the negative sample area Image features that need to be as far away from the positive sample area as possible To suppress the current class area in the background (false negative). Exemplarily, the second loss function can be as shown in Formula 4.

[0189]

[0190] Among them, L MT (S, L) represents the second loss function, C represents the category set of the prediction result S, and c represents a category in the category set of the prediction result S. Represents the image features of the positive sample area in the image of category c, l c represents the text features of category c, Image features representing negative sample regions in images of category c.

[0191] The previous section introduced the process of using a mask model to guide the semantic segmentation model to learn to segment regions more accurately. The following section describes the process of performing denoising on the training process to improve the stability of the semantic segmentation model during training.

[0192] Since the noise information contained in the pseudo-labels P generated by the semantic model will cause the model training to oscillate, this embodiment introduces a denoising model to improve the training stability. The denoising model has the same structure as the semantic segmentation model. During the forward propagation, it outputs a prediction result T as auxiliary supervision information, which is used to calculate the third loss function L with the model prediction. DT . Through the third loss function L DT To constrain the learning of the semantic segmentation model and suppress training oscillation. For example, the third loss function L DT As shown in Formula 5.

[0193] L DT (T, S) = -E T [logS] Formula 5

[0194] Among them, L DT(T, S) represents the third loss function, T represents the prediction result of the denoising model, and S represents the prediction result of the semantic segmentation model.

[0195] Furthermore, the network parameters of the denoising model are different from those of the semantic segmentation model. The network parameters of the denoising model are updated using an exponential moving average strategy. For example, assuming the network parameters of the semantic segmentation model are θ, the network parameters of the denoising model during the current training process are φ1, and the network parameters of the denoising model during the previous iterative training process are φ0, the parameter update formula for the denoising model can be, for example, the following formula 6.

[0196] φ1=λφ0+(1-λθ) Formula 6

[0197] Wherein, λ is a noise reduction factor, and its value range is [0, 1]. For example, the value of λ may be 0.9.

[0198] After obtaining the first loss function, the second loss function and the third loss function based on the above three teacher network models, the semantic segmentation model can be trained according to these three loss functions until the convergence condition is reached to obtain a trained semantic segmentation model.

[0199] Optionally, during the training process of the semantic segmentation model, the training dataset of the semantic segmentation model can be a closed set or an open set, as described below.

[0200] Closed set: The training dataset is predefined and immutable. After the semantic segmentation model is trained with the guidance of multiple teacher networks, it can be used directly for inference. During inference, after an image is input into the semantic segmentation model, the model can predict the semantic segmentation result for the image without any post-processing.

[0201] Open set: The training dataset is dynamically changing. For example, see Figure 10 , Figure 10 This is a schematic diagram of an open set-based semantic segmentation model training method provided in an embodiment of the present application. Figure 10 As shown in the figure, during the tth round of training iteration, the semantic segmentation model is trained based on the predefined training dataset and the category labels in the training dataset to obtain a semantic segmentation result that can identify the dog in the image. After the completion of the tth round of training iteration, the training dataset has changed (i.e., some new training data has been added), and the category labels in the training dataset have also changed. Therefore, the semantic segmentation model is trained for the t+1th round based on the newly added training data to obtain an updated semantic segmentation model. The updated semantic segmentation model can recognize the dog and the tree in the image.

[0202] Simply put, when the training dataset is open, the addition of training data for new categories triggers multiple teacher network models to re-mine relevant knowledge from the training data for the new categories, which is used to assist the incremental learning of the semantic segmentation model. In this way, after fine-tuning the semantic segmentation model on the new label set, it can be used for inference and segmentation of new categories.

[0203] After verification, the effect brought by the training method of the unsupervised semantic segmentation model provided in the embodiment of the present application is as described below.

[0204] On the public dataset PASCAL VOC 2012, compared with non-learning methods (such as CLIP Baseline and DINOBasline), self-supervised learning methods (such as MoCov2), and unsupervised learning methods (such as MaskCon), the unsupervised semantic segmentation model training method provided in the embodiment of the present application (hereinafter referred to as SegCLIP) achieved the best results, as shown in Table 1.

[0205] Table 1

[0206]

[0207] As can be seen from Table 1, without using manual annotation for training, the model trained by the method of the embodiment of the present application has the highest accuracy.

[0208] Among them, CLIP Baseline and DINO Baseline refer to directly using the pre-trained model CLIP or DINO to extract pixel-level feature clusters, and then calculating the similarity with the language encoding of the category label to predict its category label, thereby obtaining the semantic segmentation result.

[0209] 2. MoCo v2 is a self-supervised representation learning method that can be used to extract pixel-level features. These features are clustered in two ways (K-Means and Retrieval) to obtain category labels, thereby obtaining semantic segmentation results.

[0210] The KMeans method extracts pixel-level features from the test set images and clusters them. It then extracts pixel features from a pixel-annotated support set and clusters them. The class label of each cluster is the label of the pixels it contains. The similarity between the cluster centers on the test and support sets is calculated pairwise. For each cluster in the test set, the class label of the cluster in the support set with the highest similarity is selected. This ensures that the pixels in each cluster in the test set have semantic class labels, resulting in semantic segmentation results.

[0211] The retrieval method extracts pixel-level features from the test set images and then clusters them. The pixels in each cluster form a mask, and the average of these pixel features is used as the mask feature. Similarly, mask features are extracted from the support set, as the support set has pixel-level annotations and its masks already have category labels. The nearest neighbor mask retrieved from the support set using the mask features of the test set is used as the prediction to generate the semantic segmentation result.

[0212] MaskCon is the most accurate unsupervised semantic segmentation method available. It relies on a saliency pre-training model to learn the correlation between pixels. During testing, the mask can obtain semantic labels through the above-mentioned KMeans and Retrieval methods.

[0213] In Table 1, the evaluation metric mIoU (mean Intersection over Union) is the ratio of the intersection and union of the true and predicted values. The evaluation metric pix.acc (pixel accuracy) is the ratio of correctly predicted pixels to the total number of pixels.

[0214] In addition, this embodiment also provides an ablation experiment to verify the effectiveness of the different teacher network models proposed in this embodiment. Specifically, the results of the ablation experiment are shown in Table 2.

[0215] Table 2

[0216] Semantic Model Mask Model Noise Reduction Model mIoU (%) 38.3 √ 47.5 √ √ 48.7 √ √ 49.2 √ √ √ 51.1

[0217] 1. Semantic model.

[0218] In the method provided in this embodiment, the semantic model is a module that obtains pixel-level semantic knowledge from a pre-trained model and uses it as pseudo-annotations to train a semantic segmentation model. To minimize the functionality of the semantic model, this embodiment constructs a separate model that abandons the mining of pixel-level semantic information. The pseudo-annotations it generates contain only two categories, foreground and background. As shown in Table 2, compared to pseudo-annotations containing semantic knowledge, the accuracy of the model without semantic information decreases by 9.2% (from 47.5% to 38.3%).

[0219] 2. Mask model.

[0220] As shown in Table 2, after removing the mask model, the accuracy of the trained semantic segmentation model decreases by 1.9% (from 51.1% to 49.2%), indicating that the mask model can improve the accuracy of semantic segmentation.

[0221] 3. Noise reduction model.

[0222] As shown in Table 2, after removing the denoising model, the accuracy of the trained semantic segmentation model decreased by 2.4% (from 51.1% to 48.7%), indicating that the denoising module provided in this embodiment is very effective in the unsupervised semantic segmentation framework and can bring certain performance improvements.

[0223] See Figure 11 , Figure 11 This is a schematic diagram of the optimization of a semantic segmentation model during training provided in an embodiment of the present application. Figure 11 As shown in Figure 3, as the training progresses, the final semantic segmentation model can learn more accurate segmentation results even when the initial pseudo-labeling is inaccurate.

[0224] See Figure 12 , Figure 12 This is a schematic diagram of the semantic segmentation effect of a semantic segmentation model in different scenarios provided by the embodiment of this application. Figure 12 As shown, the semantic segmentation model trained based on the method provided in the embodiment of the present application can also obtain accurate semantic segmentation results in complex scenarios.

[0225] A. Missing pixels: In the first column, the target to be segmented is a cow. Due to the difference in color, its initial pseudo-labeling is not accurate. After model training, the semantic segmentation model is finally able to predict an accurate mask.

[0226] B. Redundant pixels: In the second column, the initial segmentation result contains redundant pixels that are incorrectly labeled as chairs. After model training, the semantic segmentation model finally obtains a more accurate chair mask.

[0227] C. Cluttered pixels: In the third column, the pixels of the cat and dog categories are easily mixed together because they are close to each other and have similar fur color and texture. However, after model training, the semantic segmentation model finally obtains more accurate results.

[0228] D. Wrong labels: In the fourth column, in the initial pseudo-labeling, cars are mistakenly classified as cows. After model training, the semantic segmentation model finally obtains the correct labels.

[0229] E. Failed mask: In the fifth column, the initial pseudo-labeling fails, but after model training, the semantic segmentation model can still make accurate predictions.

[0230] The above introduces a training method for an unsupervised semantic segmentation model provided by an embodiment of the present application. The following introduces a semantic segmentation method provided by an embodiment of the present application.

[0231] Specifically, a semantic segmentation method provided in an embodiment of the present application includes: obtaining an image to be processed, and inputting the image to be processed into a semantic segmentation model to obtain a first semantic segmentation result.

[0232] In which, the semantic segmentation model is trained based on a first loss function, the first loss function is obtained based on the output result and the second semantic segmentation result, the second semantic segmentation result is obtained by inputting the first image in the training data set into the semantic segmentation result model, the output result is obtained by inputting multiple image regions and multiple texts describing a category set into a multimodal model, the output result is used to indicate the category corresponding to each of the multiple image regions, the multimodal model is used to perform image and text matching, the multiple image regions are image regions in the first image corresponding to multiple feature regions, the multiple feature regions are obtained by clustering the feature map of the first image according to the category set of the first image, the category set is used to indicate the categories of multiple objects in the first image, and each feature region in the multiple feature regions corresponds to a category in the category set.

[0233] Specifically, the semantic segmentation model can be trained based on the training method of the unsupervised semantic segmentation model described in the above embodiment. The specific training process can refer to the introduction of the above embodiment and will not be described here.

[0234] In one possible implementation, the multimodal model is used to extract multiple image features corresponding to the multiple image regions and multiple text features corresponding to the multiple texts, and calculate the similarity between each of the multiple image features and the multiple text features to obtain the category corresponding to each of the multiple image regions; wherein the category corresponding to the first image region is the category corresponding to the text feature with the highest similarity to the first image feature, the first image region is any one of the multiple image regions, and the first image feature is the image feature among the multiple image features corresponding to the first image region.

[0235] In one possible implementation, the training data set includes multiple images and multiple category labels, the multiple images include the first image, and the multiple category labels are used to indicate the categories of objects in the multiple images; the category set of the first image is obtained based on multiple similarities, the category set includes multiple target categories, and the similarities between the features of the multiple target categories and the features of the first image are all greater than or equal to a preset threshold, the multiple similarities are multiple similarities between the features of the first image and the features of each category label in the multiple category labels, and the multiple similarities are obtained after the first image and the multiple category labels are input into the multimodal model.

[0236] In one possible implementation, the semantic segmentation model is trained based on the first loss function and the second loss function, the second loss function is obtained based on the first similarity and the second similarity, the first similarity is the similarity between the features of the first mask image and the features of the category text, the second similarity is the similarity between the features of the first mask image and the features of the second mask image, the first similarity and the second similarity are obtained by inputting the first mask image, the second mask image and the category text corresponding to the target object into the multimodal model, the first mask image and the second mask image are obtained based on the second semantic segmentation result, the first mask image only includes the image of the target object indicated in the second semantic segmentation result, the second mask image only includes the image other than the target object in the second semantic segmentation result, and the target object is an object of any category in the first semantic segmentation result.

[0237] In a possible implementation, the first similarity is negatively correlated with the second loss function, and the second similarity is positively correlated with the second loss function.

[0238] In one possible implementation, the semantic segmentation model is trained based on the first loss function and the third loss function, the third loss function is obtained based on the second semantic segmentation result and the third semantic segmentation result, the third semantic segmentation result is obtained after inputting the first image into the denoising model, the structure of the denoising model is the same as that of the semantic segmentation model, and the network parameters of the denoising model are different from the network parameters of the semantic segmentation model.

[0239] In one possible implementation, the semantic segmentation model is trained based on the first loss function, the second loss function and the third loss function, the third loss function is obtained based on the second semantic segmentation result and the third semantic segmentation result, the third semantic segmentation result is obtained after inputting the first image into the denoising model, the structure of the denoising model is the same as that of the semantic segmentation model, and the network parameters of the denoising model are different from the network parameters of the semantic segmentation model.

[0240] In a possible implementation, the network parameters of the denoising model are obtained based on the network parameters of the semantic segmentation model and the network parameters of the denoising model in the previous round of iterative training.

[0241] The above introduces the model training method and semantic segmentation method provided in the embodiments of the present application. For ease of understanding, the devices for executing the above-mentioned model training method and semantic segmentation method will be introduced below.

[0242] See Figure 13 , Figure 13 A structural diagram of a training device for an unsupervised semantic segmentation model provided in an embodiment of the present application. Figure 13 As shown, the training device 1300 for the unsupervised semantic segmentation model includes: an acquisition module 1301, a processing module 1302 and a training module 1303.

[0243] An acquisition module 1301 is used to acquire a feature map of a first image in a training data set, and cluster the feature map according to a category set of the first image to obtain multiple feature regions in the feature map, wherein the category set is used to indicate the categories of multiple objects in the first image, and each feature region corresponds to a category in the category set; a processing module 1302 is used to determine multiple image regions in the first image corresponding to the multiple feature regions; the processing module 1302 is also used to input the multiple image regions and multiple texts describing the category set into a multimodal model to obtain an output result, wherein the output result is used to indicate the category corresponding to each image region in the multiple image regions, wherein the multimodal model is used to perform image and text matching; the processing module 1302 is also used to input the first image into a semantic segmentation model to obtain a first semantic segmentation result; a training module 1303 is used to train the semantic segmentation model according to a first loss function to update the semantic segmentation model, wherein the first loss function is constructed based on the output result and the first semantic segmentation result.

[0244] In one possible implementation, the multimodal model is used to extract multiple image features corresponding to the multiple image regions and multiple text features corresponding to the multiple texts, and calculate the similarity between each of the multiple image features and the multiple text features to obtain the category corresponding to each of the multiple image regions; wherein the category corresponding to the first image region is the category corresponding to the text feature with the highest similarity to the first image feature, the first image region is any one of the multiple image regions, and the first image feature is the image feature among the multiple image features corresponding to the first image region.

[0245] In one possible implementation, the training data set includes multiple images and multiple category labels, the multiple images include the first image, and the multiple category labels are used to indicate the categories of objects in the multiple images; the processing module 1302 is further used to: input the first image and the multiple category labels into the multimodal model to obtain multiple similarities between the features of the first image and the features of each category label in the multiple category labels; based on the multiple similarities, determine a category set of the first image, wherein the category set includes multiple target categories, and the similarities between the features of the multiple target categories and the features of the first image are all greater than or equal to a preset threshold.

[0246] In one possible implementation, the processing module 1302 is also used to obtain a first mask image and a second mask image based on the first semantic segmentation result, wherein the first mask image only includes the image of the target object indicated in the first semantic segmentation result, and the second mask image only includes the image other than the target object in the first semantic segmentation result, and the target object is an object of any category in the first semantic segmentation result; the processing module 1302 is also used to input the first mask image, the second mask image and the category text corresponding to the target object into the multimodal model to obtain a first similarity between the features of the first mask image and the features of the category text, and a second similarity between the features of the first mask image and the features of the second mask image; the training module 1303 is specifically used to train the semantic segmentation model according to the first loss function and the second loss function, wherein the second loss function is obtained based on the first similarity and the second similarity.

[0247] In a possible implementation, the first similarity is negatively correlated with the second loss function, and the second similarity is positively correlated with the second loss function.

[0248] In one possible implementation, the processing module 1302 is also used to input the first image into a denoising model to obtain a second semantic segmentation result, the structure of the denoising model is the same as the structure of the semantic segmentation model, and the network parameters of the denoising model are different from the network parameters of the semantic segmentation model; the training module 1303 is specifically used to train the semantic segmentation model according to the first loss function and the third loss function, and the third loss function is obtained based on the first semantic segmentation result and the second semantic segmentation result.

[0249] In one possible implementation, the processing module 1302 is also used to input the first image into a denoising model to obtain a second semantic segmentation result. The structure of the denoising model is the same as that of the semantic segmentation model, and the network parameters of the denoising model are different from the network parameters of the semantic segmentation model. The training module 1303 is specifically used to train the semantic segmentation model according to the first loss function, the second loss function and the third loss function. The third loss function is obtained based on the first semantic segmentation result and the second semantic segmentation result.

[0250] In a possible implementation, the network parameters of the denoising model are obtained based on the network parameters of the semantic segmentation model and the network parameters of the denoising model in the previous round of iterative training.

[0251] See Figure 14 , Figure 14 This is a structural diagram of a semantic segmentation device provided in an embodiment of the present application. Figure 14As shown, a semantic segmentation device 1400 includes: an acquisition module 1401 and a processing module 1402. The acquisition module 1401 is used to acquire an image to be processed; the processing module 1402 is used to input the image to be processed into a semantic segmentation model to obtain a first semantic segmentation result; wherein the semantic segmentation model is trained based on a first loss function, the first loss function is obtained based on an output result and a second semantic segmentation result, the second semantic segmentation result is obtained by inputting a first image in a training dataset into the semantic segmentation result model, the output result is obtained by inputting multiple image regions and multiple texts describing a category set into a multimodal model, the output result is used to indicate the category corresponding to each of the multiple image regions, the multimodal model is used to perform image and text matching, the multiple image regions are image regions in the first image corresponding to multiple feature regions, the multiple feature regions are obtained by clustering the feature map of the first image according to the category set of the first image, the category set is used to indicate the categories of multiple objects in the first image, and each of the multiple feature regions corresponds to a category in the category set.

[0252] In one possible implementation, the multimodal model is used to extract multiple image features corresponding to the multiple image regions and multiple text features corresponding to the multiple texts, and calculate the similarity between each of the multiple image features and the multiple text features to obtain the category corresponding to each of the multiple image regions; wherein the category corresponding to the first image region is the category corresponding to the text feature with the highest similarity to the first image feature, the first image region is any one of the multiple image regions, and the first image feature is the image feature among the multiple image features corresponding to the first image region.

[0253] In one possible implementation, the training data set includes multiple images and multiple category labels, the multiple images include the first image, and the multiple category labels are used to indicate the categories of objects in the multiple images; the category set of the first image is obtained based on multiple similarities, the category set includes multiple target categories, and the similarities between the features of the multiple target categories and the features of the first image are all greater than or equal to a preset threshold, the multiple similarities are multiple similarities between the features of the first image and the features of each category label in the multiple category labels, and the multiple similarities are obtained after the first image and the multiple category labels are input into the multimodal model.

[0254] In one possible implementation, the semantic segmentation model is trained based on the first loss function and the second loss function, the second loss function is obtained based on the first similarity and the second similarity, the first similarity is the similarity between the features of the first mask image and the features of the category text, the second similarity is the similarity between the features of the first mask image and the features of the second mask image, the first similarity and the second similarity are obtained by inputting the first mask image, the second mask image and the category text corresponding to the target object into the multimodal model, the first mask image and the second mask image are obtained based on the second semantic segmentation result, the first mask image only includes the image of the target object indicated in the second semantic segmentation result, the second mask image only includes the image other than the target object in the second semantic segmentation result, and the target object is an object of any category in the first semantic segmentation result.

[0255] In a possible implementation, the first similarity is negatively correlated with the second loss function, and the second similarity is positively correlated with the second loss function.

[0256] In one possible implementation, the semantic segmentation model is trained based on the first loss function and the third loss function, the third loss function is obtained based on the second semantic segmentation result and the third semantic segmentation result, the third semantic segmentation result is obtained after inputting the first image into the denoising model, the structure of the denoising model is the same as that of the semantic segmentation model, and the network parameters of the denoising model are different from the network parameters of the semantic segmentation model.

[0257] In one possible implementation, the semantic segmentation model is trained based on the first loss function, the second loss function and the third loss function, the third loss function is obtained based on the second semantic segmentation result and the third semantic segmentation result, the third semantic segmentation result is obtained after inputting the first image into the denoising model, the structure of the denoising model is the same as that of the semantic segmentation model, and the network parameters of the denoising model are different from the network parameters of the semantic segmentation model.

[0258] In a possible implementation, the network parameters of the denoising model are obtained based on the network parameters of the semantic segmentation model and the network parameters of the denoising model in the previous round of iterative training.

[0259] Next, we will introduce an execution device provided by the embodiment of the present application. Figure 15 , Figure 15This is a schematic diagram of the structure of an execution device provided in an embodiment of the present application. The execution device 1500 can be specifically manifested as a server, a smart phone, a tablet computer, a personal computer or a laptop computer, etc., which is not limited here. Specifically, the execution device 1500 includes: a receiver 1501, a transmitter 1502, a processor 1503 and a memory 1504 (wherein the number of processors 1503 in the execution device 1500 can be one or more, Figure 15 (taking one processor as an example), the processor 1503 may include an application processor 15031 and a communication processor 15032. In some embodiments of the present application, the receiver 1501, the transmitter 1502, the processor 1503 and the memory 1504 may be connected via a bus or other means.

[0260] Memory 1504 may include read-only memory and random access memory, and provides instructions and data to processor 1503. A portion of memory 1504 may also include non-volatile random access memory (NVRAM). Memory 1504 stores processor and operation instructions, executable modules, or data structures, or subsets or extended sets thereof. The operation instructions may include various operation instructions for implementing various operations.

[0261] Processor 1503 controls the operation of the execution device. In specific applications, the various components of the execution device are coupled together via a bus system. In addition to a data bus, the bus system may also include a power bus, a control bus, and a status signal bus. However, for clarity, all bus systems are referred to as a bus system in the figure.

[0262] The methods disclosed in the above embodiments of the present application can be applied to or implemented by processor 1503. Processor 1503 can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits or software instructions in processor 1503. The above processor 1503 can be a general-purpose processor, a digital signal processor (DSP), a microprocessor, or a microcontroller, and can further include an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The processor 1503 can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in conjunction with the embodiments of the present application can be directly implemented as being executed by a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in memory 1504, and processor 1503 reads the information in memory 1504 and, in conjunction with its hardware, completes the steps of the above method.

[0263] Receiver 1501 can be used to receive input digital or character information and generate signal input related to executing device-related settings and function control. Transmitter 1502 can be used to output digital or character information through the first interface. Transmitter 1502 can also be used to send instructions to the disk pack through the first interface to modify data in the disk pack. Transmitter 1502 can also include a display device such as a display screen.

[0264] The present application also provides a training device. Figure 16 , Figure 16A structural diagram of a training device provided in an embodiment of the present application. Specifically, the training device 1600 is implemented by one or more servers. The training device 1600 may have relatively large differences due to different configurations or performances. It may include one or more central processing units (CPUs) 1622 (for example, one or more processors) and memory 1632, and one or more storage media 1630 (for example, one or more mass storage devices) storing application programs 1642 or data 1644. Among them, the memory 1632 and the storage medium 1630 can be temporary storage or permanent storage. The program stored in the storage medium 1630 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the training device. Furthermore, the central processing unit 1622 can be configured to communicate with the storage medium 1630 to execute a series of instruction operations in the storage medium 1630 on the training device 1600.

[0265] The training device 1600 may also include one or more power supplies 1626, one or more wired or wireless network interfaces 1650, one or more input and output interfaces 1658, and / or one or more operating systems 1641, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0266] The present application also provides a computer program product which, when executed on a computer, enables the computer to execute the aforementioned Figures 4 to 12 The illustrated embodiment describes the method.

[0267] The present application also provides a computer-readable storage medium in which a program for signal processing is stored. When the program is run on a computer, the computer executes the above-mentioned Figures 6 to 16 The illustrated embodiment describes the method.

[0268] The image processing device, model training device, execution device and training device provided in the embodiments of the present application may specifically include a chip, which includes: a processing unit and a communication unit. The processing unit may be, for example, a processor, and the communication unit may be, for example, an input / output interface, a pin or a circuit. The processing unit may execute the computer execution instructions stored in the storage unit to enable the chip to execute the above-mentioned Figures 4 to 12The method described in the embodiment shown. Optionally, the storage unit is a storage unit within the chip, such as a register, a cache, etc. The storage unit may also be a storage unit located outside the chip within the wireless access device, such as a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM), etc.

[0269] For details, please refer to Figure 17 , Figure 17 This is a schematic diagram of the structure of a chip provided in an embodiment of the present application. The chip can be represented as a neural network processor NPU 170. NPU 170 is mounted on the host CPU as a coprocessor and is assigned tasks by the host CPU. The core of the NPU is arithmetic circuit 1703, which is controlled by controller 1704 to extract matrix data from memory and perform multiplication operations.

[0270] In some implementations, arithmetic circuit 1703 includes multiple processing units (PEs). In some implementations, arithmetic circuit 1703 is a two-dimensional systolic array. Arithmetic circuit 1703 can also be a one-dimensional systolic array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some implementations, arithmetic circuit 1703 is a general-purpose matrix processor.

[0271] For example, assume there are input matrix A, weight matrix B, and output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from weight memory 1702 and caches it on each PE in the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from input memory 1701 and performs a matrix operation on matrix B. The partial or final matrix result is stored in accumulator 1708.

[0272] Unified memory 1706 is used to store input and output data. Weight data is directly transferred to weight memory 1702 through the Direct Memory Access Controller (DMAC) 1705. Input data is also transferred to unified memory 1706 through the DMAC.

[0273] BIU stands for Bus Interface Unit 1710 , which is used for interaction between the AXI bus, DMAC, and Instruction Fetch Buffer (IFB) 1709 .

[0274] The bus interface unit 1710 (BIU) is used for the instruction fetch memory 1709 to obtain instructions from the external memory, and is also used for the storage unit access controller 1705 to obtain the original data of the input matrix A or the weight matrix B from the external memory.

[0275] DMAC is mainly used to move input data in the external memory DDR to the unified memory 1706 or move weight data to the weight memory 1702 or move input data to the input memory 1701.

[0276] The vector calculation unit 1707 includes multiple operation processing units. When necessary, it further processes the output of the operation circuit, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. It is mainly used for non-convolutional / fully connected layer network calculations in neural networks, such as batch normalization, pixel-level summation, and upsampling of feature planes.

[0277] In some implementations, the vector calculation unit 1707 can store the processed output vector to the unified memory 1706. For example, the vector calculation unit 1707 can apply a linear function and / or a nonlinear function to the output of the operation circuit 1703, such as linear interpolation of the feature plane extracted by the convolution layer, or accumulate a vector of values ​​to generate an activation value. In some implementations, the vector calculation unit 1707 generates a normalized value, a pixel-level summed value, or both. In some implementations, the processed output vector can be used as an activation input to the operation circuit 1703, for example, for use in a subsequent layer in a neural network.

[0278] An instruction fetch buffer 1709 connected to the controller 1704 is used to store instructions used by the controller 1704;

[0279] Unified memory 1706, input memory 1701, weight memory 1702, and instruction fetch memory 1709 are all on-chip memories. External memories are private to the NPU hardware architecture.

[0280] The processor mentioned in any of the above places can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits for controlling the execution of the program of the above-mentioned first aspect method.

[0281] It should also be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. In addition, in the drawings of the device embodiments provided in this application, the connection relationship between the modules indicates that there is a communication connection between them, which can be specifically implemented as one or more communication buses or signal lines.

[0282] Through the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course can also be implemented by special hardware including application-specific integrated circuits, special CPUs, special memories, special components, etc. In general, all functions performed by computer programs can be easily implemented with corresponding hardware, and the specific hardware structures used to implement the same function can also be diverse, such as analog circuits, digital circuits or special circuits, etc. However, for the present application, software program implementation is a better implementation method in most cases. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer's floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk or optical disk, etc., and includes a number of instructions to enable a computer device (which can be a personal computer, training equipment, or network equipment, etc.) to execute the methods described in each embodiment of the present application.

[0283] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0284] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, a computer, a training device or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website, a computer, a training device or a data center. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium can be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

Claims

1. A training method for an unsupervised semantic segmentation model, characterized in that: include: Obtaining a feature map of a first image in a training data set, and clustering the feature map according to a category set of the first image to obtain a plurality of feature regions in the feature map, wherein the category set is used to indicate categories of a plurality of objects in the first image, and each of the plurality of feature regions corresponds to a category in the category set; determining a plurality of image regions in the first image corresponding to the plurality of feature regions; inputting the plurality of image regions and a plurality of texts describing the category set into a multimodal model to obtain an output result, the output result being used to indicate that each of the plurality of image regions corresponds to a category described by a text in the category set, wherein the multimodal model is used to perform image and text matching; Inputting the first image into a semantic segmentation model to obtain a first semantic segmentation result; The semantic segmentation model is trained according to a first loss function to update the semantic segmentation model, wherein the first loss function is constructed based on the output result and the first semantic segmentation result.

2. The method according to claim 1, characterized in that The multimodal model is used to extract a plurality of image features corresponding to the plurality of image regions and a plurality of text features corresponding to the plurality of texts, and calculate a similarity between each of the plurality of image features and the plurality of text features to obtain a category corresponding to each of the plurality of image regions; Among them, the category corresponding to the first image area is the category corresponding to the text feature with the highest similarity to the first image feature, the first image area is any one of the multiple image areas, and the first image feature is the image feature corresponding to the first image area among the multiple image features.

3. The method according to claim 1, characterized in that The training dataset includes a plurality of images and a plurality of category labels, the plurality of images including the first image, and the plurality of category labels are used to indicate categories of objects in the plurality of images; The method further comprises: Inputting the first image and the multiple category labels into the multimodal model to obtain multiple similarities between features of the first image and features of each of the multiple category labels; A category set of the first image is determined based on the multiple similarities, wherein the category set includes multiple target categories, and similarities between features of the multiple target categories and features of the first image are all greater than or equal to a preset threshold.

4. The method according to any one of claims 1 to 3, characterized in that The method further comprises: Based on the first semantic segmentation result, obtaining a first mask image and a second mask image, wherein the first mask image includes only an image of a target object indicated in the first semantic segmentation result, and the second mask image includes only an image in the first semantic segmentation result excluding the target object, where the target object is an object of any category in the first semantic segmentation result; Inputting the first mask image, the second mask image, and the category text corresponding to the target object into the multimodal model, obtaining a first similarity between features of the first mask image and features of the category text, and a second similarity between features of the first mask image and features of the second mask image; The step of training the semantic segmentation model according to the first loss function includes: The semantic segmentation model is trained according to the first loss function and a second loss function, wherein the second loss function is obtained based on the first similarity and the second similarity.

5. The method according to claim 4, characterized in that The first similarity has a negative correlation with the second loss function, and the second similarity has a positive correlation with the second loss function.

6. The method according to claim 1, characterized in that The method further comprises: Inputting the first image into a denoising model to obtain a second semantic segmentation result, wherein the structure of the denoising model is the same as that of the semantic segmentation model, and network parameters of the denoising model are different from network parameters of the semantic segmentation model; The step of training the semantic segmentation model according to the first loss function includes: The semantic segmentation model is trained according to the first loss function and a third loss function, where the third loss function is obtained based on the first semantic segmentation result and the second semantic segmentation result.

7. The method according to claim 4, characterized in that The method further comprises: Inputting the first image into a denoising model to obtain a second semantic segmentation result, wherein the structure of the denoising model is the same as that of the semantic segmentation model, and network parameters of the denoising model are different from network parameters of the semantic segmentation model; The training of the semantic segmentation model according to the first loss function and the second loss function includes: The semantic segmentation model is trained according to the first loss function, the second loss function and a third loss function, where the third loss function is obtained based on the first semantic segmentation result and the second semantic segmentation result.

8. The method according to claim 6 or 7, characterized in that The network parameters of the denoising model are obtained based on the network parameters of the semantic segmentation model and the network parameters of the denoising model in the previous round of iterative training.

9. A semantic segmentation method, characterized in that include: Get the image to be processed; Inputting the image to be processed into a semantic segmentation model to obtain a first semantic segmentation result; In which, the semantic segmentation model is trained based on a first loss function, the first loss function is obtained based on the output result and the second semantic segmentation result, the second semantic segmentation result is obtained by inputting the first image in the training data set into the semantic segmentation result model, the output result is obtained by inputting multiple image regions and multiple texts describing a category set into a multimodal model, the output result is used to indicate that each of the multiple image regions corresponds to a category described by a text in the category set, the multimodal model is used to perform image and text matching, the multiple image regions are image regions in the first image corresponding to multiple feature regions, the multiple feature regions are obtained by clustering the feature map of the first image according to the category set of the first image, the category set is used to indicate the categories of multiple objects in the first image, and each feature region in the multiple feature regions corresponds to a category in the category set.

10. The method according to claim 9, characterized in that The multimodal model is used to extract a plurality of image features corresponding to the plurality of image regions and a plurality of text features corresponding to the plurality of texts, and calculate a similarity between each of the plurality of image features and the plurality of text features to obtain a category corresponding to each of the plurality of image regions; Among them, the category corresponding to the first image area is the category corresponding to the text feature with the highest similarity to the first image feature, the first image area is any one of the multiple image areas, and the first image feature is the image feature corresponding to the first image area among the multiple image features.

11. The method according to claim 9, characterized in that The training dataset includes a plurality of images and a plurality of category labels, the plurality of images including the first image, and the plurality of category labels are used to indicate categories of objects in the plurality of images; The category set of the first image is obtained based on multiple similarities, the category set includes multiple target categories, and the similarities between the features of the multiple target categories and the features of the first image are all greater than or equal to a preset threshold. The multiple similarities are multiple similarities between the features of the first image and the features of each category label in the multiple category labels. The multiple similarities are obtained after the first image and the multiple category labels are input into the multimodal model.

12. The method according to any one of claims 9 to 11, characterized in that: The semantic segmentation model is trained based on the first loss function and the second loss function, and the second loss function is obtained based on the first similarity and the second similarity. The first similarity is the similarity between the features of the first mask image and the features of the category text, and the second similarity is the similarity between the features of the first mask image and the features of the second mask image. The first similarity and the second similarity are obtained by inputting the first mask image, the second mask image and the category text corresponding to the target object into the multimodal model. The first mask image and the second mask image are obtained based on the second semantic segmentation result. The first mask image only includes the image of the target object indicated in the second semantic segmentation result, and the second mask image only includes the image other than the target object in the second semantic segmentation result. The target object is an object of any category in the first semantic segmentation result.

13. The method according to claim 12, characterized in that The first similarity has a negative correlation with the second loss function, and the second similarity has a positive correlation with the second loss function.

14. The method according to claim 9, characterized in that The semantic segmentation model is trained based on the first loss function and the third loss function, the third loss function is obtained based on the second semantic segmentation result and the third semantic segmentation result, the third semantic segmentation result is obtained after inputting the first image into a denoising model, the structure of the denoising model is the same as that of the semantic segmentation model, and the network parameters of the denoising model are different from the network parameters of the semantic segmentation model.

15. The method according to claim 12, characterized in that The semantic segmentation model is trained based on the first loss function, the second loss function and the third loss function. The third loss function is obtained based on the second semantic segmentation result and the third semantic segmentation result. The third semantic segmentation result is obtained after inputting the first image into a denoising model. The structure of the denoising model is the same as that of the semantic segmentation model, and the network parameters of the denoising model are different from the network parameters of the semantic segmentation model.

16. The method according to claim 14 or 15, characterized in that The network parameters of the denoising model are obtained based on the network parameters of the semantic segmentation model and the network parameters of the denoising model in the previous round of iterative training.

17. A training device for an unsupervised semantic segmentation model, characterized in that: include: an acquisition module, configured to acquire a feature map of a first image in a training data set, and cluster the feature map according to a category set of the first image to obtain a plurality of feature regions in the feature map, wherein the category set is used to indicate categories of a plurality of objects in the first image, and each feature region in the plurality of feature regions corresponds to a category in the category set; a processing module, configured to determine a plurality of image regions in the first image corresponding to the plurality of feature regions; The processing module is further configured to input the plurality of image regions and the plurality of texts describing the category set into a multimodal model to obtain an output result, the output result being configured to indicate that each of the plurality of image regions corresponds to a category described by a text in the category set, wherein the multimodal model is configured to perform image and text matching; The processing module is further configured to input the first image into a semantic segmentation model to obtain a first semantic segmentation result; A training module is used to train the semantic segmentation model according to a first loss function to update the semantic segmentation model, wherein the first loss function is constructed based on the output result and the first semantic segmentation result.

18. The device according to claim 17, characterized in that The multimodal model is used to extract a plurality of image features corresponding to the plurality of image regions and a plurality of text features corresponding to the plurality of texts, and calculate a similarity between each of the plurality of image features and the plurality of text features to obtain a category corresponding to each of the plurality of image regions; Among them, the category corresponding to the first image area is the category corresponding to the text feature with the highest similarity to the first image feature, the first image area is any one of the multiple image areas, and the first image feature is the image feature corresponding to the first image area among the multiple image features.

19. The device according to claim 17, characterized in that The training dataset includes a plurality of images and a plurality of category labels, the plurality of images including the first image, and the plurality of category labels are used to indicate categories of objects in the plurality of images; The processing module is further configured to: input the first image and the multiple category labels into the multimodal model to obtain multiple similarities between features of the first image and features of each of the multiple category labels; and determine a category set of the first image based on the multiple similarities, wherein the category set includes multiple target categories, and the similarities between the features of the multiple target categories and the features of the first image are all greater than or equal to a preset threshold.

20. The device according to any one of claims 17 to 19, characterized in that The processing module is further configured to obtain a first mask image and a second mask image based on the first semantic segmentation result, wherein the first mask image includes only an image of a target object indicated in the first semantic segmentation result, and the second mask image includes only an image in the first semantic segmentation result excluding the target object, where the target object is an object of any category in the first semantic segmentation result; The processing module is further configured to input the first mask image, the second mask image, and the category text corresponding to the target object into the multimodal model, and obtain a first similarity between features of the first mask image and features of the category text, and a second similarity between features of the first mask image and features of the second mask image; The training module is specifically used to train the semantic segmentation model according to the first loss function and the second loss function, wherein the second loss function is obtained based on the first similarity and the second similarity.

21. The device according to claim 20, characterized in that The first similarity has a negative correlation with the second loss function, and the second similarity has a positive correlation with the second loss function.

22. The device according to claim 17, characterized in that The processing module is further configured to input the first image into a denoising model to obtain a second semantic segmentation result, wherein the structure of the denoising model is the same as that of the semantic segmentation model, and the network parameters of the denoising model are different from the network parameters of the semantic segmentation model; The training module is specifically used to train the semantic segmentation model according to the first loss function and the third loss function, where the third loss function is obtained based on the first semantic segmentation result and the second semantic segmentation result.

23. The device according to claim 20, characterized in that The processing module is further configured to input the first image into a denoising model to obtain a second semantic segmentation result, wherein the structure of the denoising model is the same as that of the semantic segmentation model, and the network parameters of the denoising model are different from the network parameters of the semantic segmentation model; The training module is specifically used to train the semantic segmentation model according to the first loss function, the second loss function and the third loss function, where the third loss function is obtained based on the first semantic segmentation result and the second semantic segmentation result.

24. The device according to claim 22 or 23, characterized in that The network parameters of the denoising model are obtained based on the network parameters of the semantic segmentation model and the network parameters of the denoising model in the previous round of iterative training.

25. A semantic segmentation device, characterized in that: include: An acquisition module, used for acquiring an image to be processed; a processing module, configured to input the image to be processed into a semantic segmentation model to obtain a first semantic segmentation result; In which, the semantic segmentation model is trained based on a first loss function, the first loss function is obtained based on the output result and the second semantic segmentation result, the second semantic segmentation result is obtained by inputting the first image in the training data set into the semantic segmentation result model, the output result is obtained by inputting multiple image regions and multiple texts describing a category set into a multimodal model, the output result is used to indicate that each of the multiple image regions corresponds to a category described by a text in the category set, the multimodal model is used to perform image and text matching, the multiple image regions are image regions in the first image corresponding to multiple feature regions, the multiple feature regions are obtained by clustering the feature map of the first image according to the category set of the first image, the category set is used to indicate the categories of multiple objects in the first image, and each feature region in the multiple feature regions corresponds to a category in the category set.

26. A training device, characterized in that comprising a processor and a memory, the processor being coupled to the memory, The memory is used to store programs; The processor is configured to execute the program in the memory so that the training device performs the method according to any one of claims 1 to 8.

27. An execution device, characterized in that: comprising a processor and a memory, the processor being coupled to the memory, The memory is used to store programs; The processor is configured to execute the program in the memory, so that the execution device executes the method according to any one of claims 9 to 16.

28. A computer program product, characterized in that The method comprises a computer program which, when running on a computer, causes the computer to perform the method according to any one of claims 1 to 16.

29. A computer-readable storage medium, characterized in that The method comprises a program which, when being run on a computer, causes the computer to execute the method according to any one of claims 1 to 16.

Images