Unsupervised semantic segmentation method and system based on saliency map and deep learning

Abstract

The invention relates to the technical field of image processing, in particular to an unsupervised semantic segmentation method and system based on a saliency map and deep learning. According to the method, the clear and corrected correction foreground mask is generated through unsupervised representation learning in combination with the pixel-level target attention classification head and the saliency map without depending on a strong supervision or weak supervision implementation mode under the condition of not using annotation data, so that the contour of the correction foreground mask is clearer, and the accuracy of the correction foreground mask is improved. And thus, unsupervised semantic segmentation is realized. According to the method, the quality of pseudo labels and the accuracy of semantic segmentation are improved by correcting the foreground mask, the dependence of the system on manual annotation is effectively reduced, meanwhile, the method can serve as an automatic annotation system to automatically annotate an image data set, and the labor cost of data annotation is remarkably reduced.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular, to an unsupervised semantic segmentation method and system based on saliency maps and deep learning. Background Art

[0002] Semantic segmentation is a current research hotspot in the field of computer vision. Its goal is to assign each pixel in an image to a specific semantic category, such as an object or background. Semantic segmentation has important application value in many application fields such as autonomous driving, intelligent monitoring, and medical image processing. However, in real-world scenarios, the acquisition of large-scale annotated data often faces huge challenges. Especially for complex scenes, manual annotation is time-consuming and costly, and some data sets are difficult to be annotated manually due to privacy or security reasons.

[0003] At present, the relevant technologies of semantic segmentation are all supervised or weakly supervised, and these methods rely on a large amount of strongly or weakly labeled data. For example, some semantic segmentation methods that require saliency maps as prior information, because they need labeled saliency maps to participate in training, all these methods are actually weakly supervised tasks, which limits their application in real-world scenarios or unlabeled scenarios. Summary of the invention

[0004] The present application provides an unsupervised semantic segmentation method and system based on saliency maps and deep learning to solve one or more technical problems existing in the prior art, and at least provide a beneficial choice or create conditions.

[0005] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by the practice of the present application.

[0006] According to one aspect of an embodiment of the present application, an unsupervised semantic segmentation method based on a saliency map and deep learning is provided, the method comprising:

[0007] Selecting an initial deep learning model and designing a self-supervised loss function for unsupervised representation learning for the deep learning model;

[0008] Perform unsupervised representation learning on the deep learning model according to the self-supervised loss function and the image dataset to be trained, and construct an initial semantic feature network;

[0009] Extracting features from the image dataset based on the initial semantic feature network to obtain a pixel-level feature set for each training image in the image dataset;

[0010] Generate a foreground mask for each of the training images according to a preset target attention classification head and each of the pixel-level feature sets;

[0011] For each training picture, obtaining a saliency map of the training picture, and generating a corrected foreground mask according to the saliency map and the foreground mask;

[0012] Generating an image-level feature set for each of the training images according to each of the pixel-level feature sets and each of the corrected foreground masks;

[0013] Clustering each of the image-level feature sets to obtain a feature set including multiple categories;

[0014] Generate a category map for each of the training pictures according to each of the image-level feature sets and the feature set;

[0015] Generating pseudo labels for each of the training images based on each of the category maps and each of the corrected foreground masks;

[0016] Training the initial semantic feature network according to each of the pseudo labels and preset segmentation head parameters to obtain a target semantic feature network;

[0017] Semantic segmentation is performed on each of the training images according to the target semantic feature network to obtain a semantic segmentation map of each of the training images.

[0018] In one embodiment of the present application, based on the above scheme, the self-supervised loss function is obtained by the following steps:

[0019] Design the first loss function L align , the first loss function L align It is reflected by the following formula:

[0020]

[0021] Where N is the number of training images, x i is the i-th training picture in the image dataset, f is the feature extractor, f(x i ) is the training image x i The corresponding feature, T(x i ) is the training image x i Training images after applying a preset type of image augmentation transformation;

[0022] Design the second loss function L DS , the second loss function L DS It is reflected by the following formula:

[0023]

[0024] Where N is the number of training images, K is the number of intermediate layers of the deep learning model, and x i is the i-th training image in the image dataset, f is the feature extractor, and f k (x i ) is the training image x i The feature of the kth layer in the middle layer, T(x i ) is the training image x i Training images after applying a preset type of image augmentation transformation;

[0025] According to the first loss function L dlign And the second loss function L DS Determine the self-supervised loss function L, which is embodied by the following formula:

[0026] L=λ1L align +λ2L DS

[0027] In the formula, λ1 is a preset first weight coefficient, and λ2 is a preset second weight coefficient.

[0028] In one embodiment of the present application, based on the aforementioned solution, the target attention classification head carries a target activation function, and the foreground mask of each training image is generated according to the preset target attention classification head and each pixel-level feature set, including:

[0029] For each training picture, each pixel-level feature in the pixel-level feature set is input into the target activation function of the target attention classification head to obtain an activation output value of each pixel-level feature;

[0030] For each activation output value, if the activation output value is greater than a preset activation threshold, determining the pixel-level feature corresponding to the activation output value as a target pixel-level feature for generating a foreground mask;

[0031] For each training picture, generating a foreground mask of the training picture according to each of the target pixel-level features;

[0032] Among them, the target attention classification head is adjusted by the self-supervised loss function.

[0033] In one embodiment of the present application, based on the above-mentioned solution, the step of obtaining the saliency map of the training image and generating a corrected foreground mask according to the saliency map and the foreground mask includes:

[0034] Dividing the training image into regions according to a preset image segmentation method to obtain the saliency map consisting of a plurality of saliency regions, each of the saliency regions including a plurality of similar pixel-level features, and each of the pixel-level features corresponding to a single pixel point;

[0035] Calculating a significance value of each of the significance regions;

[0036] For each salient region, if the salient value is lower than a preset salient threshold, determining that the pixel point corresponding to the salient region is a background pixel point;

[0037] The pixel-level features corresponding to the background pixels are excluded from the foreground mask to generate the corrected foreground mask.

[0038] In one embodiment of the present application, based on the above-mentioned solution, generating an image-level feature set of each training picture according to each pixel-level feature set and each corrected foreground mask includes:

[0039] For each training picture, the pixel-level feature weights of the pixel-level feature set are generated according to the corrected foreground mask, and the pixel-level features of each background pixel in the pixel-level feature set are removed based on the pixel-level feature weights and global average pooling is performed to obtain the image-level feature set of the training picture.

[0040] In one embodiment of the present application, based on the above-mentioned solution, generating a category map of each training picture according to each of the image-level feature sets and the feature set includes:

[0041] For each image-level feature set, each pixel-level feature in the image-level feature set is subjected to cosine similarity calculation with the corresponding feature set to obtain a background value corresponding to each pixel point, wherein each pixel-level feature corresponds to one feature set;

[0042] For each training picture, the category map is generated according to each background value.

[0043] In one embodiment of the present application, based on the above-mentioned solution, generating pseudo labels for each of the training images based on each of the category maps and each of the corrected foreground masks includes:

[0044] For each training image, pixels whose background values ​​in the category map are lower than a preset background threshold are removed to generate a target category map, and a pseudo label of the training image is generated based on the target category map and the corrected foreground mask.

[0045] According to one aspect of an embodiment of the present application, an unsupervised semantic segmentation system based on a saliency map and deep learning is provided, the system comprising:

[0046] A design unit, used for selecting an initial deep learning model and designing a self-supervised loss function for unsupervised representation learning for the deep learning model;

[0047] An unsupervised learning unit, used for performing unsupervised representation learning on the deep learning model according to the self-supervised loss function and the image data set to be trained, and constructing an initial semantic feature network;

[0048] A feature extraction unit, configured to extract features from the image dataset based on the initial semantic feature network to obtain a pixel-level feature set for each training image in the image dataset;

[0049] A foreground mask unit, used to generate a foreground mask for each of the training images according to a preset target attention classification head and each of the pixel-level feature sets;

[0050] A correction unit, configured to obtain, for each training picture, a saliency map of the training picture, and generate a corrected foreground mask according to the saliency map and the foreground mask;

[0051] An image feature generating unit, configured to generate an image-level feature set for each of the training images according to each of the pixel-level feature sets and each of the corrected foreground masks;

[0052] A clustering unit, used for clustering each of the image-level feature sets to obtain a feature set including multiple categories;

[0053] Generate a category map for each of the training pictures according to each of the image-level feature sets and the feature set;

[0054] A pseudo label generating unit, configured to generate a pseudo label for each of the training images based on each of the category maps and each of the corrected foreground masks;

[0055] A training unit, used for training the initial semantic feature network according to each of the pseudo labels and preset segmentation head parameters to obtain a target semantic feature network;

[0056] The semantic segmentation unit is used to perform semantic segmentation on each of the training images according to the target semantic feature network to obtain a semantic segmentation map of each of the training images.

[0057] Beneficial effects of the present application: The present application does not need to rely on strong supervision or weak supervision. Without using labeled data, the present application generates a clear and corrected foreground mask through unsupervised representation learning, combined with a pixel-level target attention classification head and a saliency map, so that the contour of the corrected foreground mask is clearer, thereby achieving unsupervised semantic segmentation. While improving the quality of pseudo-labels and the accuracy of semantic segmentation by correcting the foreground mask, the system's dependence on manual annotation is effectively reduced. At the same time, it can also be used as an automatic annotation system to automatically annotate image data sets, significantly reducing the labor cost of data annotation.

[0058] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present application, and together with the specification, are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0060] Figure 1 A flowchart of an unsupervised semantic segmentation method based on a saliency map and deep learning according to an embodiment of the present application;

[0061] Figure 2 A schematic diagram of generating a foreground mask according to an embodiment of the present application;

[0062] Figure 3 is a flow chart of generating a corrected foreground mask according to an embodiment of the present application;

[0063] Figure 4 A schematic diagram of clustering according to an embodiment of the present application;

[0064] Figure 5 A flowchart of pseudo label generation according to an embodiment of the present application is shown;

[0065] Figure 6 is a schematic diagram of segmentation head parameters according to an embodiment of the present application;

[0066] Figure 7 is an exemplary semantic segmentation diagram shown according to an embodiment of the present application;

[0067] Figure 8 A system architecture diagram of a target semantic feature network according to an embodiment of the present application;

[0068] Fig. 9is a schematic diagram of a foreground mask correction according to an embodiment of the present application;

[0069] Fig.10 A comparison diagram of the semantic segmentation effects of a fusion model in which saliency map detection is added to the baseline model shown in an embodiment of the present application and a single baseline model;

[0070] Fig.11 It is an experimental analysis diagram of a preset significance threshold according to an embodiment of the present application. DETAILED DESCRIPTION

[0071] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this application will be more comprehensive and complete and fully convey the concept of the example embodiments to those skilled in the art.

[0072] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided so as to provide a full understanding of the embodiments of the present application. However, it will be appreciated by those skilled in the art that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, components, devices, steps, etc. can be adopted. In other cases, known methods, devices, realizations or operations are not shown or described in detail to avoid blurring the various aspects of the present application.

[0073] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or micro-control node devices.

[0074] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0075] It should be noted that the "multiple" mentioned in this article refers to two or more. "And / or" describes the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0076] The following is a detailed explanation of the professional terms or terminology mentioned in the embodiments of the present application:

[0077] Saliency map: A saliency map is a map used to represent the importance or significance of each area in an image (i.e., the training image described in this application). Through the saliency map, the most concerned or most significant area in the image can be intuitively identified, which is usually used for tasks such as object detection and foreground extraction. In an image, the foreground usually refers to the person or scenery located in front of the subject or close to the lens position. Then the foreground mask and corrected foreground mask generated by this application refer to the segmentation of the human body, object, animal, etc. close to the subject. This technology can be applied to the environmental perception module in autonomous driving to help the vehicle identify pedestrians, vehicles and other objects on the road, and improve the safety of autonomous driving. It can also be used in video surveillance. This technology can automatically identify and segment foreground targets, such as people, vehicles, etc., thereby improving the intelligence level of monitoring. It can also be used for automatic annotation of image libraries, and semantic annotation of large-scale image data sets in an unsupervised manner, significantly reducing the cost of manual annotation.

[0078] Deep learning: Deep learning is a subfield of machine learning. Its core is to use a deep model composed of multiple layers (usually a neural network composed of neurons) to learn and represent complex patterns in data. Low-dimensional (RGB images are 3-dimensional) image information can be converted into high-dimensional (generally greater than or equal to 128 dimensions) feature information through a deep neural network, which effectively improves the effect of image classification. In this application, deep learning technology is mainly used to build an unsupervised semantic segmentation system based on saliency maps and deep learning, and the feature extraction and segmentation of images are achieved by introducing a deep neural network.

[0079] Unsupervised learning: Unsupervised learning is a common machine learning method that relies on the characteristics of the data itself to perform pattern recognition and classification in the absence of labeled data. The goal of unsupervised learning is to find similarities or regularities between data by exploring the inherent structure of the data. Since it does not require labeled data, it can significantly reduce the cost of manual annotation required for supervised learning, which has strong practical significance.

[0080] Semantic segmentation: Semantic segmentation refers to classifying each pixel in an image to achieve image understanding. The goal of semantic segmentation is to assign each pixel in an image to a specific semantic category, such as object or background, so as to accurately determine the location and shape of each object.

[0081] The implementation details of the technical solution of the embodiment of the present application are described in detail below:

[0082] According to one aspect of the present application, an unsupervised semantic segmentation method based on saliency map and deep learning is provided. Figure 1The flowchart of the unsupervised semantic segmentation method based on saliency map and deep learning according to the embodiment of the present application is shown. The unsupervised semantic segmentation method based on saliency map and deep learning includes at least steps S1 to S11, which are described in detail as follows:

[0083] In step S1, an initial deep learning model is selected, and a self-supervised loss function for unsupervised representation learning is designed for the deep learning model.

[0084] Specifically, by designing a self-supervised loss function for unsupervised representation learning for a deep learning model, the deep learning model can be enabled to train the image dataset to be trained by itself.

[0085] In one embodiment of the present application, the self-supervised loss function is obtained by the following steps:

[0086] Design the first loss function L align , the first loss function L align It is reflected by the following formula:

[0087]

[0088] Where N is the number of training images, x i is the i-th training picture in the image dataset, f is the feature extractor, f(x i ) is the training image x i The corresponding feature, T(x i ) is the training image x i Training images after applying a preset type of image augmentation transformation;

[0089] Design the second loss function L DS , the second loss function L DS It is reflected by the following formula:

[0090]

[0091] Where N is the number of training images, K is the number of intermediate layers of the deep learning model, and x i is the i-th training image in the image dataset, f is the feature extractor, and f k (x i ) is the training image x i The feature of the kth layer in the middle layer, T(x i ) is the training image x i Training images after applying a preset type of image augmentation transformation;

[0092] According to the first loss function L align And the second loss function LDS Determine the self-supervised loss function L, which is embodied by the following formula:

[0093] L=λ1L align +λ2L DS

[0094] In the formula, λ1 is a preset first weight coefficient, and λ2 is a preset second weight coefficient.

[0095] Specifically, this application proposes two unsupervised loss functions (i.e., the first loss function and the second loss function) to perform unsupervised representation learning. The first loss is a self-supervised contrast loss, which promotes the consistency of representation by minimizing the distance between features of different views (e.g., different image enhancement transformations). That is, when training a certain training image, if the training image x i Rotate or scale, then the first loss function can make the extracted feature f(T(x i )) and the initially extracted feature f(x i ), that is, the loss function encourages the deep learning model to learn features that are insensitive to input changes (i.e., no matter what kind of transformation enhancement is performed on the training image, the proposed features are highly similar). Among them, the preset type of image enhancement transformation can be specifically rotation, scaling, and expansion of the training image.

[0096] Furthermore, the second loss function is also a self-supervised loss. This loss function promotes the effective learning of the deep learning model by minimizing the difference between deep features and shallow features. The deep features are the features extracted by each intermediate layer of the deep learning model for the training images, and the shallow features are the features initially extracted from the training images, that is, f(x i ). The purpose of the second loss function is to ensure that the deep features remain consistent with the shallow features during the learning process, thereby enhancing the learning ability and generalization ability of the deep learning model. In the training phase, the deep learning model is trained by using the above first loss function and the second loss function together. The preset first weight coefficient and the preset second weight coefficient can be set as needed and are not limited here.

[0097] In step S2, unsupervised representation learning is performed on the deep learning model according to the self-supervised loss function and the image data set to be trained to construct an initial semantic feature network.

[0098] Specifically, in the training phase, the deep learning model is trained by using the above first loss function and the second loss function together, and the preset first weight coefficient and the preset second weight coefficient can be set as needed, which is not limited here. After the training is completed, the initial semantic segmentation feature network is obtained.

[0099] In step S3, feature extraction is performed on the image dataset based on the initial semantic feature network to obtain a pixel-level feature set of each training image in the image dataset.

[0100] Specifically, the embodiment of the present application acquires pixel-level features for each pixel of a training image based on a pixel-level attention classification head. The pixel-level features of each pixel constitute a pixel-level feature set, that is, each training image corresponds to a pixel-level feature set.

[0101] In step S4, a foreground mask of each training image is generated according to a preset target attention classification head and each pixel-level feature set.

[0102] Specifically, see Figure 2 As shown, Figure 2 Schematic diagram of the generation of foreground mask. Fig. 9 To correct the schematic diagram of the foreground mask, the preset target attention classification head can be obtained by:

[0103] First, through the above unsupervised representation learning, a network with certain semantic features (i.e., the initial semantic feature network) is obtained. Then, the initial semantic feature network learned in the first step is fixed, and an attention classification head with a Sigmoid activation function (i.e., the target activation function described in this application) is inserted after the network.

[0104] In one embodiment of the present application, the target attention classification head carries a target activation function, and the foreground mask of each training image is generated according to the preset target attention classification head and each pixel-level feature set, including:

[0105] For each training picture, each pixel-level feature in the pixel-level feature set is input into the target activation function of the target attention classification head to obtain an activation output value of each pixel-level feature;

[0106] For each activation output value, if the activation output value is greater than a preset activation threshold, determining the pixel-level feature corresponding to the activation output value as a target pixel-level feature for generating a foreground mask;

[0107] For each training picture, generating a foreground mask of the training picture according to each of the target pixel-level features;

[0108] Among them, the target attention classification head is adjusted by the self-supervised loss function.

[0109] The image-level features are obtained through the target attention classification head and a global average pooling layer (the image-level features are composed of the pixel-level features of the pixels judged as foreground classes). The formula of the Sigmoid activation function is as follows:

[0110]

[0111] Among them, x is the input value (i.e. a certain pixel of the training image), σ(x) is the final output activation value. It is easy to get from the formula that the range of the activation function is σ(x)∈(0,1), so the output of the attention classification head and the Sigmoid activation function is equivalent to a probability, which represents whether the pixel is a foreground pixel or a background pixel. The larger the activation value, the greater the probability that the pixel is a foreground pixel, and vice versa. The greater the probability that it is a background pixel. Therefore, by setting a preset activation threshold, it can be determined whether each pixel is a foreground pixel or a background pixel. The preset activation threshold can be set to any value as needed, and is not limited here.

[0112] Next, fine-tune the attention head. Use the self-supervised loss function L to fine-tune the attention classification head to learn the shape information of the foreground of the training image. Because pixels with more useful semantic information and their pixel features will be more conducive to the convergence of unsupervised representation learning, the trained target attention classification head is equivalent to a foreground extractor. Then the target pixel-level features are composed of the pixel-level features of each foreground pixel, so the foreground mask can be generated based on the pixel-level features of these foreground pixels.

[0113] In step S5, for each training picture, a saliency map of the training picture is obtained, and a corrected foreground mask is generated according to the saliency map and the foreground mask.

[0114] In one embodiment of the present application, the acquiring the saliency map of the training image and generating the corrected foreground mask according to the saliency map and the foreground mask includes:

[0115] Dividing the training image into regions according to a preset image segmentation method to obtain the saliency map consisting of a plurality of saliency regions, each of the saliency regions including a plurality of similar pixel-level features, and each of the pixel-level features corresponding to a single pixel point;

[0116] Calculating a significance value of each of the significance regions;

[0117] For each salient region, if the salient value is lower than a preset salient threshold, determining that the pixel point corresponding to the salient region is a background pixel point;

[0118] The pixel-level features corresponding to the background pixels are excluded from the foreground mask to generate the corrected foreground mask.

[0119] Specifically, see Figure 3 As shown, Figure 3 The following is a flow chart for the generation of the corrected foreground mask. Fix the target attention classification head and extract the foreground mask of each training image. Next, generate the saliency map of each training image. The steps for generating the saliency map are as follows:

[0120] First, the training image is divided into regions using a graph-based segmentation method to obtain each salient region after segmentation. Each salient region represents a group of similar pixels. A color histogram is constructed for each salient region. The histogram is a probability density function based on color. The saliency of each salient region is calculated by the color contrast, region distance, and region size between it and all other salient regions in the training image. The calculation formula is as follows:

[0121]

[0122] Among them, S(r k ) is the significant region r k The significance value, r k ≠r i Represents the significant region r k With all other significant regions r i Comparison of D s (r k ,r i ) represents the significant region r k and the significant region r i The spatial distance between them is expressed as the Euclidean distance between the centroids of the regions, s s is a parameter that controls the strength of the spatial distance weight, w(r i ) is the significant region r i The weight of the salient region (expressed as the number of pixels in the salient region to emphasize the contrast of larger regions, D r (r k ,r i ) is the color distance metric between two salient regions, calculated based on the weighted color distance of the color histogram in the salient region. The formula is as follows:

[0123]

[0124] Among them, n1 and n2 represent the number of different colors in the salient regions r1 and r2 respectively, and f(c 1,i ) represents the i-th color c in region r1 1,i The frequency of occurrence in the salient region (i.e., the value of the color histogram), D(c 1,i ,c 2,j ) represents the i-th color c in the salient region r1 1,i and the jth color c in region r2 2,j The color distance between them is calculated using the Euclidean distance in the general RGB color space.

[0125] Finally, the pixels in the foreground mask whose significance values ​​are lower than the preset significance threshold are marked as background pixels. The corrected foreground mask can be obtained by removing the pixel-level features of the background pixels and retaining the pixel-level features of the remaining foreground pixels.

[0126] In step S6, an image-level feature set of each of the training images is generated according to each of the pixel-level feature sets and each of the corrected foreground masks.

[0127] In one embodiment of the present application, generating an image-level feature set for each of the training images according to each of the pixel-level feature sets and each of the corrected foreground masks includes:

[0128] For each training picture, the pixel-level feature weights of the pixel-level feature set are generated according to the corrected foreground mask, and the pixel-level features of each background pixel in the pixel-level feature set are removed based on the pixel-level feature weights and global average pooling is performed to obtain the image-level feature set of the training picture.

[0129] Specifically, after the above processing, a corrected foreground mask is obtained, and the corrected foreground mask is used as the weight of the pixel-level feature to filter out the pixel-level features of background pixels, and then the image-level feature set is obtained through global average pooling. The image-level feature set is composed of the pixel-level features of foreground pixels.

[0130] In step S7, each of the image-level feature sets is clustered to obtain a feature set including multiple categories.

[0131] Specifically, refer to Figure 4 As shown, Figure 4 Figure 1 is a schematic diagram of clustering. By using k-means clustering to cluster each image-level feature, multiple categories of feature sets can be obtained, and feature centers can be constructed based on each feature set.

[0132] In step S8, a category map of each of the training images is generated according to each of the image-level feature sets and the feature set.

[0133] In one embodiment of the present application, generating a category map of each of the training images according to each of the image-level feature sets and the feature set includes:

[0134] For each image-level feature set, each pixel-level feature in the image-level feature set is subjected to cosine similarity calculation with the corresponding feature set to obtain a background value corresponding to each pixel point, wherein each pixel-level feature corresponds to one feature set;

[0135] For each training picture, the category map is generated according to each background value.

[0136] Specifically, see Figure 5 As shown in the figure, the pixel-level features of each training image must go through two branches. The first branch (i.e. the branch where the cosine similarity is located) calculates the cosine similarity with the corresponding feature set, and classifies each pixel into the class that is closest to it to obtain a category map. The second branch is the branch where the foreground mask is corrected. Finally, the pixels in the category map whose foreground values ​​are less than the preset background threshold are set as background pixels to obtain the final pseudo-label.

[0137] In step S9, a pseudo label of each of the training images is generated based on each of the category maps and each of the corrected foreground masks.

[0138] Specifically, generating pseudo labels for each of the training images based on each of the category maps and each of the corrected foreground masks includes:

[0139] For each training image, pixels whose background values ​​in the category map are lower than a preset background threshold are removed to generate a target category map, and a pseudo label of the training image is generated based on the target category map and the corrected foreground mask.

[0140] The first branch (i.e. the branch where the cosine similarity is located) calculates the cosine similarity with the corresponding feature set, classifies each pixel into the class that is closest to it, and obtains a category map. The second branch is the branch where the foreground mask is corrected. Finally, the pixels in the category map whose foreground values ​​are less than the preset background threshold are set as background class pixels to obtain the final pseudo-label. Among them, the preset background threshold is a value set arbitrarily according to needs.

[0141] In step S10, the initial semantic feature network is trained according to each of the pseudo labels and preset segmentation head parameters to obtain a target semantic feature network.

[0142] Specifically, the preset segmentation head parameters can be set as needed, see Figure 6As shown in the figure, a segmentation head is added to the deep learning model, and the segmentation head is supervised and trained using the pseudo labels generated above, and fine-tuned using the cross entropy loss to obtain a more accurate target semantic feature network. The cross entropy loss function formula is as follows:

[0143]

[0144] Among them, L CE is the cross entropy loss value, N is the number of categories, y i is the i-th image x i The real label, Is the model to be trained on images x i The predicted probability of class i.

[0145] In step S11, semantic segmentation is performed on each of the training images according to the target semantic feature network to obtain a semantic segmentation map of each of the training images.

[0146] Through the trained target semantic feature network, various training images can be directly segmented, such as Figure 7 As shown, Figure 7 The following are semantic segmentation maps after semantic segmentation of some exemplary training images according to the target semantic feature network.

[0147] like Fig.10 As shown, an ablation analysis is performed on the corrected foreground mask generated based on the saliency map proposed in the embodiment of the present application. Fig.10 It is proved that after adding the proposed correction algorithm to the baseline model (i.e., the corrected foreground mask obtained by the correction algorithm), the mIoU (semantic segmentation index) of the final segmentation map is improved by 2.32%; the larger the mIoU, the better the semantic segmentation effect, which qualitatively proves the effectiveness of the corrected foreground mask generated based on the saliency map. The comparison results are shown in Fig.10 shown.

[0148] This application also compares and analyzes other saliency maps and foreground masks, proving that the best combination is to set the pixel positions with saliency scores below the threshold 1 in the foreground mask as background. Fig.11 The threshold value 1 obtained by this comparative analysis method can be used as the significance threshold described in this application, and the semantic segmentation effect of the corrected foreground mask obtained thereby is excellent.

[0149] Finally, see Figure 8 As shown, Figure 8The overall system architecture of the target semantic feature network of this application. This application does not need to rely on strong supervision or weak supervision. Without using labeled data, it generates a clear and corrected foreground mask through unsupervised representation learning, combined with a pixel-level target attention classification head and a saliency map, so that the contour of the corrected foreground mask is clearer, thereby achieving unsupervised semantic segmentation. While improving the quality of pseudo-labels and the accuracy of semantic segmentation by correcting the foreground mask, it effectively reduces the system's dependence on manual annotation. At the same time, it can also be used as an automatic annotation system to automatically annotate image data sets, significantly reducing the labor cost of data annotation.

[0150] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present application, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0151] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be performed without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. An unsupervised semantic segmentation method based on saliency map and deep learning, characterized in that: The method comprises: Selecting an initial deep learning model and designing a self-supervised loss function for unsupervised representation learning for the deep learning model; Perform unsupervised representation learning on the deep learning model according to the self-supervised loss function and the image dataset to be trained, and construct an initial semantic feature network; Extracting features from the image dataset based on the initial semantic feature network to obtain a pixel-level feature set for each training image in the image dataset; Generate a foreground mask for each of the training images according to a preset target attention classification head and each of the pixel-level feature sets; For each training picture, obtaining a saliency map of the training picture, and generating a corrected foreground mask according to the saliency map and the foreground mask; Generating an image-level feature set for each of the training images according to each of the pixel-level feature sets and each of the corrected foreground masks; Clustering each of the image-level feature sets to obtain a feature set including multiple categories; Generate a category map for each of the training pictures according to each of the image-level feature sets and the feature set; Generating pseudo labels for each of the training images based on each of the category maps and each of the corrected foreground masks; Training the initial semantic feature network according to each of the pseudo labels and preset segmentation head parameters to obtain a target semantic feature network; Semantic segmentation is performed on each of the training images according to the target semantic feature network to obtain a semantic segmentation map of each of the training images.

2. The unsupervised semantic segmentation method based on saliency map and deep learning according to claim 1, characterized in that: The self-supervised loss function is obtained by the following steps: Design the first loss function L align , the first loss function L align It is reflected by the following formula: Where N is the number of training images, x i is the i-th training picture in the image dataset, f is the feature extractor, f(x i ) is the training image x i The corresponding feature, T(x i ) is the training image x i Training images after applying a preset type of image enhancement transformation; Design the second loss function L DS , the second loss function L DS It is reflected by the following formula: Where N is the number of training images, K is the number of intermediate layers of the deep learning model, and x i is the i-th training image in the image dataset, f is the feature extractor, and f k (x i ) is the training image x i The feature of the kth layer in the middle layer, T(x i ) is the training image x i Training images after applying a preset type of image enhancement transformation; According to the first loss function L align And the second loss function L DS Determine the self-supervised loss function L, which is embodied by the following formula: L=λ1L align +λ2L DS In the formula, λ1 is a preset first weight coefficient, and λ2 is a preset second weight coefficient.

3. The unsupervised semantic segmentation method based on saliency map and deep learning according to claim 2, characterized in that: The target attention classification head carries a target activation function, and the foreground mask of each training image is generated according to the preset target attention classification head and each pixel-level feature set, including: For each training picture, each pixel-level feature in the pixel-level feature set is input into the target activation function of the target attention classification head to obtain an activation output value of each pixel-level feature; For each activation output value, if the activation output value is greater than a preset activation threshold, determining the pixel-level feature corresponding to the activation output value as a target pixel-level feature for generating a foreground mask; For each training picture, generating a foreground mask of the training picture according to each of the target pixel-level features; Among them, the target attention classification head is adjusted by the self-supervised loss function.

4. The unsupervised semantic segmentation method based on saliency map and deep learning according to claim 3, characterized in that: The obtaining of the saliency map of the training image and generating a corrected foreground mask according to the saliency map and the foreground mask comprises: Dividing the training image into regions according to a preset image segmentation method to obtain the saliency map consisting of a plurality of saliency regions, each of the saliency regions including a plurality of similar pixel-level features, and each of the pixel-level features corresponding to a single pixel point; Calculating a significance value of each of the significance regions; For each salient region, if the salient value is lower than a preset salient threshold, determining that the pixel point corresponding to the salient region is a background pixel point; The pixel-level features corresponding to the background pixels are excluded from the foreground mask to generate the corrected foreground mask.

5. The unsupervised semantic segmentation method based on saliency map and deep learning according to claim 4, characterized in that: The step of generating an image-level feature set for each of the training images according to each of the pixel-level feature sets and each of the corrected foreground masks comprises: For each training picture, the pixel-level feature weights of the pixel-level feature set are generated according to the corrected foreground mask, and the pixel-level features of each background pixel in the pixel-level feature set are removed based on the pixel-level feature weights and global average pooling is performed to obtain the image-level feature set of the training picture.

6. The unsupervised semantic segmentation method based on saliency map and deep learning according to claim 5, characterized in that: Generating a category map of each of the training pictures according to each of the image-level feature sets and the feature set includes: For each image-level feature set, each pixel-level feature in the image-level feature set is subjected to cosine similarity calculation with the corresponding feature set to obtain a background value corresponding to each pixel point, wherein each pixel-level feature corresponds to one feature set; For each training picture, the category map is generated according to each background value.

7. The unsupervised semantic segmentation method based on saliency map and deep learning according to claim 6, characterized in that: The generating of pseudo labels for each of the training images based on each of the category maps and each of the corrected foreground masks comprises: For each training image, pixels whose background values ​​in the category map are lower than a preset background threshold are removed to generate a target category map, and a pseudo label of the training image is generated based on the target category map and the corrected foreground mask.

8. An unsupervised semantic segmentation system based on saliency maps and deep learning, characterized in that: The system comprises: A design unit, used for selecting an initial deep learning model and designing a self-supervised loss function for unsupervised representation learning for the deep learning model; An unsupervised learning unit, used for performing unsupervised representation learning on the deep learning model according to the self-supervised loss function and the image data set to be trained, and constructing an initial semantic feature network; A feature extraction unit, configured to extract features from the image dataset based on the initial semantic feature network to obtain a pixel-level feature set for each training image in the image dataset; A foreground mask unit, used to generate a foreground mask for each of the training images according to a preset target attention classification head and each of the pixel-level feature sets; A correction unit, configured to obtain, for each training picture, a saliency map of the training picture, and generate a corrected foreground mask according to the saliency map and the foreground mask; An image feature generating unit, configured to generate an image-level feature set for each of the training images according to each of the pixel-level feature sets and each of the corrected foreground masks; A clustering unit, used for clustering each of the image-level feature sets to obtain a feature set including multiple categories; Generate a category map for each of the training pictures according to each of the image-level feature sets and the feature set; A pseudo label generating unit, configured to generate a pseudo label for each of the training images based on each of the category maps and each of the corrected foreground masks; A training unit, used for training the initial semantic feature network according to each of the pseudo labels and preset segmentation head parameters to obtain a target semantic feature network; The semantic segmentation unit is used to perform semantic segmentation on each of the training images according to the target semantic feature network to obtain a semantic segmentation map of each of the training images.

Images