Unsupervised semantic segmentation method and system based on saliency map and deep learning

Abstract

The present application relates to the field of image processing technology, and in particular, to an unsupervised semantic segmentation method and system based on saliency maps and deep learning. The present application does not need to rely on strong or weak supervision implementation methods. Without using labeled data, it generates a clear and corrected foreground mask through unsupervised representation learning, combined with a pixel-level target attention classification head and a saliency map, so that the outline of the corrected foreground mask is clearer, thereby achieving unsupervised semantic segmentation. While improving the quality of pseudo-labels and the accuracy of semantic segmentation by correcting the foreground mask, it effectively reduces the system's dependence on manual annotation. At the same time, it can also be used as an automatic annotation system to automatically annotate image data sets, significantly reducing the labor cost of data annotation.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular, to an unsupervised semantic segmentation method and system based on saliency maps and deep learning. Background Art

[0002] Semantic segmentation is a current research hotspot in computer vision. Its goal is to assign each pixel in an image to a specific semantic category, such as object or background. Semantic segmentation holds significant value in numerous application areas, including autonomous driving, intelligent surveillance, and medical image processing. However, in real-world scenarios, acquiring large-scale annotated data often presents significant challenges. Manual annotation is particularly time-consuming and costly for complex scenes, and some datasets are difficult to manually annotate for privacy or security reasons.

[0003] Currently, semantic segmentation techniques are all performed in a supervised or weakly supervised manner. These methods rely on large amounts of strongly or weakly labeled data. For example, some semantic segmentation methods require saliency maps as prior information. However, since they require well-labeled saliency maps for training, all of these methods are actually weakly supervised tasks, limiting their application in real-world or unlabeled scenarios. Summary of the Invention

[0004] This application provides an unsupervised semantic segmentation method and system based on saliency maps and deep learning to solve one or more technical problems existing in the prior art and at least provide a beneficial option or create conditions.

[0005] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by practice of the present application.

[0006] According to one aspect of an embodiment of the present application, an unsupervised semantic segmentation method based on a saliency map and deep learning is provided, the method comprising:

[0007] Selecting an initial deep learning model and designing a self-supervised loss function for unsupervised representation learning for the deep learning model;

[0008] Performing unsupervised representation learning on the deep learning model according to the self-supervised loss function and the image dataset to be trained to construct an initial semantic feature network;

[0009] performing feature extraction on the image dataset based on the initial semantic feature network to obtain a pixel-level feature set for each training image in the image dataset;

[0010] Generate a foreground mask for each of the training images according to a preset target attention classification head and each of the pixel-level feature sets;

[0011] For each training image, obtaining a saliency map of the training image, and generating a corrected foreground mask according to the saliency map and the foreground mask;

[0012] Generating an image-level feature set for each of the training images according to each of the pixel-level feature sets and each of the corrected foreground masks;

[0013] Clustering each of the image-level feature sets to obtain a feature set including multiple categories;

[0014] Generating a category map for each of the training images according to each of the image-level feature sets and the feature set;

[0015] generating pseudo labels for each of the training images based on each of the category maps and each of the corrected foreground masks;

[0016] Training the initial semantic feature network according to each of the pseudo labels and preset segmentation head parameters to obtain a target semantic feature network;

[0017] Semantic segmentation is performed on each of the training images according to the target semantic feature network to obtain a semantic segmentation map of each of the training images.

[0018] In one embodiment of the present application, based on the above scheme, the self-supervised loss function is obtained by the following steps:

[0019] Design the first loss function L align , the first loss function L align It is reflected by the following formula:

[0020]

[0021] Where N is the number of training images, x i is the i-th training picture in the image dataset, f is the feature extractor, f(x i ) is the training image x i The corresponding feature, T(x i ) is the training image x i Training images after applying a preset type of image augmentation transformation;

[0022] Design the second loss function L DS , the second loss function L DS It is reflected by the following formula:

[0023]

[0024] Where N is the number of training images, K is the number of intermediate layers of the deep learning model, and x i is the i-th training picture in the image dataset, f is the feature extractor, f k (x i ) is the training image x i The feature of the kth layer in the middle layer, T(x i ) is the training image x i Training images after applying a preset type of image augmentation transformation;

[0025] According to the first loss function L dlign And the second loss function L DS Determine the self-supervisory loss function L, which is expressed by the following formula:

[0026] L=λ1L align +λ2L DS

[0027] Wherein, λ1 is a preset first weight coefficient, and λ2 is a preset second weight coefficient.

[0028] In one embodiment of the present application, based on the aforementioned solution, the target attention classification head carries a target activation function, and the foreground mask of each training image is generated according to the preset target attention classification head and each pixel-level feature set, including:

[0029] For each training image, input each pixel-level feature in the pixel-level feature set into the target activation function of the target attention classification head to obtain the activation output value of each pixel-level feature;

[0030] For each activation output value, if the activation output value is greater than a preset activation threshold, determining the pixel-level feature corresponding to the activation output value as a target pixel-level feature for generating a foreground mask;

[0031] For each training image, generating a foreground mask of the training image according to each of the target pixel-level features;

[0032] The target attention classification head is adjusted by the self-supervised loss function.

[0033] In one embodiment of the present application, based on the above solution, obtaining the saliency map of the training image and generating a corrected foreground mask according to the saliency map and the foreground mask includes:

[0034] Dividing the training image into regions according to a preset image segmentation method to obtain the saliency map consisting of a plurality of saliency regions, each of the saliency regions including a plurality of similar pixel-level features, each of the pixel-level features corresponding to a single pixel;

[0035] Calculating a significance value of each of the significant regions;

[0036] For each salient region, if the saliency value is lower than a preset saliency threshold, the pixel corresponding to the saliency region is determined to be a background pixel;

[0037] The pixel-level features corresponding to the background pixels are excluded from the foreground mask to generate the corrected foreground mask.

[0038] In one embodiment of the present application, based on the aforementioned solution, generating an image-level feature set for each of the training images according to each of the pixel-level feature sets and each of the corrected foreground masks includes:

[0039] For each training picture, the pixel-level feature weight of the pixel-level feature set is generated according to the corrected foreground mask, and the pixel-level features of each background pixel point in the pixel-level feature set are removed based on the pixel-level feature weight and global average pooling is performed to obtain the image-level feature set of the training picture.

[0040] In one embodiment of the present application, based on the aforementioned solution, generating a category map for each of the training images according to each of the image-level feature sets and the feature set includes:

[0041] For each image-level feature set, performing cosine similarity calculation on each pixel-level feature in the image-level feature set and the corresponding feature set to obtain a background value corresponding to each pixel point, where each pixel-level feature corresponds to one feature set;

[0042] For each training picture, the category map is generated according to each background value.

[0043] In one embodiment of the present application, based on the aforementioned solution, generating pseudo labels for each of the training images based on each of the category maps and each of the corrected foreground masks includes:

[0044] For each training image, pixels whose background values ​​are lower than a preset background threshold in the category map are removed to generate a target category map, and a pseudo label of the training image is generated based on the target category map and the corrected foreground mask.

[0045] According to one aspect of an embodiment of the present application, an unsupervised semantic segmentation system based on a saliency map and deep learning is provided, the system comprising:

[0046] A design unit, configured to select an initial deep learning model and design a self-supervised loss function for unsupervised representation learning for the deep learning model;

[0047] An unsupervised learning unit, configured to perform unsupervised representation learning on the deep learning model based on the self-supervised loss function and the image dataset to be trained, and to construct an initial semantic feature network;

[0048] a feature extraction unit, configured to perform feature extraction on the image dataset based on the initial semantic feature network to obtain a pixel-level feature set for each training image in the image dataset;

[0049] a foreground mask unit, configured to generate a foreground mask for each of the training images based on a preset target attention classification head and each of the pixel-level feature sets;

[0050] a correction unit, configured to obtain, for each training image, a saliency map of the training image, and generate a corrected foreground mask according to the saliency map and the foreground mask;

[0051] An image feature generation unit, configured to generate an image-level feature set for each of the training images based on each of the pixel-level feature sets and each of the corrected foreground masks;

[0052] a clustering unit, configured to cluster each of the image-level feature sets to obtain a feature set comprising multiple categories;

[0053] Generating a category map for each of the training images according to each of the image-level feature sets and the feature set;

[0054] a pseudo label generating unit, configured to generate a pseudo label for each of the training images based on each of the category maps and each of the corrected foreground masks;

[0055] A training unit, configured to train the initial semantic feature network according to each of the pseudo labels and preset segmentation head parameters to obtain a target semantic feature network;

[0056] The semantic segmentation unit is used to perform semantic segmentation on each of the training images according to the target semantic feature network to obtain a semantic segmentation map of each of the training images.

[0057] Beneficial effects of this application: This application does not rely on strong or weak supervision. Without using labeled data, it generates a clear and corrected foreground mask through unsupervised representation learning, combined with a pixel-level target attention classification head and a saliency map, so that the contour of the corrected foreground mask is clearer, thereby achieving unsupervised semantic segmentation. While improving the quality of pseudo-labels and the accuracy of semantic segmentation by correcting the foreground mask, it effectively reduces the system's dependence on manual annotation. At the same time, it can also be used as an automatic annotation system to automatically annotate image datasets, significantly reducing the labor cost of data annotation.

[0058] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The accompanying drawings are incorporated into and constitute a part of the specification, illustrating embodiments consistent with the present application and, together with the specification, explaining the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and those skilled in the art can derive other drawings based on these drawings without inventive effort. In the drawings:

[0060] Figure 1 This is a flowchart of an unsupervised semantic segmentation method based on saliency maps and deep learning according to an embodiment of the present application;

[0061] Figure 2 Schematic diagram of generating a foreground mask according to an embodiment of the present application;

[0062] Figure 3 1 is a flow chart of generating a corrected foreground mask according to an embodiment of the present application;

[0063] Figure 4 is a schematic diagram of clustering according to an embodiment of the present application;

[0064] Figure 5 A flowchart of pseudo-label generation according to an embodiment of the present application is shown;

[0065] Figure 6 Schematic diagram of segmentation head parameters according to an embodiment of the present application;

[0066] Figure 7 is an exemplary semantic segmentation map shown according to an embodiment of the present application;

[0067] Figure 8 1 is a system architecture diagram of a target semantic feature network according to an embodiment of the present application;

[0068] Figure 9is a schematic diagram of a foreground mask correction according to an embodiment of the present application;

[0069] Figure 10 A comparison chart of the semantic segmentation effects of a fusion model that adds saliency map detection to the baseline model shown in an embodiment of the present application and a single baseline model;

[0070] Figure 11 This is an experimental analysis diagram of a preset significance threshold according to an embodiment of the present application. DETAILED DESCRIPTION

[0071] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this application will be thorough and complete and will fully convey the concepts of the example embodiments to those skilled in the art.

[0072] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner.In the following description, many specific details are provided so as to provide a full understanding of the embodiments of the present application. However, it will be appreciated by those skilled in the art that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, components, devices, steps etc. can be adopted. In other cases, known methods, devices, implementations or operations are not shown or described in detail to avoid blurring the various aspects of the application.

[0073] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically separate entities. That is, these functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or micro-controller node devices.

[0074] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, while others may be combined or partially combined. Therefore, the actual execution order may vary depending on the actual situation.

[0075] It should be noted that the term "plurality" used in this document refers to two or more. "And / or" describes a relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A alone, A and B together, or B alone. The character " / " generally indicates an "or" relationship between the associated objects.

[0076] The following is a detailed explanation of the professional terms or terminology mentioned in the embodiments of this application:

[0077] Saliency map: A saliency map is a map used to represent the importance or significance of each area in an image (i.e., the training image described in this application). Through the saliency map, the most popular or most significant area in the image can be intuitively identified, which is usually used for tasks such as object detection and foreground extraction. In an image, the foreground usually refers to the person or scenery located in front of the subject or close to the lens position. Then the foreground mask and the corrected foreground mask generated by this application refer to the segmentation of the human body, object, animal, etc. close to the subject. This technology can be applied to the environmental perception module in autonomous driving to help the vehicle identify pedestrians, vehicles and other objects on the road, thereby improving the safety of autonomous driving. It can also be used in video surveillance. This technology can automatically identify and segment foreground targets, such as people, vehicles, etc., thereby improving the intelligence level of monitoring. It can also be used for automatic annotation of image libraries, and semantic annotation of large-scale image data sets in an unsupervised manner, significantly reducing the cost of manual annotation.

[0078] Deep learning: Deep learning is a subfield of machine learning. Its core is to use deep models composed of multiple layers (usually neural networks composed of neurons) to learn and represent complex patterns in data. Low-dimensional image information (RGB images are 3-dimensional) can be converted into high-dimensional feature information (generally greater than or equal to 128 dimensions) through deep neural networks, effectively improving the effectiveness of image classification. In this application, deep learning technology is mainly used to build an unsupervised semantic segmentation system based on saliency maps and deep learning. By introducing deep neural networks, feature extraction and segmentation of images are achieved.

[0079] Unsupervised learning: Unsupervised learning is a common machine learning method that relies on the inherent characteristics of the data to perform pattern recognition and classification without labeled data. The goal of unsupervised learning is to discover similarities or patterns between data by exploring its inherent structure. Because it doesn't require labeled data, it can significantly reduce the manual annotation costs required for supervised learning, making it highly practical.

[0080] Semantic segmentation: Semantic segmentation refers to classifying each pixel in an image to achieve image understanding. The goal of semantic segmentation is to assign each pixel in an image to a specific semantic category, such as object or background, so as to accurately determine the location and shape of each object.

[0081] The following is a detailed description of the implementation details of the technical solution of the embodiment of the present application:

[0082] According to one aspect of the present application, an unsupervised semantic segmentation method based on saliency map and deep learning is provided. Figure 1This is a flowchart of an unsupervised semantic segmentation method based on a saliency map and deep learning according to an embodiment of the present application. The unsupervised semantic segmentation method based on a saliency map and deep learning includes at least steps S1 to S11, which are described in detail as follows:

[0083] In step S1, an initial deep learning model is selected, and a self-supervised loss function for unsupervised representation learning is designed for the deep learning model.

[0084] Specifically, by designing a self-supervised loss function for unsupervised representation learning for a deep learning model, the deep learning model can be enabled to train the image dataset to be trained.

[0085] In one embodiment of the present application, the self-supervised loss function is obtained by the following steps:

[0086] Design the first loss function L align , the first loss function L align It is reflected by the following formula:

[0087]

[0088] Where N is the number of training images, x i is the i-th training picture in the image dataset, f is the feature extractor, f(x i ) is the training image x i The corresponding feature, T(x i ) is the training image x i Training images after applying a preset type of image augmentation transformation;

[0089] Design the second loss function L DS , the second loss function L DS It is reflected by the following formula:

[0090]

[0091] Where N is the number of training images, K is the number of intermediate layers of the deep learning model, and x i is the i-th training picture in the image dataset, f is the feature extractor, f k (x i ) is the training image x i The feature of the kth layer in the middle layer, T(x i ) is the training image x i Training images after applying a preset type of image augmentation transformation;

[0092] According to the first loss function L align And the second loss function LDS Determine the self-supervisory loss function L, which is expressed by the following formula:

[0093] L=λ1L align +λ2L DS

[0094] Wherein, λ1 is a preset first weight coefficient, and λ2 is a preset second weight coefficient.

[0095] Specifically, this application proposes two unsupervised loss functions (i.e., the first loss function and the second loss function) to perform unsupervised representation learning. The first loss is the self-supervised contrast loss, which promotes the consistency of representation by minimizing the distance between features of different views (e.g., different image enhancement transformations). That is, when training a certain training image, if the training image x i Rotate or scale, then the first loss function can make the extracted feature f(T(x i )) and the initially extracted features f(x i ), that is, this loss function encourages the deep learning model to learn features that are insensitive to input changes (i.e., the proposed features are highly similar regardless of the transformation enhancement performed on the training image). Specifically, the preset type of image enhancement transformation can be rotation, scaling, or expansion of the training image.

[0096] Furthermore, the second loss function is also a self-supervised loss. This loss function promotes the effective learning of the deep learning model by minimizing the difference between deep features and shallow features. Deep features are the features extracted by each intermediate layer of the deep learning model for the training image, and shallow features are the features initially extracted for the training image, i.e., f(x i The purpose of the second loss function is to ensure that the deep features remain consistent with the shallow features during the learning process, thereby enhancing the learning and generalization capabilities of the deep learning model. During the training phase, the deep learning model is trained by using the above first and second loss functions together. The preset first and second weight coefficients can be set as needed and are not limited here.

[0097] In step S2, unsupervised representation learning is performed on the deep learning model according to the self-supervised loss function and the image dataset to be trained to construct an initial semantic feature network.

[0098] Specifically, during the training phase, the deep learning model is trained by using the first loss function and the second loss function. The preset first weight coefficient and the preset second weight coefficient can be set as needed and are not limited here. After the training is completed, the initial semantic segmentation feature network is obtained.

[0099] In step S3, feature extraction is performed on the image dataset based on the initial semantic feature network to obtain a pixel-level feature set for each training image in the image dataset.

[0100] Specifically, the embodiment of the present application is based on a pixel-level attention classification head to obtain pixel-level features for each pixel point of the training image. The pixel-level features of each pixel point constitute a pixel-level feature set, that is, each training image corresponds to a pixel-level feature set.

[0101] In step S4, a foreground mask of each training image is generated according to the preset target attention classification head and each pixel-level feature set.

[0102] Specifically, see Figure 2 As shown, Figure 2 Schematic diagram of the generation of foreground mask. Figure 9 To correct the foreground mask diagram, the preset target attention classification head can be obtained by:

[0103] First, through the above-mentioned unsupervised representation learning, a network with certain semantic features (i.e., the initial semantic feature network) is obtained. Then, the initial semantic feature network learned in the first step is fixed, and an attention classification head with a Sigmoid activation function (i.e., the target activation function described in this application) is inserted after the network.

[0104] In one embodiment of the present application, the target attention classification head carries a target activation function, and the foreground mask of each training image is generated according to the preset target attention classification head and each pixel-level feature set, including:

[0105] For each training image, input each pixel-level feature in the pixel-level feature set into the target activation function of the target attention classification head to obtain the activation output value of each pixel-level feature;

[0106] For each activation output value, if the activation output value is greater than a preset activation threshold, determining the pixel-level feature corresponding to the activation output value as a target pixel-level feature for generating a foreground mask;

[0107] For each training image, generating a foreground mask of the training image according to each of the target pixel-level features;

[0108] The target attention classification head is adjusted by the self-supervised loss function.

[0109] Image-level features are obtained through a target attention classification head and a global average pooling layer (image-level features consist of pixel-level features of pixels that are judged to be foreground classes). The formula of the Sigmoid activation function is as follows:

[0110]

[0111] Here, x is the input value (i.e., a pixel in the training image), and σ(x) is the final output activation value. The formula shows that the range of the activation function is σ(x)∈(0,1). Therefore, the output of the attention classification head and the Sigmoid activation function is equivalent to a probability. This probability represents whether the pixel is a foreground pixel or a background pixel. The larger the activation value, the greater the probability that the pixel is a foreground pixel, and vice versa. The greater the probability that the pixel is a background pixel. Therefore, by setting a preset activation threshold, it is possible to determine whether each pixel is a foreground pixel or a background pixel. The preset activation threshold can be set to any value as needed and is not limited here.

[0112] Next, fine-tune the attention head. Using the self-supervised loss function L, the attention classification head is fine-tuned to learn the shape information of the foreground of the training image. Because pixels and their pixel features with more useful semantic information are more conducive to the convergence of unsupervised representation learning, the trained target attention classification head is equivalent to a foreground extractor. The target pixel-level features are then composed of the pixel-level features of each foreground pixel. From these pixel-level features, a foreground mask can be generated.

[0113] In step S5, for each training image, a saliency map of the training image is obtained, and a corrected foreground mask is generated according to the saliency map and the foreground mask.

[0114] In one embodiment of the present application, obtaining a saliency map of the training image and generating a corrected foreground mask according to the saliency map and the foreground mask includes:

[0115] Dividing the training image into regions according to a preset image segmentation method to obtain the saliency map consisting of a plurality of saliency regions, each of the saliency regions including a plurality of similar pixel-level features, each of the pixel-level features corresponding to a single pixel;

[0116] Calculating a significance value of each of the significant regions;

[0117] For each salient region, if the saliency value is lower than a preset saliency threshold, the pixel corresponding to the saliency region is determined to be a background pixel;

[0118] The pixel-level features corresponding to the background pixels are excluded from the foreground mask to generate the corrected foreground mask.

[0119] Specifically, see Figure 3 As shown, Figure 3 This is a flowchart for generating a corrected foreground mask. After fixing the target attention classification head, extract the foreground mask for each training image. Next, generate a saliency map for each training image. The steps for generating a saliency map are as follows:

[0120] First, a graph-based segmentation method is used to segment the training image into regions. Each salient region after segmentation is obtained. Each salient region represents a group of similar pixels. A color histogram is constructed for each salient region. The histogram is a probability density function based on color. The saliency of each salient region is calculated by comparing its color contrast with all other salient regions in the training image, its region distance, and its region size. The calculation formula is as follows:

[0121]

[0122] Among them, S(r k ) is the significant region r k The significance value, r k ≠r i Represents the significant region r k Compared with all other significant regions r i Comparison of D s (r k ,r i ) represents the significant region r k and the significant region r i The spatial distance between them is expressed as the Euclidean distance between the region centroids, s s is a parameter that controls the strength of spatial distance weight, w(r i ) is the significant region r i The weight of the salient region (expressed as the number of pixels in the salient region to emphasize the contrast of the larger region, D r (r k ,r i ) is the color distance metric between two salient regions, calculated based on the weighted color distance of the color histogram in the salient region. The formula is as follows:

[0123]

[0124] Among them, n1 and n2 represent the number of different colors in the salient regions r1 and r2 respectively, and f(c 1,i ) represents the i-th color c in region r1 1,i The frequency of occurrence in the salient region (i.e., the value of the color histogram), D(c 1,i ,c 2,j ) represents the i-th color c in the salient region r1 1,i and the jth color c in region r2 2,j The color distance between them is calculated using the Euclidean distance in the general RGB color space.

[0125] Finally, the pixels in the foreground mask whose significance values ​​are lower than the preset significance threshold are marked as background pixels. By removing the pixel-level features of the background pixels and retaining the pixel-level features of the remaining foreground pixels, the corrected foreground mask can be obtained.

[0126] In step S6, an image-level feature set of each of the training images is generated according to each of the pixel-level feature sets and each of the corrected foreground masks.

[0127] In one embodiment of the present application, generating an image-level feature set for each of the training images based on each of the pixel-level feature sets and each of the corrected foreground masks includes:

[0128] For each training picture, the pixel-level feature weight of the pixel-level feature set is generated according to the corrected foreground mask, and the pixel-level features of each background pixel point in the pixel-level feature set are removed based on the pixel-level feature weight and global average pooling is performed to obtain the image-level feature set of the training picture.

[0129] Specifically, after the above processing, a corrected foreground mask is obtained, and the corrected foreground mask is used as the weight of the pixel-level feature to filter out the pixel-level features of the background pixels. Then, global average pooling is performed to obtain an image-level feature set, which is composed of the pixel-level features of the foreground pixels.

[0130] In step S7, each of the image-level feature sets is clustered to obtain a feature set including multiple categories.

[0131] Specifically, refer to Figure 4 As shown, Figure 4 Figure 2 is a schematic diagram of clustering. By using k-means clustering to cluster each image-level feature, we can obtain feature sets of multiple categories and construct feature centers based on each feature set.

[0132] In step S8, a category map of each of the training images is generated according to each of the image-level feature sets and the feature set.

[0133] In one embodiment of the present application, generating a category map for each of the training images according to each of the image-level feature sets and the feature set includes:

[0134] For each image-level feature set, performing cosine similarity calculation on each pixel-level feature in the image-level feature set and the corresponding feature set to obtain a background value corresponding to each pixel point, where each pixel-level feature corresponds to one feature set;

[0135] For each training picture, the category map is generated according to each background value.

[0136] Specifically, see Figure 5 As shown, the pixel-level features of each training image pass through two branches. The first branch (the branch where cosine similarity is calculated) performs cosine similarity calculations with the corresponding feature set, classifying each pixel into the class to which it is most similar, thus generating a category map. The second branch is the branch where the foreground mask is corrected. Finally, pixels in the category map whose foreground values ​​are less than a preset background threshold are assigned to the background class, thus generating the final pseudo-labels.

[0137] In step S9, pseudo labels for the training images are generated based on the category maps and the corrected foreground masks.

[0138] Specifically, generating pseudo labels for each of the training images based on each of the category maps and each of the corrected foreground masks includes:

[0139] For each training image, pixels whose background values ​​are lower than a preset background threshold in the category map are removed to generate a target category map, and a pseudo label of the training image is generated based on the target category map and the corrected foreground mask.

[0140] The first branch (the branch where cosine similarity is calculated) calculates cosine similarity with the corresponding feature set, classifying each pixel into the class to which it is most similar, thus generating a class map. The second branch is the branch where the foreground mask is corrected. Finally, pixels in the class map whose foreground value is less than a preset background threshold are assigned to background class pixels, thereby generating the final pseudo-labels. The preset background threshold is an arbitrary value set as needed.

[0141] In step S10, the initial semantic feature network is trained according to each of the pseudo labels and preset segmentation head parameters to obtain a target semantic feature network.

[0142] Specifically, the preset segmentation head parameters can be set as needed, see Figure 6As shown in the figure, a segmentation head is added to the deep learning model, and the segmentation head is supervised and trained using the pseudo labels generated above. The cross entropy loss is used for fine-tuning to obtain a more accurate target semantic feature network. The cross entropy loss function formula is as follows:

[0143]

[0144] Among them, L CE is the cross entropy loss value, N is the number of categories, y i is the i-th image x i The true label, Is the model to be trained on images x i The predicted probability of class i.

[0145] In step S11, semantic segmentation is performed on each of the training images according to the target semantic feature network to obtain a semantic segmentation map of each of the training images.

[0146] Through the trained target semantic feature network, semantic segmentation can be performed directly on various training images, such as Figure 7 As shown, Figure 7 The following are semantic segmentation maps after semantic segmentation of some exemplary training images based on the target semantic feature network.

[0147] like Figure 10 As shown, an ablation analysis is performed on the corrected foreground mask generated based on the saliency map proposed in the embodiment of the present application. Figure 10 It is proved that after adding the proposed correction algorithm to the baseline model (i.e., the corrected foreground mask obtained by the correction algorithm), the mIoU (semantic segmentation index) of the final segmentation map is improved by 2.32%; among them, the larger the mIoU, the better the semantic segmentation effect, which qualitatively proves the effectiveness of the corrected foreground mask generated based on the saliency map. The comparison results are shown in Figure 2. Figure 10 shown.

[0148] This application also compares and analyzes other saliency maps and foreground masks, and proves that the best combination is to set the pixel positions with saliency scores lower than the threshold 1 in the foreground mask as the background class, that is, by Figure 11 The threshold value 1 obtained by this comparative analysis method can be used as the significance threshold described in this application, and the semantic segmentation effect of the corrected foreground mask obtained thereby is excellent.

[0149] Finally, see Figure 8 As shown, Figure 8This is the overall system architecture of the target semantic feature network of this application. This application does not need to rely on strong supervision or weak supervision implementation methods. Without using labeled data, it generates a clear and corrected foreground mask through unsupervised representation learning, combined with pixel-level target attention classification head and saliency map, so that the outline of the corrected foreground mask is clearer, thereby realizing unsupervised semantic segmentation. While improving the quality of pseudo-labels and the accuracy of semantic segmentation by correcting the foreground mask, it effectively reduces the system's dependence on manual annotation. At the same time, it can also be used as an automatic annotation system to automatically annotate image data sets, significantly reducing the labor cost of data annotation.

[0150] Furthermore, the above-mentioned figures are merely illustrative of the processes included in the methods according to exemplary embodiments of the present application and are not intended to be limiting. It is readily understood that the processes illustrated in the above-mentioned figures do not indicate or limit the temporal order of these processes. Furthermore, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0151] It should be understood that the present application is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes can be performed without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. An unsupervised semantic segmentation method based on saliency map and deep learning, characterized in that: The method comprises: Selecting an initial deep learning model and designing a self-supervised loss function for unsupervised representation learning for the deep learning model; Performing unsupervised representation learning on the deep learning model according to the self-supervised loss function and the image dataset to be trained to construct an initial semantic feature network; performing feature extraction on the image dataset based on the initial semantic feature network to obtain a pixel-level feature set for each training image in the image dataset; Generating a foreground mask for each of the training images according to a preset target attention classification head and each of the pixel-level feature sets, wherein the target attention classification head carries a target activation function; For each training image, obtaining a saliency map of the training image, and generating a corrected foreground mask according to the saliency map and the foreground mask; Generating an image-level feature set for each of the training images according to each of the pixel-level feature sets and each of the corrected foreground masks; Clustering each of the image-level feature sets to obtain a feature set including multiple categories; Generating a category map for each of the training images according to each of the image-level feature sets and the feature set; generating pseudo labels for each of the training images based on each of the category maps and each of the corrected foreground masks; Training the initial semantic feature network according to each of the pseudo labels and preset segmentation head parameters to obtain a target semantic feature network; Performing semantic segmentation on each of the training images according to the target semantic feature network to obtain a semantic segmentation map of each of the training images; The step of generating a category map for each of the training images according to each of the image-level feature sets and the feature set includes: For each image-level feature set, performing cosine similarity calculation on each pixel-level feature in the image-level feature set and the corresponding feature set to obtain a background value corresponding to each pixel point corresponding to each pixel-level feature, where each pixel-level feature corresponds to one feature set and each pixel-level feature corresponds to a single pixel point; For each training image, generating the category map according to each background value; Generating pseudo labels for each of the training images based on each of the category maps and each of the corrected foreground masks includes: For each training image, pixels whose background values ​​are lower than a preset background threshold in the category map are removed to generate a target category map, and a pseudo label of the training image is generated based on the target category map and the corrected foreground mask.

2. The unsupervised semantic segmentation method based on saliency map and deep learning according to claim 1, characterized in that The self-supervised loss function is obtained by the following steps: Design the first loss function , the first loss function It is reflected by the following formula: Where, is the number of training images, is the first i Training pictures, is the feature extractor, For training pictures The corresponding features, For training pictures Training images after applying a preset type of image augmentation transformation; Design the second loss function , the second loss function It is reflected by the following formula: Where, is the number of training images, is the number of intermediate layers in the deep learning model, is the first i Training pictures, is the feature extractor, For training pictures In the middle layer k Layer characteristics, For training pictures Training images after applying a preset type of image augmentation transformation; According to the first loss function And the second loss function Determine the self-supervised loss function , the self-supervised loss function It is reflected by the following formula: Where, is the preset first weight coefficient, is the preset second weight coefficient.

3. The unsupervised semantic segmentation method based on saliency map and deep learning according to claim 2, characterized in that Generating a foreground mask for each training image according to a preset target attention classification head and each pixel-level feature set includes: For each training image, input each pixel-level feature in the pixel-level feature set into the target activation function of the target attention classification head to obtain the activation output value of each pixel-level feature; For each activation output value, if the activation output value is greater than a preset activation threshold, determining the pixel-level feature corresponding to the activation output value as a target pixel-level feature for generating a foreground mask; For each training image, generating a foreground mask of the training image according to each of the target pixel-level features; The target attention classification head is adjusted by the self-supervised loss function.

4. The unsupervised semantic segmentation method based on saliency map and deep learning according to claim 3, characterized in that The obtaining of the saliency map of the training image and generating a corrected foreground mask according to the saliency map and the foreground mask includes: Dividing the training image into regions according to a preset image segmentation method to obtain the saliency map consisting of a plurality of saliency regions, each of the saliency regions including a plurality of similar pixel-level features, each of the pixel-level features corresponding to a single pixel; Calculating a significance value of each of the significant regions; For each salient region, if the saliency value is lower than a preset saliency threshold, the pixel corresponding to the saliency region is determined to be a background pixel; The pixel-level features corresponding to the background pixels are excluded from the foreground mask to generate the corrected foreground mask.

5. The unsupervised semantic segmentation method based on saliency map and deep learning according to claim 4, characterized in that Generating an image-level feature set for each of the training images according to each of the pixel-level feature sets and each of the corrected foreground masks includes: For each training picture, the pixel-level feature weight of the pixel-level feature set is generated according to the corrected foreground mask, and the pixel-level features of each background pixel point in the pixel-level feature set are removed based on the pixel-level feature weight and global average pooling is performed to obtain the image-level feature set of the training picture.

6. An unsupervised semantic segmentation system based on saliency maps and deep learning, characterized by: The system comprises: A design unit, configured to select an initial deep learning model and design a self-supervised loss function for unsupervised representation learning for the deep learning model; An unsupervised learning unit, configured to perform unsupervised representation learning on the deep learning model based on the self-supervised loss function and the image dataset to be trained, and to construct an initial semantic feature network; a feature extraction unit, configured to perform feature extraction on the image dataset based on the initial semantic feature network to obtain a pixel-level feature set for each training image in the image dataset; a foreground mask unit, configured to generate a foreground mask for each of the training images based on a preset target attention classification head and each of the pixel-level feature sets, wherein the target attention classification head carries a target activation function; a correction unit, configured to obtain, for each training image, a saliency map of the training image, and generate a corrected foreground mask according to the saliency map and the foreground mask; An image feature generation unit, configured to generate an image-level feature set for each of the training images based on each of the pixel-level feature sets and each of the corrected foreground masks; a clustering unit, configured to cluster each of the image-level feature sets to obtain a feature set comprising multiple categories; Generating a category map for each of the training images according to each of the image-level feature sets and the feature set; a pseudo label generating unit, configured to generate a pseudo label for each of the training images based on each of the category maps and each of the corrected foreground masks; A training unit, configured to train the initial semantic feature network according to each of the pseudo labels and preset segmentation head parameters to obtain a target semantic feature network; A semantic segmentation unit, configured to perform semantic segmentation on each of the training images according to the target semantic feature network to obtain a semantic segmentation map of each of the training images; The step of generating a category map for each of the training images according to each of the image-level feature sets and the feature set includes: For each image-level feature set, performing cosine similarity calculation on each pixel-level feature in the image-level feature set and the corresponding feature set to obtain a background value corresponding to each pixel point corresponding to each pixel-level feature, where each pixel-level feature corresponds to one feature set and each pixel-level feature corresponds to a single pixel point; For each training image, generating the category map according to each background value; Generating pseudo labels for each of the training images based on each of the category maps and each of the corrected foreground masks includes: For each training image, pixels whose background values ​​are lower than a preset background threshold in the category map are removed to generate a target category map, and a pseudo label of the training image is generated based on the target category map and the corrected foreground mask.

Images