An unsupervised semantic segmentation ambiguity elimination method based on adaptive clustering difficult point processing

Abstract

The application discloses an unsupervised semantic segmentation ambiguity elimination method based on adaptive clustering difficult point processing, which comprises six processes of feature extraction, adaptive k-means clustering, difficult point processing, similarity adjacency matrix construction, graph convolution and semantic segmentation. The application selects points with segmentation ambiguity through k-means clustering, processes the difficult points through multiple clustering weights, and performs feature fusion using a graph convolution network. The application can process points that are not clearly classified into a certain category, so that the points are more clearly classified into a certain category after feature fusion, and the segmentation accuracy is improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer vision, and particularly relates to an unsupervised semantic segmentation ambiguity elimination method based on adaptive clustering difficult point processing. BACKGROUND

[0002] Semantic segmentation is an important task in the field of computer vision, and its goal is to classify each pixel in an image into a specific semantic category. Unlike traditional image classification tasks, semantic segmentation requires pixel-level classification of images, thereby achieving more detailed understanding and analysis. Semantic segmentation is widely used in automatic driving, medical influence analysis, geographic information systems and other fields.

[0003] Unsupervised segmentation is an important research direction in the field of computer vision, aiming to automatically divide image or video data into different regions or objects without human annotation. This method is particularly suitable for situations where annotation costs are high or annotation data is scarce. Unsupervised segmentation mainly focuses on mining potential structures and patterns from raw data without relying on pre-labeled category information. It is commonly used in image segmentation, object detection, clustering and other tasks.

[0004] Traditional convolutional neural networks can only process Euclidean space data, but there are many data in real life in the form of graphs, so the introduction of graph convolution technology can well extract the features of the graph.

[0005] For a picture, each object in the picture has its unique texture and contextual features, and people also classify objects by observing these features. The types of objects in a picture are diverse, and their contextual relationships are quite complex. It is difficult to extract the relationship between different objects through convolutional networks alone.

[0006] In a picture, some pixels are difficult to divide into an accurate class through a convolutional neural network alone, because they may have texture features of multiple classes, resulting in uncertainty in the segmentation result. Although there are many techniques in the field of natural images that can enhance the distinguishability between pixels, most of these methods require the assistance of semantic labels. SUMMARY

[0007] To solve the above problems, the present application provides an unsupervised semantic segmentation ambiguity elimination method based on adaptive clustering difficult point processing, which can solve the problem of difficult accurate segmentation of some difficult points in natural images without semantic labels, and overcome the difficulty of excessive averaging of information between different categories.

[0008] To achieve the above purpose, the technical scheme of the present application is as follows:

[0009] An unsupervised semantic segmentation ambiguity elimination method based on adaptive clustering difficult point processing, comprising the following steps:

[0010] 1) Feature extraction: obtain an image to be segmented, select a segmentation network, and extract a feature map through a Backbone;

[0011] 2) Adaptive clustering: performing k-means clustering on all nodes in the feature map extracted in step 1), the k value of clustering, i.e., the number of clusters, is determined by the silhouette coefficient, and all nodes of the feature map are independently clustered by k-means M times, and each cluster represents a semantic class in the image;

[0012] 3) Cluster communication: unifying M sets of clustering results, sorting each cluster in each result according to the number of points, and determining which clusters in the M clustering results belong to the same semantic class in the image according to the same number of points and similar point distribution, and merging the points of the same class into the same cluster in the new set;

[0013] 4) Difficult point processor: screening out points that are divided into different classes in multiple clustering, counting the clustering proportion of these points in M clustering, and splicing the feature of the cluster center according to the proportion;

[0014] 5) Similarity graph convolution: calculating the cosine similarity between each point and other points in the set, and constructing an adjacency matrix,

[0015] and performing graph convolution on the feature vector obtained through the backbone to obtain a new feature vector after feature fusion;

[0016] 6) Segmentation: splicing the new feature vector and the original feature vector, and putting them back into the original segmentation network for subsequent segmentation to obtain a segmentation result.

[0017] Further, the feature map F1 obtained through the Backbone has a shape of H*W*C, H is the length of the feature map, W is the width of the feature map, and C is the feature dimension, which is adjusted to N*C, wherein N=H*W, N points are clustered through the feature C, and the result obtained after clustering is K*N, K is the number of clusters, and N is the number of points.

[0018] Further, in step 2), the value of k-means clustering is selected by an adaptive method for clustering each image, and the most suitable number of clusters in each image is mainly determined by the silhouette coefficient, the value range of k is 2-10 (the common number of semantic categories in the task of semantic segmentation), clustering is performed in turn, and the value of the largest silhouette coefficient is selected as the number of clusters. The silhouette coefficient of a single point is calculated as follows:

[0019]

[0020] Wherein a(i): the average distance of sample i to other samples in the cluster it belongs to, b(i): the average distance of sample i to the nearest sample in other clusters, the average of all point silhouette coefficients is the silhouette coefficient of this clustering.

[0021] Further, a new set S0 needs to be created in step 3) for storing the processed clusters, and the processing method is as follows: after determining the clustering result, in M times of clustering, select M clusters belonging to the same cluster and put them into set S0[0], and remove the repeated points.

[0022] Further, the difficult points are processed after being screened out, and the difficult points are spliced with the cluster centers to which they are divided, the feature vector of the difficult point is modified, K*N of the cluster can determine whether the point belongs to which cluster, the cosine similarity between the points in the same cluster is calculated as the value of the adjacency matrix, and an adjacency matrix of N*N is constructed, and the original feature map F1 is graph-convolved to optimize the original feature to obtain a new feature map F2.

[0023] Further, the specific operation process of step 6) is to splice the original feature map F1 and the new feature map F2 and put them back to the original position of the original segmentation network to obtain a new segmentation result.

[0024] The beneficial effects of the present application are:

[0025] The present application selects points with segmentation ambiguity by k-means clustering, processes the feature of the difficult point by multiple clustering weights, and fuses the features using the graph convolution network, so that the points which are not clear to belong to a certain category can be processed and more clearly divided into a certain category after feature fusion. BRIEF DESCRIPTION OF DRAWINGS

[0026] Figure 1 The present application is a general flowchart. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be further described in detail below in combination with specific embodiments and with reference to the drawings.

[0028] Reference Figure 1 An unsupervised semantic segmentation ambiguity elimination method based on adaptive clustering difficult point processing includes six processes of feature extraction, adaptive k-means clustering, difficult point processing, adjacency matrix construction, graph convolution and semantic segmentation.

[0029] Step 1: Obtain the original natural image, select a segmentation network, and extract the feature vector F1 through Backbone. The shape is H*W*C, where H is the length of the feature map, W is the width of the feature map, and C is the feature dimension. This step uses deeplabv3+ as the semantic segmentation network and Resnet-50 as the Backbone. After obtaining the original natural image, Backbone performs convolution, maximum pooling, and normalization on the natural image, gradually reducing the size of the feature map, extracting information, and finally obtaining the feature vector F1.

[0030] Step 2: Perform adaptive k-means clustering on the nodes in the feature map. Since each map may contain a different number of semantic categories, the adaptive method can improve the accuracy of clustering. First, perform k-means clustering on the N (N = H * W) points in the feature map F1. The value of k gradually increases from 2 to 10. The clustering performance is compared by using the silhouette coefficient. The k value with the best clustering performance is selected as the number of clusters of the image (generally, the number of clusters of natural images is in the range of 2 to 10).

[0031] The silhouette coefficient of a single point is calculated as follows: The average of all the points' silhouette coefficients is the clustering silhouette coefficient. A smaller silhouette coefficient indicates a more compact cluster, which translates to a better clustering effect. The silhouette coefficient can be used to select the most appropriate k value.

[0032] By performing M independent clustering on F1 using the selected k values, we can obtain M groups of clustering results, each of which contains k clusters. The results are saved in M ​​sets, each of which contains k sets, which means that M clustering results are saved.

[0033] Step 3: communicate and merge the M clustering results to create a new set S0. Set S0 contains k sets as a set to merge the final clustering results. Select a set S1 from the M clustering results and extract its first cluster S1. 1 , calculate S1 1 The intersection-union ratio of k clusters in the other (M-1) clustering results is used to find the cluster with the largest intersection-union ratio in each clustering result. (M-1) clusters can be selected, which means that this (M-1) cluster is the same as S1. 1 Should belong to the same semantic category in the picture, and this (M-1) cluster is combined with S1 1 Merge and put into S0 1 In the middle, check for duplicate points removal, S0 1 It contains all the points that may belong to this cluster. In the same way for S1 2 、S1 3 ...S1 kThe same operation is performed, and the results are merged into S0 2 , S0 3 ... S0 k , and the S0 obtained in this way contains k clusters and all points that may be contained therein, and there may be repetitions between the points, and these repeated points are the difficult points.

[0034] Step 4: Screening difficult points and processing the feature vectors of the difficult points: In step 3, k sets are obtained in S0, and points that appear more than 2 times in the k sets are screened out. For these points, the results of their division into M times of clustering are counted, for example, point A is divided into cluster 1 for 8 times and cluster 2 for 2 times in 10 times of clustering. Then, the feature vector of point A can be processed according to this result, so that the feature of the cluster center is contained in the feature of point A. Still taking point A as an example, the cluster center feature vectors F1 and F2 of cluster 1 and cluster 2 are selected, and the feature vector F of point A is processed according to the proportion of 8:2: Fn = 0.8*F1 + 0.2*F2 + F. Fn obtained according to this formula contains the features of F itself and the cluster centers to which it is divided. According to this processing method, the feature of all difficult points is updated, and the original point feature vector in F1 is replaced by the updated difficult point feature vector to obtain a new feature map F1.

[0035] Step 5: Based on S0, an adjacency matrix M is constructed, and the updated feature map F1 and M are used for feature fusion through graph convolution to obtain a new feature F2: The construction of the adjacency matrix needs to determine which points in the N points are connected. In the graph convolution network, points belonging to the same category can be considered as fully connected, and the value in the corresponding matrix is considered as 1. However, this method may not be able to well reflect the connection relationship between different points. Because some points may be connected far away in the cluster, and some points may be connected close, considering all as 1 cannot well perform feature fusion. Therefore, in this experiment, the similarity between points is used as the value of the adjacency matrix.

[0036] The similarity is calculated by using cosine similarity:

[0037]

[0038] For N points, if two points appear in a set in S0 at the same time, they can be considered as connected, and their cosine similarity is calculated and filled into the adjacency matrix. In this way, the connection relationship between all N*N points is processed, and an N*N adjacency matrix M is obtained, and the value in the matrix is the cosine similarity of the corresponding points, and the size is 0-1.

[0039] The graph convolution is performed by using the adjacency matrix M and the feature map F1, and the graph convolution formula is Y = A * F * W, wherein A is the adjacency matrix, F is the feature map, and W is a learnable weight. When M and F1 are substituted into the graph convolution, a new feature map F2 is obtained.

[0040] In step 6, F2 is spliced with F1, and the segmentation is continued after the feature vector backbone of the deeplabv3+ is returned to obtain a segmentation result.

[0041] The above specific description further specifically describes the purpose and technical solution of the application, and any modification, equivalent replacement, improvement and the like made by those skilled in the art shall be included in the protection scope of the application.

Claims

1. An unsupervised semantic segmentation ambiguity elimination method based on adaptive clustering difficult point processing, characterized by: The steps include: 1) Feature extraction: Obtain the image to be segmented, select the segmentation network, and extract the feature map through Backbone; 2) Adaptive clustering: Perform k-means clustering on all nodes in the feature map extracted in step 1). The k value, i.e., the number of clusters, is determined by the silhouette coefficient. All nodes in the feature map are clustered M times independently using k-means, with each cluster representing a semantic category in the image. 3) Cluster communication: Unify the M groups of clustering results and sort each cluster in each result according to the number of points. The same number of points and similar point distribution can be used to determine which clusters in the M clustering results belong to the same semantic category in the image. Points of the same category are merged into the same cluster in the new set. 4) Difficult Point Processor: Filter out points that are classified into different categories in multiple clusterings, calculate the cluster ratios of these points in M ​​clusterings, perform cluster center feature splicing on this point based on the ratio, and replace the origin feature vector with the spliced ​​difficult point feature vector to obtain the updated new feature map; 5) Similarity Graph Convolution: Calculate the cosine similarity between each point in the set and every other point, construct an adjacency matrix, and perform graph convolution with the updated feature vector to obtain a new feature vector after feature fusion. 6) Segmentation: The new feature vector is concatenated with the original feature vector and put back into the original segmentation network for subsequent segmentation to obtain the segmentation result.

2. The unsupervised semantic segmentation ambiguity elimination method based on adaptive clustering difficult point processing according to claim 1, characterized in that: In step 2), each image is clustered by selecting the k-means clustering value through an adaptive method. The silhouette coefficient is mainly used to determine the most suitable number of clusters for each image. The value of k ranges from 2 to 10. Clustering is performed sequentially, and the k value with the largest silhouette coefficient is selected as the number of clusters.

3. The unsupervised semantic segmentation ambiguity elimination method based on adaptive clustering difficult point processing according to claim 1, characterized in that: The feature map F1 obtained by Backbone has a shape of H*W*C, where H is the length of the feature map, W is the width of the feature map, and C is the feature dimension, which is adjusted to N*C, where N=H*W. N points are clustered through feature C, and the result obtained after clustering is K*N, where K is the number of clusters.

4. The unsupervised semantic segmentation ambiguity elimination method based on adaptive clustering difficult point processing according to claim 3, characterized in that: After being screened out, the difficult points will be specially processed. The difficult points will be spliced ​​with the cluster centers to which they are divided, and the feature vectors of the difficult points will be modified. The K*N obtained by clustering can determine whether the point belongs to which cluster. The cosine similarity between the points in the same cluster is calculated as the value of the adjacency matrix. In this way, an N*N adjacency matrix is ​​constructed, which is convolved with the updated feature map to optimize the original features and obtain the new feature map F2.

5. The unsupervised semantic segmentation ambiguity elimination method based on adaptive clustering difficult point processing according to claim 4, characterized in that: The specific operation process of step 6) is to splice the original feature map F1 and the new feature map F2, put them back to the original position of the original segmentation network, and obtain a new segmentation result.

Images