Unsupervised semantic segmentation ambiguity elimination method based on adaptive clustering difficult point processing

Abstract

The invention discloses an unsupervised semantic segmentation ambiguity elimination method based on adaptive clustering difficult point processing. The unsupervised semantic segmentation ambiguity elimination method comprises six processes of feature extraction, adaptive k-means clustering, difficult point processing, similarity adjacent matrix construction, graph convolution and semantic segmentation. According to the method, points with segmentation ambiguity are selected through k-means clustering, feature processing is carried out on difficult points through multiple clustering weights, feature fusion is carried out by using the graph convolutional network, points which are not definitely belong to a certain category can be processed, the points are more definitely divided into a certain category after feature fusion, and the segmentation accuracy is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision, and in particular relates to an unsupervised semantic segmentation ambiguity elimination method based on adaptive clustering difficult point processing. Background Art

[0002] Semantic segmentation is an important task in the field of computer vision. Its goal is to classify each pixel in an image into a specific semantic category. Unlike traditional image classification tasks, semantic segmentation requires pixel-level classification of images to achieve more sophisticated understanding and analysis. Semantic segmentation is widely used in autonomous driving, medical impact analysis, geographic information systems and other fields.

[0003] Unsupervised segmentation is an important research direction in the field of computer vision. It aims to automatically divide image or video data into different regions or objects through algorithms without manual annotation. This method is particularly suitable for situations where the annotation cost is high or the annotated data is scarce. Unsupervised segmentation focuses on mining potential structures and patterns from raw data without relying on pre-annotated category information. It is often used for tasks such as image segmentation, object detection, and clustering.

[0004] Traditional convolutional neural networks can only process Euclidean spatial data, but in real life, a lot of data exists in the form of graphs, so the introduction of graph convolution technology can effectively extract the features of graphs.

[0005] For pictures, each object in the picture has its own unique texture and context features, and people classify objects by observing these features. There are many types of objects in a picture, and their contextual relationships are also quite complex. It is difficult to extract the relationship between different objects by only extracting features through convolutional networks.

[0006] In an image, some pixels are difficult to be accurately classified into a certain category only through convolutional neural networks, because they may have texture features of multiple categories at the same time, resulting in uncertainty in the segmentation results. Although there are many technologies in the field of natural images that can enhance the distinction between pixels, most of these methods require the assistance of semantic labels. Summary of the invention

[0007] In view of the above problems, the purpose of the present invention is to provide an unsupervised semantic segmentation ambiguity elimination method based on adaptive clustering difficult point processing, which can solve the problem that certain difficult points in natural images are difficult to accurately segment without the need for semantic labels, and overcome the difficulty of excessive averaging of information between different categories.

[0008] In order to achieve the above object, the technical solution of the present invention is as follows:

[0009] An unsupervised semantic segmentation ambiguity elimination method based on adaptive clustering difficult point processing comprises the following steps:

[0010] 1) Feature extraction: Get the image to be segmented, select the segmentation network, and extract the feature map through Backbone;

[0011] 2) Adaptive clustering: Perform k-means clustering on all nodes in the feature graph extracted in step 1). The k value of clustering, i.e. the number of clusters, is determined by the silhouette coefficient. Perform M independent k-means clustering on all nodes in the feature graph, and each cluster represents a semantic category in the image.

[0012] 3) Cluster communication: Unify the M groups of clustering results, sort each cluster in each result according to the number of points, and determine which clusters in the M clustering results belong to the same semantic category in the image with the same number of points and similar point distribution, and merge the points of the same category into the same cluster of the new set;

[0013] 4) Difficult point processor: Filter out the points that are classified into different categories in multiple clusterings, count the cluster ratios of these points in M ​​clusterings, and perform feature splicing of the cluster center for this point according to the ratio;

[0014] 5) Similarity graph convolution: Calculate the cosine similarity between each point in the set and other points, and construct an adjacency matrix.

[0015] Perform graph convolution with the feature vector obtained through backbone to obtain a new feature vector after feature fusion;

[0016] 6) Segmentation: Concatenate the new feature vector and the original feature vector and put them back into the original segmentation network for subsequent segmentation to obtain the segmentation result.

[0017] Furthermore, the feature map F1 obtained through Backbone has a shape of H*W*C, where H is the length of the feature map, W is the width of the feature map, and C is the feature dimension, which is adjusted to N*C, where N=H*W. The N points are clustered through feature C, and the result after clustering is K*N, where K is the number of clusters and N is the number of points.

[0018] Furthermore, in step 2), each image is clustered by selecting the k-means clustering value through an adaptive method. The most suitable number of clusters in each image is determined mainly by the silhouette coefficient. The value range of k is 2-10 (the number of common semantic types in semantic segmentation tasks). Clustering is performed in sequence, and the value with the largest silhouette coefficient is selected as the number of clusters. The silhouette coefficient of a single point is calculated as follows:

[0019]

[0020] Where a(i) is the average distance from sample i to other samples in its cluster, b(i) is the average distance from sample i to the nearest samples in other clusters, and the average of all point silhouette coefficients is the silhouette coefficient of this clustering.

[0021] Furthermore, in step 3), a new collection S0 needs to be created to store the processed clusters. The processing method is as follows: after the clustering results are determined above, M clusters belonging to the same cluster are selected from the M clusters and put into the collection S0[0], and duplicate points are removed.

[0022] Furthermore, after the difficult points are screened out, they will be specially processed. The difficult points will be spliced ​​with the cluster centers to which they are divided, and the feature vectors of the difficult points will be modified. The K*N obtained by clustering can determine whether the point belongs to which cluster. The cosine similarity between the points in the same cluster is calculated as the value of the adjacency matrix, thereby constructing an adjacency matrix with a shape of N*N, and performing graph convolution with the original feature map F1. The original features are optimized to obtain the new feature map F2.

[0023] Furthermore, the specific operation process of step 6) is to splice the original feature map F1 and the new feature map F2, put them back to the original position of the original segmentation network, and obtain a new segmentation result.

[0024] The beneficial effects of the present invention are:

[0025] The present invention selects points with segmentation ambiguity through k-means clustering, performs feature processing on difficult points through multiple clustering weights, and uses a graph convolutional network for feature fusion. The present invention can process points that do not clearly belong to a certain category, so that they can be more clearly classified into a certain category after feature fusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is the overall flow chart of the present invention. DETAILED DESCRIPTION

[0027] In order to make the objectives, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0028] Reference Figure 1 , an unsupervised semantic segmentation ambiguity elimination method based on adaptive clustering difficult point processing, including six processes: feature extraction, adaptive k-means clustering, difficult point processing, adjacency matrix construction, graph convolution, and semantic segmentation.

[0029] Step 1: Get the original natural image, select a segmentation network, and extract the feature vector F1 through Backbone. The shape is H*W*C, where H is the length of the feature map, W is the width of the feature map, and C is the feature dimension. This step uses deeplabv3+ as the semantic segmentation network, and Backbone is Resnet-50. After obtaining the original natural image, Backbone performs convolution, maximum pooling, and normalization on the natural image, gradually reducing the size of the feature map, extracting information, and finally obtaining the feature vector F1.

[0030] Step 2, perform adaptive k-means clustering on the nodes in the feature map. Since each map may contain a different number of semantic categories, the adaptive method can improve the accuracy of clustering. First, perform k-means clustering on the N (N=H*W) points in the feature map F1. The value of k gradually increases from 2 to 10. The clustering performance is compared through the silhouette coefficient, and the k value with the best clustering performance is selected as the number of clusters of the image (generally, the number of clusters of natural images is in the range of 2-10).

[0031] The single point silhouette coefficient is calculated as follows: The average of all the silhouette coefficients is the silhouette coefficient of this cluster. The smaller the silhouette coefficient, the more compact the points in a cluster are, which means the better the clustering effect. The most suitable k value can be selected through the silhouette coefficient.

[0032] By performing M independent clustering on F1 through the selected k value, M groups of clustering results can be obtained, and each group of clustering results has k clusters. The results are saved in M ​​sets, each set contains k sets, that is, M clustering results are saved.

[0033] Step 3: Communicate and merge the M clustering results to create a new set S0. Set S0 contains k sets as a set to merge the final clustering results. Select a set S1 from the M clustering results and extract its first cluster S1. 1 , calculate S1 1 The intersection-and-union ratio of k clusters in the other (M-1) groups of clustering results is calculated. In each group of clustering results, the cluster with the largest intersection-and-union ratio can be selected. (M-1) clusters can be selected, which means that this (M-1) cluster is related to S1. 1 They should belong to the same semantic category in the picture. This (M-1) cluster is combined with S1 1 Merge and put into S0 1 In, check for duplicate points removal, S0 1 contains all the points that may belong to this cluster. In the same way for S1 2 、S1 3 ……S1 kPerform the same operation and merge the results into S0 2 、S0 3 ……S0 k In this way, S0 obtained includes k clusters and all the points they may contain. There may be duplications between points, and these repeated points are difficult points.

[0034] Step 4, filter out the difficult points and process the feature vectors of the difficult points: In step 3, k sets are obtained in S0, and the points that appear 2 times or more in the k sets are filtered out. For these points, the results of their being divided in M ​​clusters are counted. For example, point A is divided into cluster 1 8 times in 10 clusters and is divided into cluster 2 2 times. Then, the feature vector of point A can be processed according to this result, so that its features include some features of its cluster center; still taking point A as an example, select the cluster center feature vectors F1 and F2 of cluster 1 and cluster 2, and process the feature vector F of point A according to the ratio of 8:2 where point A is divided: Fn = 0.8*F1+0.2*F2+F. Fn obtained according to this formula includes the features of F itself and the features of the cluster center to which it is divided; according to this processing method, the features of all difficult points are updated, and the updated difficult point feature vector is used to replace the origin feature vector in F1 to obtain a new feature map F1.

[0035] Step 5, construct the adjacency matrix M based on S0, use the updated feature graph F1 and M to perform feature fusion through graph convolution to obtain the new feature F2: The construction of the adjacency matrix requires determining which points among the N points are connected. In the graph convolution network, we can regard points belonging to the same category as fully connected and set the corresponding values ​​in the matrix as 1, but this method may not be able to well reflect the connection relationship between different points. Because some points may be connected far away in the cluster, and some are close, all of them are regarded as 1 and feature fusion cannot be performed well. Therefore, in this experiment, the similarity between points is used as the value of the adjacency matrix.

[0036] The similarity is calculated using cosine similarity:

[0037]

[0038] For N points, if two points appear in a set in S0 at the same time, they can be considered connected. Their cosine similarity is calculated and filled into the adjacency matrix. After processing the connection relationship between all N*N points in this way, an N*N adjacency matrix M is obtained. The values ​​in the matrix are the cosine similarities of the corresponding points, and the value range is 0-1.

[0039] Use the adjacency matrix M and the feature map F1 for graph convolution. The graph convolution formula is: Y = A*F*W, where A is the adjacency matrix, F is the feature map, and W is the learnable weight. When M and F1 are substituted into the graph convolution, a new feature map F2 is obtained.

[0040] Step 6: Concatenate F2 and F1, put them back into the feature vector backbone of deeplabv3+, and continue segmentation to obtain the segmentation result.

[0041] The above-mentioned specific description further illustrates the purpose and technical solution of the invention in detail. For those skilled in the art, any modifications, equivalent substitutions, improvements, etc. made should be included in the protection scope of the present invention.

Claims

1. An unsupervised semantic segmentation ambiguity elimination method based on adaptive clustering difficult point processing, characterized in that: The steps include: 1) Feature extraction: Get the image to be segmented, select the segmentation network, and extract the feature map through Backbone; 2) Adaptive clustering: Perform k-means clustering on all nodes in the feature graph extracted in step 1). The k value of clustering, i.e. the number of clusters, is determined by the silhouette coefficient. Perform M independent k-means clustering on all nodes in the feature graph, and each cluster represents a semantic category in the image. 3) Cluster communication: Unify the M groups of clustering results, sort each cluster in each result according to the number of points. The same number of points and similar point distribution can determine which clusters in the M clustering results belong to the same semantic category in the image, and merge the points of the same category into the same cluster of the new set; 4) Difficult point processor: Filter out the points that are classified into different categories in multiple clusterings, count the cluster ratios of these points in M ​​clusterings, and perform feature splicing of the cluster center for this point according to the ratio; 5) Similarity graph convolution: Calculate the cosine similarity between each point in the set and other points, construct an adjacency matrix, and perform graph convolution with the feature vector obtained through the backbone to obtain a new feature vector after feature fusion; 6) Segmentation: Concatenate the new feature vector and the original feature vector and put them back into the original segmentation network for subsequent segmentation to obtain the segmentation result.

2. The unsupervised semantic segmentation ambiguity elimination method based on adaptive clustering difficult point processing as claimed in claim 1, characterized in that: In step 2), each image is clustered by selecting the k-means clustering value through an adaptive method. The most suitable number of clusters in each image is determined mainly by the silhouette coefficient. The value range of k is 2-10. Clustering is performed in sequence, and the k value with the largest silhouette coefficient is selected as the number of clusters.

3. The unsupervised semantic segmentation ambiguity elimination method based on adaptive clustering difficult point processing according to claim 1, characterized in that: The feature map F1 obtained by Backbone has a shape of H*W*C, where H is the length of the feature map, W is the width of the feature map, and C is the feature dimension, which is adjusted to N*C, where N=H*W. The N points are clustered through feature C, and the result after clustering is K*N, where K is the number of clusters.

4. The unsupervised semantic segmentation ambiguity elimination method based on adaptive clustering difficult point processing as claimed in claim 3, characterized in that: After being screened out, the difficult points will be specially processed. The difficult points will be spliced ​​with the cluster centers to which they are divided, and the feature vectors of the difficult points will be modified. The K*N obtained by clustering can determine whether the point belongs to which cluster. The cosine similarity between the points in the same cluster is calculated as the value of the adjacency matrix, thereby constructing an adjacency matrix of shape N*N, which is convolved with the original feature map F1, and the original features are optimized to obtain the new feature map F2.

5. The unsupervised semantic segmentation ambiguity elimination method based on adaptive clustering difficult point processing as claimed in claim 4, characterized in that: The specific operation process of step 6) is to splice the original feature map F1 and the new feature map F2, put them back to the original position of the original segmentation network, and obtain a new segmentation result.

Images