A Plane Segmentation Method Based on Superpixels and Graph Convolutional Networks

Abstract

The present invention discloses a plane segmentation method based on superpixels and graph convolutional networks; firstly, the input color image is preprocessed to obtain the preprocessed image; then, according to the image resolution, the preprocessed image is segmented into a suitable number of superpixels and converted into an undirected graph structure; a graph convolutional network is constructed and trained using a data set; finally, the trained graph convolutional network is used to predict the graph composed of superpixels, and each superpixel is plane classified, thereby completing the plane segmentation. The present invention segments the image into superpixels, which can well preserve the edge information in the original image, and reduce the learning burden of the subsequent graph neural network, preventing the segmented plane edge from being too far away from the actual situation; using the existing data set, a specific label is extracted through a specific algorithm as a training set for the subsequent supervised neural network learning, which solves the problem of no data set for plane segmentation and avoids the huge cost of manual labeling.

Description

Technical Field

[0001] The present invention belongs to the field of plane segmentation calculation, and specifically relates to a plane segmentation method based on superpixels and graph convolutional networks. Technical Background

[0002] Computer vision is a highly interdisciplinary and complex subject that is developing rapidly and is widely used in security, transportation, medical care and other fields, and is closely related to people's lives. The research goal of computer vision is to enable computers to complete visual tasks such as target detection, image description, and face recognition instead of humans. On the scale of human perception, planes are one of the most common structures in the environment. They have strong constraints, constraining a large number of points / lines and the information they carry. In addition, various curved surfaces can be approximated by planes, and the number of fitted planes is selected according to the accuracy requirements. In practical applications, many computer vision tasks require plane information. For example, in the field of robotics, identifying planes such as the ground and walls can be used for visual navigation, and identifying planes such as desktops and bookshelves can assist manipulators in grabbing and placing objects. In three-dimensional scene reconstruction, using planes instead of point clouds can achieve large-scale and concise reconstruction of a city.

[0003] Traditional plane segmentation methods use geometric primitives such as feature points, line segments, and superpixels to segment plane instances through methods such as Markov random fields, but the segmentation effect is not satisfactory. In recent years, deep learning has achieved great success in various computer vision tasks. Compared with geometric methods, the effect of using convolutional neural networks to segment planes is greatly improved. However, no matter which method is used, the edges of the segmented planes are somewhat different from the actual situation, and the real plane edges cannot be segmented well. For neural networks, the problem is that their segmentation planes are essentially clustering pixels, and the network cannot learn edge information well during the clustering process. The superpixels used in geometric methods can well preserve the edge information in the image, but ordinary convolutional neural networks can only be used to process inputs with regular distribution of information such as images. After generalizing regular information such as images, they can be represented using the structure of a graph, and then processed using the graph convolutional network in deep learning. Summary of the invention

[0004] In order to solve the problem of plane segmentation of a single color image, the present invention provides a plane segmentation method based on superpixels and graph convolutional networks. Its input is a single color image, and its output is the regional information of the plane instance in the image. The method of the invention uses superpixels to construct an undirected graph structure, and then uses a graph convolutional network to classify plane instance labels, thereby completing the plane segmentation task.

[0005] The present invention proposes a plane segmentation method based on superpixels and graph convolutional networks, comprising the following steps:

[0006] Step (1), preprocessing the input color image to obtain a preprocessed image;

[0007] Step (2), according to the image resolution, the preprocessed image is divided into an appropriate number of superpixels and converted into an undirected graph structure;

[0008] Step (3), construct a graph convolutional network and train it using the dataset;

[0009] Step (4), using the trained graph convolutional network to predict the graph composed of superpixels, classifying each superpixel into planes, thereby completing plane segmentation;

[0010] The beneficial effects of the present invention are as follows:

[0011] 1. The present invention divides the image into superpixels, which can well preserve the edge information in the original image, reduce the learning burden of the subsequent graph neural network, and prevent the segmented plane edge from being too far away from the actual situation;

[0012] 2. The present invention uses an undirected graph structure composed of superpixels as the input of the graph neural network, which greatly reduces the computational complexity and simplifies the structure of the neural network;

[0013] 3. The present invention utilizes the existing data set and extracts specific labels from the existing information therein through a specific algorithm as a training set for subsequent supervised neural network learning, thereby solving the problem of the lack of data sets for plane segmentation and avoiding the huge cost of manual labeling. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is an overall flow chart of the method of the present invention;

[0015] Figure 2 This is a schematic diagram of the graph convolutional network structure used in the present invention;

[0016] Figure 3 A schematic diagram of a subnetwork of a graph convolutional network used in the present invention; DETAILED DESCRIPTION

[0017] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0018] The plane segmentation method based on superpixel and graph convolutional network proposed in this invention is as follows: Figure 1 As shown, the following steps are included:

[0019] Step (1), preprocessing the input color image to obtain a preprocessed image;

[0020] 1-1. First, convert the input color image from RGB color space to HIV color space, then use histogram equalization algorithm for H, I, and V channels respectively, and then convert the image from HIV color space back to RGB color space to improve the image contrast;

[0021] 1-2. Use the Sobel operator to extract the edge information of the image obtained in step 1-1, and then add it to the image obtained in step 1-1, so as to enhance the edge of the image and obtain the preprocessed image;

[0022] Step (2), according to the image resolution, the preprocessed image is divided into an appropriate number of superpixels and converted into an undirected graph structure;

[0023] 2-1. An important parameter in the superpixel segmentation algorithm SLIC is the number of superpixels S. Assuming that the number of pixels in the image is P, the number of pixels contained in a single superpixel is N = P / S. Since the convolution operation in the subsequent graph neural network requires a fixed input size, N =

[0024] 30*30. In addition, considering that the plane is the basic geometric structure in the scene, in order to distinguish the plane in the image, the resolution of the input image must not be too low. If the number of image pixels is less than 180,000 (that is, the number of superpixels is less than 200) or greater than 720,000 (that is, the number of superpixels is greater than 800), the linear interpolation algorithm is used to scale the image to a resolution of 180,000 to 720,000 according to the original ratio to prevent the image quality from being too low, resulting in a decrease in segmentation quality, or to prevent the image from being too large and causing a lot of calculations. When the image resolution is appropriate, the SLIC algorithm is used to extract superpixels from the image, where the number of superpixels S = P / N;

[0025] 2-2. Take each superpixel as a node, and store each pixel in a list from top to bottom and from left to right to obtain the matrix representation of the node. Since each pixel contains three values ​​of R, G, and B, the matrix dimension is N*3. At the same time, adjacent superpixels are connected using undirected edges with a weight of 1, and each node is connected to itself using an undirected edge with a weight of 2 (indicating that for each node, its own features are more important than the features of adjacent nodes), and an undirected graph G is obtained. All superpixels are stacked according to their positions in the image (from top to bottom and from left to right) to form the feature matrix of G, with a dimension of S*N*3;

[0026] Step (3), construct a graph convolutional network and train it using the dataset;

[0027] 3-1. The structure diagram of the entire graph convolutional network is as follows Figure 2 As shown in Figure 1. The graph convolutional network adopts the UNet++ structure, which is composed of three nested sub-networks of different sizes. Each sub-network has the structure of UNet (such as Figure 3 ), but the depth of each sub-network is different (that is, the number of downsampling used is different, 1, 2, and 3 respectively), and the sub-networks are connected through downsampling and skip connections. Each sub-network consists of two parts: an encoder and a decoder. The encoder is used to encode the undirected graph, which includes convolution operations and downsampling operations; the decoder is used to extract the abstract encoding information and finally restore it to the plane instance classification of each superpixel, which includes convolution operations and upsampling operations;

[0028] 3-2. Convolution operation: Assume that the input undirected graph G c = {V c ,E c},in

[0029] Represents n in an undirected graph c nodes, Represents m in an undirected graph c undirected edges. First, construct G c The adjacency matrix A c , A c n c A real symmetric matrix of order, where each element a ij (i,j=0,1,…,n-1) represents the connection relationship between the i-th node and the j-th node, and the value is the weight of the undirected edge; then construct the matrix A c The degree matrix D c , D c Same as n c A real symmetric matrix with all elements except the main diagonal being 0 and the main diagonal elements being Reuse the input undirected graph G c The feature matrix X c , calculate the feature matrix X′ output by the convolution operation c :

[0030]

[0031] Where σ is the ReLU activation function, W c is the weight matrix that can be learned during the training process. It should be noted that the structure of the undirected graph does not change during the whole process, that is, V c 、E c The convolution operation only changes G c The feature matrix, that is, the information carried by each node;

[0032] 3-3. Downsampling operation is used to reduce the number of nodes in an undirected graph to achieve the function of compressing information. Assume that the input undirected graph G s = {V s ,Es},in Indicates n s Node. G s The adjacency matrix of s , the feature matrix is ​​X s (Dimension is S s *N s *3). First, we use a projection matrix M (with dimensions N) that can be learned during training. s *3) From the undirected graph G s The feature matrix X s Extract information Y from:

[0033]

[0034] Among them, ||·|| represents the modulus length. The dimension of Y is S s *3, take the average of the three elements in each row, and get the dimension S s *1 column vector Then set a subscript for each element (starting from 0) and record the subscripts of the largest 40% of the elements in the vector form, recorded as idx (dimension is 0.4S s *1). From the undirected graph G s Select the node corresponding to idx from the nodes of , and get the undirected graph G of the down-sampling output d , using the adjacency matrix:

[0035] A d =A s (id,id),id∈idx (3)

[0036] The output undirected graph G d The feature matrix X d for

[0037]

[0038] in, Represents the Hadamard product, and Broadcast() means using replication to Broadcast to X s (id) dimensions are consistent, X d The dimension is 0.4S s *N s *3;

[0039] 3-4. The upsampling operation converts the simplified undirected graph in the downsampling operation (i.e., the output undirected graph G d) is restored to the original complex structure, which is the inverse operation of downsampling. Each upsampling corresponds to the downsampling at the same layer, that is, the undirected graph structure of the upsampling input is the same as the output of the downsampling at the same layer, and both use the adjacency matrix A d The undirected graph structure of the up-sampled output has the same structure as the down-sampled input at the same layer, and the subscript vector idx in the down-sampled output is used in the up-sampling process. d The feature matrix is ​​X d After the same-layer upsampling operation, it is X′ s :

[0040]

[0041] Among them, Zeros (N s ,3) indicates N s *3-size zero matrix, X′ s Dimension S s *N s *3. The undirected graph output by the entire upsampling operation is equivalent to setting the features of the nodes that do not belong to the id in the undirected graph of the downsampling input at the same layer to zero;

[0042] 3-5. Since there is no special dataset for plane extraction, an algorithm is used to add the required labels to the existing dataset. Considering that the dataset collected in real life inevitably has a certain amount of noise due to hardware defects, the SYNTHIA dataset from the artificial environment is used. First, the semantic labels in the dataset are divided into two categories: planes (buildings, sidewalks, roads, etc.) and non-planes (pedestrians, plants, cars, etc.). Each scene in the dataset contains a color image, a depth image, and a semantic segmentation map. The three images have the same resolution and correspond to each other. First, use the method in step 2-1 to scale the three images in the scene to the appropriate resolution, and then use the SLIC algorithm to initialize S seed points on the color map, each seed point corresponds to a superpixel. Then, take out the semantic areas belonging to the plane class in the semantic segmentation map in turn, and use the J-Linkage clustering algorithm to extract plane instances at the corresponding positions in the depth map. Then correspond each plane instance to the color map, and continue to extract superpixels based on the existing seed points. For the semantic areas belonging to the non-plane class in the semantic segmentation map, they are fused into one area, correspond to the color map, and continue to extract superpixels based on the existing seed points. Finally, the superpixel is converted into an undirected graph using the method in step 2-2;

[0043] 3-6. For each scene in the SYNTHIA dataset, use the method in step 2 to convert it into an undirected graph to complete the construction of the dataset, and the ratio of the training set to the test set is 4:1 for training the graph convolutional network. During the training process, L2 regularization is used, the learning rate is 0.001, the optimizer is Adam, and the batch size of training and testing is 1;

[0044] Step (4): Use the trained graph convolutional network to predict the graph composed of superpixels and classify each superpixel into planes, thereby completing plane segmentation.

Claims

1. A plane segmentation method based on superpixels and graph convolutional networks, characterized in that: The following steps are involved: Step (1), preprocessing the input color image to obtain a preprocessed image; Step (2), according to the image resolution, the preprocessed image is divided into an appropriate number of superpixels and converted into an undirected graph structure; Step (3), construct a graph convolutional network and train it using the dataset; Step (4), using the trained graph convolutional network to predict the graph composed of superpixels, classifying each superpixel into planes, thereby completing plane segmentation; The specific operations of step (3) are as follows: 3-1. The graph convolutional network adopts the UNet++ structure, which is composed of three nested sub-networks of different sizes. Each sub-network has a UNet structure, but the depth of each sub-network is different. The sub-networks are connected through downsampling and jump connections. Each sub-network consists of two parts: an encoder and a decoder. The encoder is used to encode the undirected graph, which includes convolution operations and downsampling operations. The decoder is used to extract the abstract encoded information and finally restore it to the plane instance classification of each superpixel, which includes convolution and upsampling operations; 3-2. Convolution operation: Assume that the input undirected graph G c = {V c ,E c },in Represents n in an undirected graph c nodes, Represents m in an undirected graph c undirected edges; first construct G c The adjacency matrix A c , A c n c A real symmetric matrix of order, where each element a ij (i,j=0,1,…,n-1) represents the connection relationship between the i-th node and the j-th node, and the value is the weight of the undirected edge; then construct the matrix A c The degree matrix D c , D c Same as n c A real symmetric matrix with all elements except the main diagonal being 0 and the main diagonal elements being Reuse the input undirected graph G c The feature matrix X c , calculate the feature matrix X′ output by the convolution operation c : Where σ is the ReLU activation function, W c is the weight matrix that can be learned during training; 3-3. The downsampling operation is used to reduce the number of nodes in an undirected graph; assuming that the input undirected graph G s = {V s ,E s },in N s Nodes; G s The adjacency matrix of s , the feature matrix is ​​X s , X s The dimension is S s *N s *3;E s To represent the undirected edges in an undirected graph, we first use a projection matrix M that can be learned during training to transform the undirected graph G s The feature matrix X s Extract information Y from: Among them, ||·|| represents the modulus length; the dimension of Y is S s *3, take the average of the three elements in each row, and get the dimension S s *1 column vector Then set a subscript for each element and record the subscripts of the largest 40% of the elements in the vector form, denoted as idx; s Select the node corresponding to idx from the nodes of , and get the undirected graph G of the down-sampling output d , using the adjacency matrix: A d =A s (id,id),id∈idx (3) The output undirected graph G d The feature matrix X d for: in, Represents the Hadamard product, and Broadcast() means using replication to Broadcast to X s (id) dimensions are consistent, X d The dimension is 0.4S s *N s *3; 3-4. The upsampling operation restores the simplified undirected graph in the downsampling operation to the original complex structure, which is the inverse operation of downsampling. The undirected graph structure of the upsampling input is the same as the output of the downsampling at the same layer, using the adjacency matrix A d Indicates that the undirected graph structure of the upsampling output has the same structure as the input of the downsampling at the same layer, and the subscript vector idx in the downsampling will be used in the upsampling process; the undirected graph G of the downsampling output d The feature matrix is ​​X d After the same-layer upsampling operation, it is X′ s : Among them, Zeros (N s ,3) indicates N s *3-size zero matrix, X′ s Dimension S s *N s *3.

2. According to claim 1, a plane segmentation method based on superpixels and graph convolutional networks is characterized in that: The specific operation of step (1) is as follows: 1-1. First, convert the input color image from RGB color space to HIV color space, then use histogram equalization algorithm for H, I, and V channels respectively, and then convert the image from HIV color space back to RGB color space to improve the image contrast; 1-2. Use the Sobel operator to extract the edge information of the image obtained in step 1-1, and then add it to the image obtained in step 1-1, so as to enhance the edge of the image and obtain the preprocessed image.

3. The plane segmentation method based on superpixel and graph convolutional network according to claim 2, characterized in that: The specific operation of step (2) is as follows: 2-1. The number of superpixels S in the superpixel segmentation algorithm SLIC. Assuming that the number of pixels in the image is P, a single superpixel contains the number of pixels N = P / S, where N = 30*30. Use a linear interpolation algorithm to scale the image to a resolution of 180,000 to 720,000 according to the original ratio. When the image resolution is appropriate, use the SLIC algorithm to extract superpixels from the image, where the number of superpixels S = P / N. 2-2. Take each superpixel as a node, and store each pixel in a list from top to bottom and from left to right to obtain a matrix representation of the node, with a matrix dimension of N*3; at the same time, adjacent superpixels are connected using undirected edges with a weight of 1, and each node is connected to itself using an undirected edge with a weight of 2, obtaining an undirected graph G. All superpixels are stacked up according to their positions in the image to form the feature matrix of G, with a dimension of S*N*3.

4. The plane segmentation method based on superpixel and graph convolutional network according to claim 3, characterized in that: Step (3) also includes: 3-5. Use the SYNTHIA dataset from an artificial environment; first, divide the semantic labels in the dataset into two categories: plane and non-plane; each scene in the dataset contains a color image, a depth image, and a semantic segmentation map, and the three images have the same resolution and correspond to each other; first, use the method in step 2-1 to scale the three images in the scene to an appropriate resolution, and then use the SLIC algorithm to initialize S seed points on the color image, each seed point corresponds to a superpixel; then take out the semantic areas belonging to the plane class in the semantic segmentation map in turn, and use the J-Linkage clustering algorithm to extract plane instances at the corresponding positions in the depth map, and then correspond each plane instance to the color map, and continue to extract superpixels based on the existing seed points; for the semantic areas belonging to the non-plane class in the semantic segmentation map, merge them into one area, correspond to the color map, and continue to extract superpixels based on the existing seed points; finally, use the method in step 2-2 to convert the superpixels into an undirected graph; 3-6. For each scene in the SYNTHIA dataset, use the method in step 2 to convert it into an undirected graph, thereby completing the construction of the dataset, and the training set and test set The ratio is 4:1, which is used for training graph convolutional networks. During training, L2 regularization is used. The learning rate is 0.001, the optimizer is Adam, and the batch size is 1 for both training and testing.

Images