Line Detection Method Based on Template Classification and Explicit Line Descriptors

Abstract

The present invention relates to the field of digital image processing technology, and particularly to a line detection method based on template classification and explicit line descriptors. The present invention uses the local linear structure between adjacent pixels to assist in judging the pixels located on the line segment. The local linear structure template constructed by the present invention provides a robust relationship between adjacent pixel points and indicates the direction of the line at the same time. The explicit line descriptor has distinctiveness for the extension and merging of short line segments. It solves the problem of generating more redundant short line segments and incorrect line segments in the existing traditional methods. Compared with the deep learning method, competitive results can also be obtained by using only 1% of the training data, saving computing resources. The method proposed by the present invention not only has high detection accuracy, but also has good robustness to rotation and noise.

Description

Technical Field

[0001] The invention relates to the technical field of digital image processing, and in particular to a line detection method based on template classification and explicit line descriptors. Background Art

[0002] Straight lines provide the most basic geometric structure in an image, which is crucial for further image processing. Line detection is a very important and challenging basic task in computer vision, and it has been widely used in various practical scenarios, such as 3D reconstruction, scene parsing, and image stitching. These applications in natural scenes must not only ensure accurate detection of line segments, but also ensure robustness in special situations such as noise and rotation.

[0003] Based on different implementation methods, the existing line detection methods are divided into two categories, namely traditional methods and methods based on deep learning. Traditional methods are divided into two categories, namely, line detection algorithms based on spatial transformation and line detection algorithms based on image gradients. These two algorithms have their own characteristics: the line detection method based on spatial transformation can find more, longer and more continuous line segments, but it is easy to have erroneous long straight lines; the edge detection method based on image gradients is easier to locate the endpoint position of the straight line, but it is easy to have shorter straight lines. The line detection method based on deep learning has attracted widespread attention for its remarkable performance. Discrete features are usually used to locate the endpoints of the line segments, while ignoring the spatial distribution relationship between adjacent pixels. At the same time, a relatively complete training set is required, and the computational cost is high. Considering that the edge detection method based on image gradients has a higher detection efficiency, the present invention hopes to detect more robust straight lines through improvements on this basis. Summary of the invention

[0004] The present invention provides a line detection method based on template classification and explicit line descriptors. The local linear structure template includes linear structures in different directions, which represents the spatial distribution relationship of adjacent pixels and can reflect whether adjacent pixels are collinear. With the help of the linear structure template, the present invention utilizes the correlation between the direction and texture formed by all pixels on the line to obtain a more accurate and stable line segment.

[0005] The technical solution of the present invention is as follows:

[0006] A line detection method based on template classification and explicit line descriptor, the steps are as follows:

[0007] Step 100, constructing a local linear structure template by K-means clustering, the template including linear structures in different directions; based on the linear structure template, training a random forest classifier to predict the probability that an image pixel belongs to each type of template;

[0008] Step 200: For each image to be detected, use the trained classifier to calculate the probability that each pixel point falls on the line, and obtain a probability map.

[0009] Step 300: Based on the probability map, connect and segment the pixel chains according to the differences between adjacent pixels, and use the least squares method to fit the initial line segments.

[0010] Step 400: Construct line descriptors based on the probability distribution of pixel points to extend and merge short line segments, evaluate the consistency of online pixels, and obtain robust line segments.

[0011] Preferably, in Step 100, clustering means dividing the input data samples without original categories into several different categories of data samples according to the similarity principle, and the sample data in each category has similar attributes. K-means is a commonly used unsupervised clustering method. Due to its simple design concept, relatively easy implementation, fast convergence speed, and the clustering result being independent of the input order of the data, it is widely used. Based on the feature that there are similar texture features around the pixel points where the line is located, the present invention uses K-means to cluster the image patches with similar linear structures around each pixel point into one category, obtaining several templates with different linear structures to represent lines in different directions. For the image patch represented by each pixel point, after clustering, it is known which category it belongs to. The color gradient feature and autocorrelation feature of the image patch are used as the input features of the classifier for training to predict the template category to which each pixel point belongs.

[0012] Preferably, in Step 200, for a picture to be detected, it is necessary to use the trained classification model to classify all pixel points in the picture to be detected one by one. This category can be one of the template classes or the background class. When inputting a new picture, each pixel point of the picture will be traversed, and the feature vector of its corresponding image patch will be input into the trained random forest. Finally, the probability that the pixel point belongs to each line template class and the probability of the background will be output. The sum of the probabilities belonging to all template classes is the probability that the pixel point belongs to the line. Finally, the non-maximum suppression algorithm is used to extract the contour of the line, and finally a matrix with the same size as the original image is obtained. The value at each position in the matrix represents the probability that the pixel point at the corresponding position in the original image belongs to the line, which is the probability map.

[0013] Preferably, in step 300, since the larger the straight-line probability value of a pixel point, the more prominent the straight line where the point is located, the maximum value point in the mapping probability map is selected as the anchor point, and then pixel points with similar directions are searched within the eight-neighborhood range of the pixel and connected into a chain. Due to the inherent complexity of the image, there are pixels in the pixel chain that satisfy the minimum direction constraint but are not on the same straight line. If the number of pixel points in the pixel chain is greater than a certain threshold, a line segment is generated by connecting both ends of the pixel points. The pixel point farthest from the line in the pixel chain is found, and the distance from the pixel point to the line segment is calculated. If it is greater than one pixel, the pixel chain is split into two with this pixel point, and the iteration is performed until all pixel chains are collinear. The least squares method is used to fit a series of almost collinear pixel chains into initial short line segments.

[0014] Preferably, in step 400, in order to obtain more complete and accurate line segments, an explicit line descriptor is proposed to represent the characteristics of the line. In step 200, the probability that each pixel point belongs to each type of template is obtained. At the same time, it can be known that the distribution of pixel points on each straight line on the N templates is roughly the same and concentrated on certain templates. An N-dimensional "voter" is set corresponding to the N types of templates. The templates corresponding to the first n maximum probability values of each pixel point on the straight line are respectively selected as "votes", and then the first m templates with the most votes in the "voter" (m < n) are selected. Finally, the average value of the probabilities of all pixel points on the straight line corresponding to these m templates is calculated as the characteristic of the straight line.

[0015] For the initial short line segments obtained in step 300, the method of step 400 is used to calculate the corresponding line characteristics. The adjacent pixel points are found along the straight line direction, and if the sum of the squares of the differences from the probability values corresponding to the current line characteristic template is less than a certain threshold, they are added to the pixel chain for re-fitting to obtain an extended straight line.

[0016] The beneficial effects of the present invention: A line detection method based on template classification and explicit line descriptors proposed by the present invention uses the local linear structure between adjacent pixels to assist in judging the pixels located on the line segment. The local linear structure template constructed by the present invention provides a robust relationship between adjacent pixel points and at the same time indicates the direction of the line. The explicit line descriptor is discriminative for the extension and merging of short line segments. It solves the problem of generating more redundant short line segments and incorrect line segments in existing traditional methods. Compared with deep learning methods, competitive results can also be obtained using only 1% of the training data, saving computing resources. The method proposed by the present invention not only has high detection accuracy but also has good robustness to rotation and noise. Description of the Drawings

[0017] Figure 1 is the flowchart of the line detection method based on template classification and explicit line descriptors of the present invention;

[0018] Figure 2 is a schematic diagram of a local linear template;

[0019] Figure 3 is a schematic diagram of extracting image patches and training a random forest classifier.

[0020] Figure 4 is a schematic diagram of a pixel chain segmentation strategy.

[0021] Figure 5(a) is a schematic diagram of marking three line segments on an image, Figures 5(b) to 5(d) which are respectively schematic diagrams of the probability distributions of the pixels on the three line segments over 50 template categories.

[0022] Figure 6(a) is the probability distribution of N points over template categories, and Figure 6(b) is a schematic diagram of the voting strategy of the line descriptors constructed by the N points.

[0023] Figure 7 is a schematic diagram of a line segment extension and merging strategy. Detailed implementation manners

[0024] The following further describes the detailed implementation manners of the present invention in combination with the accompanying drawings and technical solutions.

[0025] As Figure 1 shown, a line detection method based on template classification and explicit line descriptors of the present invention includes the following processes:

[0026] Step 100, constructing a local linear structure template through K-means clustering. The template includes linear structures in different directions. Based on the linear structure template, training a random forest classifier to predict the probability that an image pixel point belongs to each type of template.

[0027] In this step, for the manually annotated binary map corresponding to each RGB image, extracting the image patches whose center points are located on the marked lines. Each pixel point on the line corresponds to an image patch of a fixed size, which is used to represent various local linear structures. To obtain robustness to slight displacements, using Daisy descriptors to obtain features. Then using the K-means method (k = 50 in this embodiment) to cluster the features of the linear templates. To visually represent the cluster centers, using the average value of the image patches of the same class as the template of the center. As Figure 2 shown, these cluster centers represent the most common local linear structures in the real-world scene, and the corresponding image patches are called linear structure templates.

[0028] As Figure 3As shown by the double arrows, corresponding to each template category, corresponding image patches in the sampled RGB image are obtained. Since there are 50 linear templates, these image patches belong to 50 different categories. The pixel points falling on the line segment are denoted as on-line pixels. To distinguish on-line pixels from non-on-line pixels, image patches with centers not on the line segment are also sampled as a non-linear category. To make each detected pixel correspond to one category in the template, the classifier needs to be discriminative and accurately classify multiple categories. The random forest classifier meets the above conditions, and the color gradient features and autocorrelation features of the image patches are used as the input features of the classifier.

[0029] Specific description: Figure 3 It is a schematic diagram of extracting image patches and training a random forest classifier. After clustering, it can be known which category each binary image patch belongs to. This category is regarded as the label of the image patch, and the corresponding image patch features are used as the input of the classifier for training.

[0030] Step 200, for each image to be detected, use the trained classifier to calculate the probability that each pixel point falls on the line, and obtain the probability map.

[0031] For each image to be detected, extract the image patches centered on each pixel point, and input the corresponding features into the trained classifier, the probability that the image pixel point belongs to each type of template can be obtained, that is, the probability map. The probabilities on 50 linear template categories are added to represent the probability that the pixel point falls on the line segment. Considering the clarity and quantity of the lines in the image, an adaptive threshold setting is adopted to determine the position of the final line pixel points. When the sum of the probabilities of the pixels in the probability map on the linear template categories is higher than the threshold, it is regarded as an on-line pixel.

[0032] Step 300, based on the probability map, connect and segment the pixel chains according to the differences of adjacent pixels, and use the least squares method to fit the initial line segment.

[0033] In this step, the pixel point with the maximum on-line probability is selected as the starting point, and the adjacent pixels with similar gradient directions in its eight-neighborhood are connected to generate a series of pixel chains. Due to the inherent complexity of the image, there are pixels in the pixel chains that satisfy the minimum direction constraint but are not on the same straight line. To solve this problem, a point-line distance segmentation strategy is adopted to evaluate the collinearity of the pixels in the pixel chains.

[0034] As Figure 4 shown, each grid represents a pixel. Assume there is a pixel chain between A and B, and the pixels in it are represented by gray grids. The length of the pixel chain is the number of pixels in it. If the length exceeds the threshold θ l , then connect the two ends of the pixel chain to generate a virtual line segment L AB(Indicated by the long line segments in the figure). If the pixels in the pixel chain belong to the same line segment, then they should be close to L AB . Detect the pixel D farthest from L in the pixel chain AB and calculate its distance to the line segment L AB . If the distance is greater than 1 pixel, split the pixel chain in two at pixel D to obtain pixel chains AC and BD. Iteratively split the pixel chain so that the pixels on the pixel chain are nearly collinear. Finally, use the least squares method to fit these split pixel chains to obtain the initial short line segments.

[0035] Step 400: Construct a line descriptor based on the probability distribution of pixel points to extend and merge short line segments, evaluate the consistency of online pixels, and obtain robust line segments.

[0036] This step proposes a new line descriptor to evaluate the consistency of adjacent short line segments. Collinear pixels exhibit similar probability distributions over 50 clustering centers. As Figures 5(a) to 5(d) shown, three line segments are marked in Fig. 5(a). Taking L1 in Fig. 5(b) as an example, the horizontal axis represents 50 template categories, and the vertical axis represents the corresponding probability values. Each color represents the probability distribution of an online pixel, and it can be seen that the probability distributions of collinear points are very similar. L2 and L3 show that the probability distributions of non-collinear points are very different. This step uses this as a basis to evaluate the consistency of adjacent line segments.

[0037] This step designs a voting strategy for all online pixels to construct a robust line descriptor. As shown in Figs. 6(a) and 6(b), assume there are N pixel points on a line segment, and the probability distributions of two pixel points are shown at the top. Since collinear points have similar distribution trends but different peaks in the same template category, the categories with peaks are used for voting. For each pixel point, select the top 20 probability values and vote for the corresponding category. As shown in the histogram, select the top 15 categories with the most votes among the 50 categories. The category labels of the top 15 are shown in the bottom array. Calculate the average value of the probabilities of all pixel points corresponding to these 15 categories, and use a 15-dimensional vector as the line descriptor. To make the line descriptor insensitive to the other 35 categories, a simple normalization scheme is adopted.

[0038] Based on the fitted line segments and their descriptors, the existing line segments can be extended into a longer line segment. As Figure 7As shown, line segments L1 and L2 are respectively fitted by different pixels. Expand the line segment along the direction of L1. Based on one of its endpoints, adjacent pixel A and its two upper and lower pixels B and C can be found. Sort the distances from pixels A, B, and C to L1 in ascending order, and compare the features of the candidate pixels and the features of L1 in turn. Since the line descriptor is a 15-dimensional feature, in this embodiment, the probabilities of the pixel points on the corresponding 15 categories are used as the features of the pixels. If the Euclidean distance between the pixel feature and the current line descriptor is less than the predefined threshold θ m = 1, then add the pixel to L1 and discard other candidate pixels. Then fit L1 and the added pixels to obtain a new extended line segment. The expansion process is iterative.

[0039] During the expansion process, if there are candidate pixels that belong to another fitted line segment L2 at the same time, such as pixels D and E, and the Euclidean distance between the line descriptors of L1 and L2 is less than the threshold θ m , L1 and L2 will be merged by the least squares fitting method. To ensure the collinearity of L1 and L2, the fitting error is used as a constraint condition. If the fitting error is less than 1, then retain the new fitted line segment L3. The same strategy is adopted for the two endpoints of the line segment. If a line segment satisfies the consistency of both the gradient magnitude and direction at the same time, it is marked as the detected line segment.

Claims

1. A line detection method based on template classification and explicit line descriptors, characterized in that, The steps are as follows: Step 100: Construct a local linear structure template through K-means clustering. The template includes linear structures in different directions. Based on the linear structure template, train a random forest classifier to predict the probability that an image pixel belongs to each type of template. Step 200: For each image to be detected, use the trained classifier to calculate the probability that each pixel falls on a straight line, and obtain a probability map. Step 300: Based on the probability map, connect and segment pixel chains according to the differences between adjacent pixels, and use the least squares method to fit the initial line segments. Step 400: Construct a line descriptor based on the probability distribution of pixel points to extend and merge short line segments, and evaluate the consistency of online pixels to obtain robust line segments. In step 300, since the larger the straight line probability value of a pixel point, the more significant the straight line where the point is located, select the maximum value point in the mapped probability map as the anchor point, and then search for pixel points with similar directions within the eight-neighborhood range of this pixel and connect them into a chain. Due to the inherent complexity of the image, there are pixels in the pixel chain that satisfy the minimum direction constraint but are not on the same straight line. If the number of pixel points in the pixel chain is greater than a certain threshold, generate a line segment by connecting the two ends of the pixel points, find the pixel point in the pixel chain that is farthest from the line, calculate the distance from the pixel point to the line segment. If it is greater than one pixel, divide the pixel chain into two with this pixel point, and iterate until all pixel chains are collinear. Use the least squares method to fit a series of almost collinear pixel chains into initial short line segments. In step 400, in order to obtain more complete and accurate line segments, an explicit line descriptor is proposed to represent the characteristics of the line. In step 200, the probability that each pixel point belongs to each type of template is obtained. At the same time, the distribution of pixel points on each straight line on N templates is roughly the same and is concentrated on certain templates. Set an N-dimensional "voter" corresponding to N types of templates, and respectively select the templates corresponding to the first n maximum probability values of each pixel point on the straight line as "votes", and then select the first m templates with the most votes in the "voter". Finally, calculate the average value of the probabilities of all pixel points on the straight line corresponding to these m templates as the characteristic of the straight line. For the initial short line segments obtained in step 300, use the method in step 400 to calculate the corresponding line characteristics. Find adjacent pixel points along the straight line direction. If the sum of the squares of the differences between the probability values corresponding to the current straight line feature template is less than a certain threshold, add them to the pixel chain for re-fitting to obtain an extended straight line.

2. The straight line detection method based on template classification and explicit straight line descriptors according to claim 1, characterized in that Specifically as follows: In step 100, use K-means to cluster image patches with similar linear structures centered around each pixel point into one category, obtaining several templates with different linear structures to represent straight lines in different directions. For the image patch represented by each pixel point, after clustering, know the category it belongs to, and use the color gradient feature and autocorrelation feature of the image patch as the input features of the classifier for training to predict the template category to which each pixel point belongs. In step 200, for a picture to be detected, it is necessary to use the trained classification model to classify all pixel points in the picture to be detected one by one. This category can be one of the template categories or the background category. When a new picture is input, each pixel point of the picture will be traversed, and the feature vector of its corresponding image patch will be input into the trained random forest. Finally, the probability that the pixel point belongs to each straight-line template class and the probability of the background will be output. The sum of the probabilities belonging to all template classes is the probability that the pixel point belongs to a straight line. Finally, the non-maximum suppression algorithm is used to extract the outline of the straight line, and finally a matrix with the same size as the original image is obtained. The value at each position in the matrix represents the probability that the pixel point at the corresponding position in the original image belongs to a straight line, which is the probability mapping diagram.

Images