An image region boundary line segment detection method, device, and storage medium

Abstract

The present invention provides an image region boundary line segment detection method, device, and storage medium, belonging to the technical field of pattern recognition. It solves the problems of unclear contour segmentation and redundant detection in non-boundary regions of image objects existing in existing straight line detection algorithms or applications during the straight line detection process. The image region boundary line segment detection method of the present invention includes the following steps: Step S1: Calculate the truncated nuclear norm of the image set; Step S2: Input the image set with labeled region boundary line segments as the training sample set, and in Step S2.1: Perform deep neural network training on the training sample set; Step S3: Obtain the trained neural network capable of performing image boundary line segment detection. The present invention has the advantages that the calculation of the truncated nuclear norm can highlight the structural details of the image set, thereby improving the training effect of the neural network, so that when the trained neural network performs image boundary line segment detection, obvious contour segmentation and clear and concise non-boundary regions of image objects can be obtained.

Description

Technical Field

[0001] The present invention belongs to the technical field of pattern recognition, and relates to a method, device, and storage medium for detecting boundary line segments of an image region, and particularly to a method, device, and storage medium for detecting boundary line segments of an image region using truncated nuclear norm regularization constraints. Background Art

[0002] Inferring three-dimensional geometric information from a two-dimensional image scene is a fundamental problem in computer vision. Many scenes in two-dimensional images, especially images of the built environment, can be approximated as piecewise planes. Traditional methods for constructing an environment or a three-dimensional model usually rely on line detection, triangulation of image features, or image patch matching. As an important but challenging low-level task, line segment detection (LSD) aims to extract visible line segments, contours, etc. in an image scene. Line segments in a digital image are located on geometric objects such as building boundaries, baseline passing through vanishing points, etc. Detecting lines in an image helps to perform subsequent multiple high-level vision tasks, including road line detection in autonomous driving, image segmentation, wireframe detection in architectural design, three-dimensional reconstruction, horizontal line detection in assisted driving, vanishing point calculation, and image vectorization, etc.

[0003] Currently, line detection in an image is usually described as a heuristic search problem, where edge pixels are grouped and then fitted into several line segments. The classic Hough Transform (HT) algorithm transforms the image into the polar coordinate system. Given an angle value θ and a length value ρ, a straight line in the corresponding image domain can be determined. Subsequently, the straight line in the corresponding image domain is deduced based on the ordered real number pair (θ, ρ) with the highest density in the polar coordinate system.

[0004] The line detection algorithm based on HT only calculates the form of straight lines or line segments in the image, and cannot perform segmentation detection on the image according to the image semantic information. Such methods rely too much on the selection of thresholds and are prone to incorrect recognition of edge pixels. In addition, some straight lines containing geometric information (such as the intersection of two planes of the same color) usually have a low local edge response and are not easily recognized.

[0005] In recent years, image semantic segmentation based on deep learning has achieved breakthrough results. Therefore, line detection algorithms combined with image semantic segmentation have also received increasing attention from scholars. The line detection algorithm based on image semantic segmentation can extract different objects in the image, and then no redundant line detection is performed on the same object. Moreover, the straight line detection effect of the boundary can be further enhanced by relying on the region boundary of image segmentation.

[0006] LSD can be regarded as a kind of image structure extraction. As is well known, most images have a low-rank or approximately low-rank structure. For images, methods based on nuclear norm minimization are usually used to regularize the underlying matrix with an approximately low-rank structure. However, these methods may obtain suboptimal performance because the nuclear norm assumes that each non-zero singular value has an equal contribution. At the same time, the truncated nuclear norm is closer to the matrix rank than the nuclear norm.

[0007] As mentioned above, LSD focuses on image structure extraction, especially non-orthogonal structure information is more important for complete line segment detection. In short, most previous work is based on "complete" images through heuristic algorithms or convolutional neural network (CNN) training. Inspired by the success of CNN in image semantic segmentation, in order to avoid the complex design of heuristic algorithms, we re-think the essential attributes of images and propose a learning region attraction method based on truncated nuclear norm for line segment detection.

[0008] After patent query statistics, there are already many patents related to line detection algorithms or applications: for example, a method and system for detecting vertical lines in images (CN202111040211.4), an adaptive line detection method based on Hough transform (CN202110960406.4), a method and device for detecting glass bottle defects by line detection based on deep learning (CN202110050012.5).

[0009] However, the existing line detection algorithms or applications have problems such as unclear contour segmentation and redundant detection in non-boundary regions of image objects during the line detection process. Summary of the Invention

[0010] The purpose of the present invention is to provide a method, device, and storage medium for detecting boundary line segments of image regions in view of the above problems existing in the prior art.

[0011] The first object of the present invention can be achieved by the following technical solutions: A method for detecting boundary line segments of image regions, characterized by comprising the following steps:

[0012] Step S1: Calculate the truncated nuclear norm of the image set;

[0013] Step S2: Input the image set with labeled boundary line segments of the region as the training sample set,

[0014] - Step S2.1: Train the training sample set with a deep neural network;

[0015] Step S3: Obtain the trained neural network capable of detecting boundary line segments of the image.

[0016] Working principle of the present invention: The truncated nuclear norm calculation can highlight the structural details of the image set, thereby improving the training effect of the neural network, so that the contour segmentation obtained when the trained neural network performs image boundary line segment detection is obvious, and the non-boundary area of the image object is clear and concise.

[0017] In the above image region boundary line segment detection method using truncated nuclear norm regularization constraint, step S2 further includes step S2.2: performing image region boundary squeezing on the training sample set.

[0018] In the above image region boundary line segment detection method using truncated nuclear norm regularization constraint, the training sample set includes an image and the horizontal and vertical coordinates of the starting point and ending point of the contour line segment in the image, the line segment intersection point, and the included angle of the line segment intersection point.

[0019] In the above image region boundary line segment detection method using truncated nuclear norm regularization constraint, let the training sample set be X = [X 1 , X 2 , …, X c ∈ R m×n ;

[0020] Let UAV’ be the SVD decomposition of the real matrix X ∈ R m×n , where U = (u 1 ,..., u m ) ∈ R m×m and V = (v 1 ,..., v m ) ∈ R n×n are orthogonal matrices to each other, A ∈ R m×n is a diagonal matrix, and the elements on the diagonal are arranged from large to small, which are the singular values of the matrix X. The nuclear norm of the matrix is defined as where σ i (x) is the i-th singular value of the matrix X. The truncated calculation of the nuclear norm is:

[0021]

[0022] s.t. P Ω (X) = P Ω (M)

[0023] where C ∈ R r×m , D ∈ R r×n are the truncated matrices of the matrices U and V respectively, and r is the number of truncated singular values.

[0024] In the above image region boundary line segment detection method using truncated nuclear norm regularization constraint, step S2.2: accurately segment the image region using the region squeezing model of the line segment.

[0025] In the above-mentioned method for detecting image region boundary line segments with truncated nuclear norm regularization constraint, let the region boundary line segments in the image be L = {l 1 ,..., l n}, and let each pixel in the image be p.

[0026] The pixel p makes a perpendicular line to each line segment region boundary line segment l i .

[0027] When the perpendicular line of the pixel p to the line segment l i falls on the line segment l i , the length of the perpendicular line is counted as the distance d(p, l i ) between the pixel p and the line segment l i .

[0028] When the perpendicular line of the pixel p to the line segment l i falls on the extension line of the line segment li, then directly calculate the distance between the pixel p and the endpoint of the line segment l i . The distance between the pixel p and the endpoint of the adjacent line segment l is counted as the distance d(p, l i ) between the pixel p and the line segment l i . i )

[0029] Select the minimum value of the distance d(p, l i ) between the pixel p and the line segment l i , and assign the pixel p to the region of the line segment l i with the closest distance;

[0030] Obtain the region segmentation result.

[0031] In the above-mentioned method for detecting image region boundary line segments with truncated nuclear norm regularization constraint, the distance formula between the pixel p and the line segment li is:

[0032]

[0033] where t represents the ratio of the line segment from the starting point to the ending point.

[0034] In the above-mentioned method for detecting image region boundary line segments with truncated nuclear norm regularization constraint, the pixels p in the region of each line segment l i are given colors for distinction. When the perpendicular line of the pixel p to the line segment l i falls on the line segment l i , the first color is given. When the perpendicular line of the pixel p to the line segment l i falls on the extension line of the line segment l i , the second color is given.

[0035] The second object of the present invention can be achieved by the following technical solution: An image region boundary line segment detection device includes an input end, an output end, one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the programs include those for executing the above-mentioned image region boundary line segment detection method.

[0036] The third object of the present invention can be achieved by the following technical solution: A computer-readable storage medium stores a computer program, and the computer program can be executed by a processor to complete the above-mentioned image region boundary line segment detection method.

[0037] Compared with the prior art, the present invention has the advantages of obvious contour segmentation and clarity and conciseness in the non-boundary regions of image objects during the straight line detection process. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 is a schematic diagram of the logic flow of the present invention.

[0039] Figure 2 is a schematic diagram of the deep neural network structure of the present invention.

[0040] Figure 3 is a schematic diagram of the region extrusion model of the line segments of the present invention.

[0041] Figure 4 is the first test image for inputting the trained neural network in the present invention.

[0042] Figure 5 is the first test image output after the trained neural network in the present invention performs image region boundary line segment detection.

[0043] Figure 6 is the second test image for inputting the trained neural network in the present invention.

[0044] Figure 7 is the second test image output after the trained neural network in the present invention performs image region boundary line segment detection. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0045] The following are specific embodiments of the present invention in combination with the accompanying drawings to further describe the technical solutions of the present invention, but the present invention is not limited to these embodiments.

[0046] As Figure 1 shown, the present image region boundary line segment detection method includes the following steps:

[0047] Step S1: Calculate the truncated nuclear norm of the image set;

[0048] Step S2: Input the image set of the boundary line segments of the labeled regions as the training sample set.

[0049] - Step S2.1: Conduct deep neural network training on the training sample set.

[0050] - Step S2.2: Perform image region boundary squeezing on the training sample set.

[0051] Step S3: Obtain the trained neural network capable of performing image boundary line segment detection.

[0052] The region boundary line segments in the training sample set can be labeled before or after the truncated nuclear norm calculation.

[0053] Specifically, the training sample set includes the image and the horizontal and vertical coordinates of the starting and ending points of the contour line segments in the image, the line segment intersections, and the included angles of the line segment intersections.

[0054] Specifically, let the training sample set be X = [X1, X2, …, Xc] ∈ Rm×n;

[0055] Let the SVD decomposition of the real matrix X ∈ R m×n be UAV’, where U = (u1,..., um) ∈ R m×m and V = (v1,..., vm) ∈ R n×n are orthogonal matrices to each other, and A ∈ R m×n is a diagonal matrix, and the elements on the diagonal are arranged from large to small, which are the singular values of the matrix X. The nuclear norm of the matrix is defined as where σ i (x) is the i-th singular value of the matrix X. The truncated calculation of the nuclear norm is:

[0056]

[0057] s.t. P Ω (X) = P Ω (M)

[0058] where C ∈ R r×m , D ∈ R r×n are the truncated matrices of matrices U and V respectively, and r is the number of truncated singular values.

[0059] Taking the image Xi as an example, Xi is a three-dimensional color image. The nuclear norm truncated calculation is performed on each of the RGB three dimensions of Xi, and then the RGB three dimensions are recombined into a color image for neural network training.

[0060] For example: The input three-dimensional image is 3×320×320, representing an image with 320×320 pixels in the RGB three dimensions.

[0061] The structure diagram of the deep neural network is as follows Figure 2 shown, including a convolutional layer, a ReLU layer, and a pooling layer. The convolutional layer is provided with input channels, input channels, convolutional kernels, and other default settings, including stride (default "1"), padding (default "no"), dilation parameter (default "1"), group parameter (default "1"), bias parameter (default "1"), padding mode (default "zeros"). The default parameters are the same in the network structure.

[0062] For example: the convolutional layer is 64×256×1×1, where 64 is the input channel, 256 is the output channel, 1×1 is the convolutional kernel, and the rest of the parameters adopt the default values.

[0063] As Figure 3 shown:

[0064] Step S2.2: Use the region squeezing model of line segments to accurately segment the image region.

[0065] To elaborate further, let the region boundary line segments in the image be L = {l1,..., ln}, and let each pixel in the image be p.

[0066] The pixel p makes a perpendicular line to each line segment region boundary line segment li.

[0067] When the perpendicular line of the pixel p to the line segment li falls on the line segment li, the length of the perpendicular line is counted as the distance d(p, li) between the pixel p and the line segment li.

[0068] When the perpendicular line of the pixel p to the line segment li falls on the extension line of the line segment li, then directly calculate the distance between the pixel p and the endpoint of the line segment li The distance between the pixel p and the closer endpoint of the line segment li is counted as the distance d(p, li) between the pixel p and the line segment li.

[0069] Select the minimum value of the distance d(p, li) between the pixel p and the line segment li, and assign the pixel p to the region of the line segment li with the closest distance;

[0070] Obtain the region segmentation result.

[0071] To elaborate further, the distance formula between the pixel p and the line segment li is:

[0072]

[0073] where t represents the ratio of the line segment from the starting point to the ending point.

[0074] To elaborate further, pixels p in the region of each line segment li are assigned colors for differentiation. When the perpendicular from pixel p to line segment li falls on line segment li, the first color is assigned; when the perpendicular from pixel p to line segment li falls on the extension of line segment li, the second color is assigned.

[0075] The first color is black and the second color is red.

[0076] By dividing the region, the semantic segmentation effect in the training model can be enhanced, thereby improving the result of line detection, making the contour segmentation obvious and the non-boundary regions of the image objects concise.

[0077] As Figures 4-7 shown, the trained neural network can detect the boundary line segments of the image.

[0078] This image region boundary line segment detection device includes an input end, an output end, one or more processors, a memory, and one or more programs, where the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the programs include those for executing the above-mentioned image region boundary line segment detection method.

[0079] This computer-readable storage medium stores a computer program, and the computer program can be executed by a processor to complete the above-mentioned image region boundary line segment detection method.

[0080] The specific embodiments described herein are merely illustrative of the spirit of the present invention. Those skilled in the art of the present invention can make various modifications or supplements to the described specific embodiments or use similar ways to substitute them, but will not deviate from the spirit of the present invention or exceed the scope defined by the appended claims.

[0081] Although a large number of terms are used herein, the possibility of using other terms is not excluded. These terms are used only to more conveniently describe and explain the essence of the present invention; interpreting them as any additional limitation is contrary to the spirit of the present invention.

Claims

1. A method for detecting boundary line segments of an image region, characterized in that, it includes the following steps: Step S1: Calculate the truncated nuclear norm of the image set; Step S2: Input the image set with labeled boundary line segments of the region as the training sample set, - Step S2.1: Train the deep neural network on the training sample set and perform image region boundary squeezing on the training sample set; Step S2.2: Use the region squeezing model of the line segment to accurately segment the image region; Let the boundary segments of the regions in the image be \(L = \{l 1 ,\cdots, l n \}\), and let each pixel in the image be \(p\). Pixel p makes a perpendicular line to each boundary line segment l of the line segment region i and When the perpendicular line from pixel p to line segment li falls on line segment l i , the length of the perpendicular line is counted as the distance d(p, l i ) between pixel p and line segment li When the perpendicular line from pixel p to line segment l i falls on the extension of line segment l i , the distance between pixel p and the endpoints of line segment li is directly calculated, and the distance between pixel p and the nearer endpoint of line segment l i is counted as the distance d(p, l i ) between pixel p and line segment l i ). Select the minimum distance d(p, l) between the pixel p and the line segment l i and assign the pixel p to the region of the line segment l i with the closest distance; i ​ Obtain the region segmentation result; Step S3: Obtain the trained neural network capable of detecting the image boundary line segment; Let the training sample set be ; Let UAV’ be the SVD decomposition of the real matrix X ∈ R m×n , where U = (u 1 ,..., u m ) ∈ R m×m and V = (v 1 ,..., v m ) ∈ R n×n are orthogonal matrices to each other, A ∈ R m×n is a diagonal matrix, and the elements on the diagonal are arranged from large to small, which are the singular values of the matrix X. The nuclear norm of the matrix is defined as , where is the i-th singular value of the matrix X. The truncated calculation of the nuclear norm is: s.t.P Ω (X) = P Ω (M) where \(C\in R\) r×m , \(D\in R\) r×n , are the truncated matrices of matrices \(U\) and \(V\) respectively, and \(r\) is the number of truncated singular values.

2. A method for detecting boundary line segments of an image region according to claim 1, characterized in that, the training sample set includes the image and the horizontal and vertical coordinates of the starting point and the ending point of the contour line segment in the image, the line segment intersection point, and the included angle of the line segment intersection point.

3. A method for detecting boundary line segments of an image region according to claim 2, characterized in that, The distance formula between pixel p and line segment l i is as follows: where t , represents the ratio of the line segment from the starting point to the ending point.

4. A method for detecting boundary line segments of an image region according to claim 3, characterized in that, Color the pixels p in the region of each line segment l i for differentiation. When the perpendicular line from the pixel p to the line segment l i falls on the line segment l i , assign the first color. When the perpendicular line from the pixel p to the line segment l i falls on the extension of the line segment l i , assign the second color.

5. An apparatus for detecting boundary line segments of an image region, including an input end, an output end, one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the programs include a method for detecting boundary line segments of an image region according to any one of claims 1-4.

6. A computer-readable storage medium storing a computer program, where the computer program can be executed by a processor to complete a method for detecting boundary line segments of an image region according to any one of claims 1-4.

Images