A GPU-based method for image line extraction

Abstract

The present invention relates to the technical field of image data processing, and particularly relates to a method for extracting image lines based on a GPU. The method includes: calculating the convolution of the first-order partial derivative and the second-order partial derivative of a two-dimensional Gaussian function with an image; calculating the line normal direction according to the Hessian matrix of a two-dimensional discrete image, performing a Taylor expansion on the gray distribution function along the normal direction to obtain the sub-pixel position of the line center point; connecting the contour points according to connection rules based on information such as the distance between adjacent contour points and the normal direction, and extracting the lines in the image. The present invention provides a method for extracting image lines based on a GPU, which can accurately and efficiently extract bright and dark lines in an image.

Description

Technical Field

[0001] The present invention relates to the technical field of image data processing, and particularly relates to a method for extracting image lines based on a GPU. Background Art

[0002] In the light stripe image collected by a camera, due to the characteristics of the light source or the object itself, in the direction normal to the light stripe, the light intensity of the light stripe cross-section approximately follows a Gaussian distribution. Therefore, the position of the extreme point of each light stripe cross-section can be calculated and connected according to rules, so as to extract the light stripe in the digital image.

[0003] For the extraction of lines, it is necessary to first calculate the convolution of the first-order partial derivative, second-order partial derivative of the two-dimensional Gaussian function and the image, and then calculate the line normal direction and the extreme point. For images with a small number of lines, the calculation of these convolutions, line normal directions and extreme points occupies most of the time-consuming of the line extraction method. For medium-sized pictures, a good acceleration ratio can be obtained by accelerating with the AVX instruction set on an x64 processor, but the acceleration ability is very limited for large-sized images. Therefore, how to accelerate the line extraction speed for large-sized images is a technical problem to be solved. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a method with a faster line extraction speed for large-sized images, so as to improve the image line extraction speed.

[0005] The technical solution adopted by the present invention is as follows: A method for extracting image lines based on a GPU, and its specific technical solution is as follows:

[0006] A method for extracting image lines based on a GPU includes the following steps:

[0007] Step S1: Calculate the convolution of the first-order partial derivative, second-order partial derivative of the two-dimensional Gaussian function and the image;

[0008] Step S2: Calculate the line normal direction according to the Hessian matrix of the two-dimensional discrete image, and perform a Taylor expansion on the gray distribution function along the normal direction to obtain the sub-pixel position of the line center point;

[0009] Step S3: Connect the contour points according to the distance between adjacent contour points and the normal direction information, and extract the lines in the image according to the connection rules.

[0010] Further, the specific steps of step S1 include:

[0011] Step S1.1: Calculate the convolution kernel size according to the size of the input Gaussian smoothing parameter Sigma;

[0012] Step S1.2: Calculate the first-order partial derivatives of the two-dimensional Gaussian function and the second-order partial derivatives , generate a convolution kernel and perform convolution with the two-dimensional image respectively to generate convolution results and .

[0013] Furthermore, the specific steps of step S2 include:

[0014] Step S2.1: Calculate the maximum absolute eigenvalue of the two-dimensional discrete image Hessian matrix and its corresponding eigenvector, and preliminarily screen the center points of the lines according to whether the maximum absolute eigenvalue is higher than the low threshold and the bright and dark information of the extracted lines;

[0015] Step S2.2: Perform a second-order Taylor expansion on the image gray value distribution function along the normal direction of the current pixel , calculate the center point position t of the light strip in the section, which is the sub-pixel position of the extreme point, and use the points whose sub-pixel positions are still within the range of the current pixel as the final center points of the lines to be connected. Store the sub-pixel position coordinates, normal direction, and the maximum absolute eigenvalue higher than the high threshold of this point in a picture with the same size as the original picture.

[0016] Furthermore, the specific steps of step S3 include:

[0017] Step S3.1: Traverse each pixel of the eigenvalue image in step S2.2, sort the non-zero eigenvalues in descending order, and start from the first point higher than the high threshold that has not been connected to the line as the starting connection point of this line;

[0018] Step S3.2: Divide the normal angle into 8 directions, search for the adjacent line center points according to the normal angle of the current starting connection point, and at the same time divide the line center points into two parts, right and left, according to the normal direction with the current point as the center. The maximum number of line center points on each side is 3;

[0019] Step S3.3: According to the distance and normal direction information between adjacent contour points, continuously connect and extend the line in the same way as calculating the adjacent line center points of each point in step S3.2 until there are no line center points to connect at the end or connect to the endpoints or midpoints of other lines, and complete the extraction of the current line;

[0020] Step S3.4: Start from the next point that has not been connected to the line as the starting connection point according to the order of the eigenvalues in step S3.1, execute steps S3.2 and S3.3, and complete the extraction of the next line; repeat steps S3.1, S3.2, and S3.3 until all starting connection points are traversed;

[0021] Step S3.5: Remove the lines with less than 2 points to obtain the finally filtered result.

[0022] Furthermore, the Hessian matrix can be expressed in the following form:

[0023] ,

[0024] where, is the result of the convolution of the second-order partial derivative of the two-dimensional Gaussian function in Step S1 and the image.

[0025] Furthermore, the two eigenvalues of the Hessian matrix can be expressed as:

[0026] ,

[0027] The corresponding eigenvector can be expressed as:

[0028] .

[0029] Furthermore, the second-order Taylor expansion of the image gray value distribution function along the normal direction of the current pixel can be expressed in the following form:

[0030] ,

[0031] where represents the result of the convolution of the first-order partial derivative of the two-dimensional Gaussian function and the image, represents the gray value of the current pixel , is the result of the convolution of the second-order partial derivative of the two-dimensional Gaussian function in Step S1 and the image, are the x and y components of the eigenvector, and t is the central point position of the light stripe in the profile.

[0032] Furthermore, the central point position t of the light stripe in the profile is expressed as:

[0033] ,

[0034] where are the x and y components of the eigenvector, t is the central point position of the light stripe in the profile, represents the result of the convolution of the first-order partial derivative of the two-dimensional Gaussian function and the image, is the result of the convolution of the second-order partial derivative of the two-dimensional Gaussian function in Step S1 and the image.

[0035] In the present invention, the two-dimensional filter is split into a row filter and a column filter, and some row filters are split into the same form, effectively reducing the amount of calculation. Image filtering and the extraction of sub-pixel center points are both completed on the GPU, effectively accelerating the extraction speed of the lines in the image. Especially for images of larger sizes, obvious acceleration effects can be achieved. For lines whose cross-sectional light intensity approximately follows a Gaussian distribution, the present invention can efficiently and accurately extract them, having high application value in practical applications. Description of the Drawings

[0036] Figure 1 It is a schematic flowchart of the method of the present invention;

[0037] Figure 2 It is a schematic diagram of the division result of each image after row filtering into 4 parts in the embodiment of the present invention;

[0038] Figure 3 It is a schematic diagram of the process of performing calculations and data transmissions using 4 CUDA streams in the embodiment of the present invention;

[0039] Figure 4 It is a schematic diagram of the search range of adjacent connection points in 8 different normal directions in the embodiment of the present invention;

[0040] Figure 5 It is a schematic diagram of the extraction result of bright lines in the embodiment of the present invention. Detailed Embodiment

[0041] The present invention will be further clarified below in conjunction with the drawings and the detailed embodiment. The following detailed embodiment is only used to illustrate the present invention and not to limit the scope of the present invention.

[0042] Figure 1 It is a schematic flowchart of the method of the present invention, including the following steps:

[0043] Step S1: Calculate the convolution of the first-order partial derivative, second-order partial derivative of the two-dimensional Gaussian function and the image;

[0044] Step S1 specifically includes:

[0045] Step S1.1: Calculate the convolution kernel size according to the magnitude of the input Gaussian smoothing parameter Sigma;

[0046] Step S1.2: Calculate the first-order partial derivative and the second-order partial derivative of the two-dimensional Gaussian function, generate convolution kernels and respectively perform convolution with the two-dimensional image to generate convolution results and .

[0047] Step S2: Calculate the line normal direction based on the Hessian matrix of the two-dimensional discrete image, perform a Taylor expansion on the gray distribution function along the normal direction, and obtain the sub-pixel position of the line center point;

[0048] Step S2 specifically includes:

[0049] Step S2.1: Calculate the maximum absolute eigenvalue of the Hessian matrix of the two-dimensional discrete image and its corresponding eigenvector, and based on the maximum absolute eigenvalue whether it is higher than the low threshold and the bright and dark information of the extracted line, and preliminarily screen the possible line center points;

[0050] Step S2.2: Perform a second-order Taylor expansion on the image gray value distribution function along the normal direction of the current pixel to calculate the center point position t of the light strip in the profile, which is the sub-pixel position of the extreme point. Take the points whose sub-pixel positions are still within the current pixel range as the final line center points to be connected. Store the sub-pixel position coordinates, normal direction, and the maximum absolute eigenvalue higher than the high threshold of this point in a picture of the same size as the original image.

[0051] Step S3: According to information such as the distance between adjacent contour points and the normal direction, connect the contour points according to the connection rules to extract the lines in the image;

[0052] Step S3 specifically includes:

[0053] Step S3.1: Traverse each pixel of the eigenvalue image in Step S2.2, sort the non-zero eigenvalues in descending order, and start from the first point higher than the high threshold that has not been connected to the line in this order as the starting connection point of this possible line;

[0054] Step S3.2: Divide the normal angle into 8 directions. According to the normal angle of the current starting connection point, search for its adjacent line center points in the corresponding direction. At the same time, divide the line center points into two parts, the right side and the left side, according to the normal direction with the current point as the center. The maximum number of line center points on each side is 3;

[0055] Step S3.3: According to the distance between adjacent contour points and the normal direction information, continuously connect and extend the line in the method of calculating the adjacent line center points of each point in Step S3.2 until there are no line center points to connect at the end or it is connected to the end point or intermediate point of other lines, and complete the extraction of the current line;

[0056] Step S3.4: Starting from the next point that has not been connected to the line as the starting connection point, in the order of the eigenvalues in Step S3.1, execute Steps S3.2 and S3.3 to complete the extraction of the next line; repeat Steps S3.1, S3.2, and S3.3 until all starting connection points have been traversed;

[0057] Step S3.5: Remove the lines with less than 2 points to obtain the final filtered result.

[0058] In a digital image with lines, the gray value distribution along the normal direction of the line approximately follows a Gaussian distribution, and the extreme points of the line profile are the potential line center points to be filtered. Connecting the qualified line center points according to the rules can obtain the lines in the image.

[0059] To calculate the direction vector of the line normal, it is necessary to introduce the Hessian matrix of the two-dimensional discrete image to determine the direction. The eigenvector corresponding to the largest absolute eigenvalue of the Hessian matrix is the normal vector of the line. This matrix can be expressed in the following form:

[0060] ,

[0061] where represents the result of the convolution of the second-order partial derivative of the two-dimensional Gaussian function and the image. Calculate the corresponding Hessian matrix for each point on the image. The two eigenvalues and the corresponding eigenvectors can be expressed as:

[0062] ,

[0063] ,

[0064] For the pixel point, its corresponding sub-pixel coordinates of the line center are:

[0065] ,

[0066] where are the x and y components of the line normal vector (i.e., the eigenvector).

[0067] The second-order Taylor expansion of the gray value distribution function of the current pixel along the normal direction can be expressed in the following form:

[0068] ,

[0069] where represents the result of the convolution of the first-order partial derivative of the two-dimensional Gaussian function with the image, represents the current pixel and its grayscale value. Taking the derivative of the above formula with respect to t, the position t where the derivative is 0 is the center point of the light strip in this profile. The calculation method for this position is as follows:

[0070] ,

[0071] The center point position t of the line can be calculated as:

[0072] ,

[0073] Record the sub-pixel points of the line centers that meet the requirements in the entire image and connect them according to the rules, and the lines in the entire image can be extracted.

[0074] The following is the specific calculation process of an embodiment:

[0075] First, determine the size of the filtering kernel of the partial derivative of the two-dimensional Gaussian function according to the input value:

[0076] ,

[0077] Since these two first-order partial derivatives and three second-order partial derivative filtering kernels are separable, they can be split into row and column filtering kernels as follows:

[0078] ,

[0079] ,

[0080] ,

[0081] ,

[0082] ,

[0083] ,

[0084] ,

[0085] ,

[0086] ,

[0087] ,

[0088] Complete the creation of the one-dimensional filtering kernel according to the above formula.

[0089] In this embodiment, CUDA programming is used to implement the calculations on the GPU. The host processor used for testing is an Intel Core i5 13600K, and the device is an NVIDIA GeForce GTX 1650.

[0090] First, apply for 5 image memories on the host side with the same size as the image to be processed. Among them, the pixel types of 4 images are float, which are used to store the x - coordinate, y - coordinate, normal angle, and the absolute value of the eigenvalue higher than the high threshold of the extracted light - strip center points respectively; the pixel type of 1 image is unsigned char, which stores the connection information of the extracted points. Then, apply for the global memory of the image to be processed and the size of the one - dimensional filtering kernel on the host side on the device side, and complete the transfer of the image and the filtering kernel from the host to the device. Note that when the filtering kernel is split into row and column filtering, dx row is the same as dxy row , and dy row is the same as dyy row . You can first perform the row - filtering calculation on the original image to reduce the 5 row - filtering calculations to 3.

[0091] In the kernel function filter_row for row filtering, the thread - block size is set to 128. Each thread corresponds to a pixel to be processed in the image. The boundary is filled with the boundary pixel values. The temporary accumulation variable during filtering is stored in the register memory, and after the accumulation calculation, the result is written into the global memory on the device side used to store the row - filtering calculation results.

[0092] After completing the 3 row - filterings of the image, perform the 5 column - filterings and the extraction of information such as the position of the line center point. These calculations are all completed within a kernel function points_cal_stream, and the calculation method used is the above formula. If the absolute values of the two eigenvalues of the Hessian matrix are the same, it indicates that there is no significant normal direction at that place, that is, there is no line center point here. When calculating the position t of the line center point, only when the calculated t satisfies the following two formulas simultaneously does it indicate that this point is a reasonable line center point:

[0093] ,

[0094] At this time, it indicates that the position of the sub - pixel extreme point still lies within the current pixel and is a reasonable line center point. If the maximum absolute eigenvalue is greater than the low threshold and meets the sign requirements (the eigenvalue should be negative for bright lines and positive for dark lines), then this point is used as the final line center point passing the screening. At this time, the x - coordinate, y - coordinate, normal angle, and the connection information of the point are stored in the corresponding global memory on the device side. The normal angle ɑ can be calculated in the following way:

[0095] ,

[0096] If the maximum absolute eigenvalue is greater than the high threshold, it is stored in the global memory of the eigenvalue image on the device side.

[0097] For the pixel points where no reasonable extreme points are extracted, the corresponding points on the x - coordinate, y - coordinate, and normal angle images are stored as - 1 to mark here, and the connection information of the points and the corresponding points on the eigenvalue image are stored as 0.

[0098] In the calculation of the kernel function points_cal_stream, each thread independently calculates the information of its corresponding pixel position. Since the calculations between different parts of the image are independent, first create N CUDA streams and divide each image after row filtering into N parts, and different parts are handed over to different CUDA streams for processing. As Figure 2 shown is the segmentation situation when N = 4; then register the memory parts of the 5 pictures allocated in the host memory as pinned memory to achieve the synchronous execution of the calculation of the kernel function points_cal_stream and the process of transferring the result pictures from the device to the host between different CUDA streams. As Figure 3 shown is the process of 4 CUDA streams performing calculations and data transmissions when N = 4. It can be seen that the data transmission time is less than the calculation time, and the data transmission time can be masked by the calculation time. By creating an appropriate number of CUDA streams, the transmission time can be effectively masked. After completing all calculations and data transmissions, the device - side memory can be released, and the remaining calculation tasks are completed by the host side.

[0099] The eigenvalue image stores the eigenvalues higher than the high threshold. Each point with an eigenvalue higher than the high threshold is used as a possible starting point for line connection. Starting from these points, connect to the points with eigenvalues higher than the low threshold, traverse the eigenvalue image and sort these eigenvalues from large to small, and perform the connection in this order.

[0100] For each extracted center point, its normal angle is divided into 8 directions. According to the normal direction of the current point, determine the search direction of the next point to be connected to this point. Divide the points to be searched into two parts, the right side and the left side, according to the normal direction with the current point as the center. The maximum number of points to be searched on each side is 3. As Figure 4 shown, is the search range of adjacent connection points for 8 different normal directions. The pixels marked as 1, 2, and 3 in the figure are the connection points to be searched on the right side of the current point P, and the pixels marked as 4, 5, and 6 are the connection points to be searched on the left side of the current point P. If point P is at the edge of the image or there are no connection points higher than the low threshold within its search range, the number of connection points to be calculated on each side may be less than 3.

[0101] Starting from the current point P, the order of connecting and extending the line is as follows: first, connect and extend to the right until there is no center point of the line to connect at the end or connect to the endpoint or midpoint of other lines, and then connect and extend to the left until there is no center point of the line to connect at the end or connect to the endpoint or midpoint of other lines.

[0102] Taking Figure 4 the case of connecting to the right when -22.5° < ɑ ≤ 22.5° as an example, among the pixels marked as 1, 2, and 3 on the right, the point that will ultimately be selected to connect to P should meet the following requirements: the difference in the normal angles between the current point P and the point to be connected should be less than , and among the points that meet this requirement, select the point with the smallest sum of the weights of the distance d and the difference in angles as the final connection point to the right of point P. The difference in angles and the sum of the weights of the distance and the difference in angles can be calculated by the following formulas:

[0103] ,

[0104] ,

[0105] ,

[0106] where the weight coefficient takes a value of 1.0 in this embodiment.

[0107] After completing the selection of the point to be connected, execute the corresponding connection logic according to the current connection status of the point to be connected. There are a total of 5 connection statuses for the point to be connected, and these 5 statuses and their corresponding connection logics are as follows:

[0108] 1. This point to be connected has not been added to any line: At this time, add this point to the line where point P is located, and at the same time mark this point as the midpoint of the line that has been added to the line. Continue to search for the points to be connected to this point. If none exist, mark this point as the endpoint of the line again.

[0109] 2. This point to be connected belongs to the endpoint of a certain line, and the normal angle meets the requirements for connection: At this time, connect this line to that line without adding a new point. When connecting, insert the line where the current point P is located into the beginning position of the new line in order, delete this line, and mark the line number to which the center point on this line belongs as the new line number.

[0110] 3. This point to be connected belongs to the endpoint of a certain line, and the normal angle does not meet the requirements: At this time, add this new point to this line. This line and the found line are not connected together, and at the same time mark this point as a multi-junction point, indicating that this point is the endpoint of multiple lines.

[0111] 4. The connection point to be connected belongs to the midpoint of a certain line: At this time, the line should be divided into two from this point, and a new line number is separated to store the divided half of the line, and at the same time, this division point is marked as a multi-junction point.

[0112] 5. The connection point to be connected belongs to a multi-junction point: Add this point to this line and end the search.

[0113] Repeat the above steps until each starting search point has been traversed, and then remove the lines with less than 2 points, that is, the lines extracted from the final image are obtained. As Figure 5 shown, the horizontal railings in (a) present a bright line pattern, and the extracted bright lines are shown in (b). It can be clearly seen from the extraction results that the horizontal railings presenting a bright line pattern are accurately extracted, and for this large-size image, there is still a very fast calculation speed, fully reflecting the efficient and accurate characteristics of the present invention in extracting lines from large-size images.

[0114] The above embodiments are only used to illustrate the technical means adopted by the present invention and are not limited to the above technical means. The technical means derived, modified, or recombined from the above methods are still within the protection scope of the present invention.

Claims

1. A GPU-based image line extraction method, characterized in that: The following steps are involved: Step S1: Calculate the convolution of the first-order partial derivative and the second-order partial derivative of the two-dimensional Gaussian function with the image; Step S2: Calculate the normal direction of the line according to the Hessian matrix of the two-dimensional discrete image, perform Taylor expansion on the grayscale distribution function along the normal direction, and obtain the sub-pixel position of the center point of the line; Step S3: According to the distance between adjacent contour points and normal direction information, the contour points are connected according to the connection rules to extract the lines in the image; The step S2 specifically includes: Step S2.1: Calculate the maximum absolute eigenvalue and the corresponding eigenvector of the Hessian matrix of the two-dimensional discrete image, and preliminarily screen the center points of the lines according to whether the maximum absolute eigenvalue is higher than the low threshold and the brightness and darkness information of the extracted lines; Step S2.2: For the current pixel Perform a second-order Taylor expansion on the grayscale distribution function of the image along the normal direction, calculate the center point position t of the light strip in the cross section, which is the sub-pixel position of the extreme point, and take the point where the sub-pixel position is still within the current pixel range as the center point of the final line to be connected. Store the sub-pixel position coordinates, normal direction, and maximum absolute eigenvalue above the high threshold of the point in a picture of the same size as the original image; The step S3 specifically includes: Step S3.1: traverse each pixel of the eigenvalue image in step S2.2, sort the non-zero eigenvalues ​​in descending order, and start from the first point above the high threshold that has not been connected to the line in this order as the starting connection point of this line; Step S3.2: Divide the normal angle into 8 directions, and search for the center point of the line adjacent to the current starting connection point according to the normal angle in the corresponding direction. At the same time, divide the center point of the line into two parts, the right side and the left side, with the current point as the center and the normal direction. The number of the center points of the line on each side is at most 3; Step S3.3: According to the distance and normal direction information between adjacent contour points, the extended lines are continuously connected according to the method of calculating the center point of the adjacent lines of each point in step S3.2, until there is no line center point at the end that can be connected or connected to the end point or middle point of other lines, thus completing the extraction of the current line; Step S3.4: According to the order of the feature values ​​in step S3.1, starting from the next point that has not been connected to the line as the starting connection point, execute steps S3.2 and S3.3 to complete the extraction of the next line; repeat steps S3.1, S3.2 and S3.3 until all the starting connection points have been traversed; Step S3.5: remove lines with less than 2 points to obtain the final screening result; Current Pixel( , )The second-order Taylor expansion of the image gray value distribution function along the normal direction can be expressed as follows: ， in Represents the result of convolution of the first-order partial derivative of the two-dimensional Gaussian function with the image. Represents the current pixel The gray value of is the result of convolution of the second-order partial derivative of the two-dimensional Gaussian function with the image in step S1 are the x and y components of the eigenvector, and t is the center position of the light strip in the cross section.

2. The GPU-based image line extraction method according to claim 1, characterized in that: The step S1 specifically includes: Step S1.1: Calculate the convolution kernel size according to the input Gaussian smoothing parameter Sigma; Step S1.2: Calculate the first-order partial derivative of the two-dimensional Gaussian function and the second-order partial derivatives , generate the convolution kernel and convolve it with the two-dimensional image to generate the convolution result and.

3. The GPU-based image line extraction method according to claim 1, characterized in that: The Hessian matrix can be expressed as follows: ， in, is the result of convolution of the second-order partial derivative of the two-dimensional Gaussian function with the image in step S1.

4. The GPU-based image line extraction method according to claim 1, characterized in that: Two eigenvalues ​​of the Hessian matrix It can be expressed as: ， The corresponding eigenvector It can be expressed as: 。 5. The GPU-based image line extraction method according to claim 1, characterized in that: The position t of the center point of the light strip in the cross section is expressed as: ， in are the x and y components of the eigenvector, t is the center point of the light strip in the cross section, Represents the result of convolution of the first-order partial derivative of the two-dimensional Gaussian function with the image. is the result of convolution of the second-order partial derivative of the two-dimensional Gaussian function with the image in step S1.

Images