Image line extraction method based on GPU

Abstract

The invention relates to the technical field of image data processing, in particular to an image line extraction method based on a GPU. The method comprises the following steps: calculating convolution of a first-order partial derivative and a second-order partial derivative of a two-dimensional Gaussian function and an image; calculating a line normal direction according to a Hessian matrix of the two-dimensional discrete image, and performing Taylor expansion on the gray level distribution function along the normal direction to obtain a line center point sub-pixel position; and connecting the contour points according to the distance between the adjacent contour points, the normal direction and other information and a connection rule, and extracting lines in the image. The invention provides an image line extraction method based on a GPU (Graphics Processing Unit), which can accurately and efficiently extract bright and dark lines in an image.

Description

Technical Field

[0001] The present invention relates to the technical field of image data processing, and in particular to an image line extraction method based on a GPU. Background Art

[0002] In the light strip image captured by the camera, due to the characteristics of the light source or the object itself, the light intensity of the light strip cross section along the normal direction of the light strip approximately obeys the Gaussian distribution. Therefore, the position of the extreme points of each light strip cross section can be calculated and connected according to the rules, thereby extracting the light strips in the digital image.

[0003] To extract lines, it is necessary to first calculate the convolution of the first-order partial derivative and the second-order partial derivative of the two-dimensional Gaussian function with the image, and then calculate the normal direction and extreme points of the lines. For images with a small number of lines, these convolutions, the calculation of the normal direction of the lines and the extreme points take up most of the time of the line extraction method. For medium-sized images, using the AVX instruction set acceleration on the x64 processor can achieve a good acceleration ratio, but the acceleration capability is very limited for large-size images. Therefore, how to speed up the line extraction speed for large-size images is a technical problem to be solved. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a method for extracting lines on a large-size image at a faster speed, so as to improve the speed of extracting lines from the image.

[0005] The technical solution adopted by the present invention is: a GPU-based image line extraction method, and its specific technical solution is as follows: A GPU-based image line extraction method comprises the following steps: Step S1: Calculate the convolution of the first-order partial derivative and the second-order partial derivative of the two-dimensional Gaussian function with the image; Step S2: Calculate the normal direction of the line according to the Hessian matrix of the two-dimensional discrete image, perform Taylor expansion on the grayscale distribution function along the normal direction, and obtain the sub-pixel position of the center point of the line; Step S3: According to the distance between adjacent contour points and normal direction information, the contour points are connected according to the connection rules to extract the lines in the image.

[0006] Furthermore, the step S1 specifically includes: Step S1.1: Calculate the convolution kernel size according to the input Gaussian smoothing parameter Sigma; Step S1.2: Calculate the first-order partial derivative of the two-dimensional Gaussian function and the second-order partial derivatives , generate the convolution kernel and convolve it with the two-dimensional image to generate the convolution result and .

[0007] Furthermore, the step S2 specifically includes: Step S2.1: Calculate the maximum absolute eigenvalue and the corresponding eigenvector of the Hessian matrix of the two-dimensional discrete image, and preliminarily screen the center points of the lines according to whether the maximum absolute eigenvalue is higher than the low threshold and the brightness and darkness information of the extracted lines; Step S2.2: For the current pixel The grayscale value distribution function of the image along the normal direction is expanded by the second order Taylor, and the center point position t of the light strip in the section is calculated, which is the sub-pixel position of the extreme point. The point whose sub-pixel position is still within the current pixel range is taken as the center point of the final line to be connected, and the sub-pixel position coordinates, normal direction, and maximum absolute eigenvalue above the high threshold of the point are stored in a picture of the same size as the original image.

[0008] Furthermore, the step S3 specifically includes: Step S3.1: traverse each pixel of the eigenvalue image in step S2.2, sort the non-zero eigenvalues ​​in descending order, and start from the first point above the high threshold that has not been connected to the line in this order as the starting connection point of this line; Step S3.2: Divide the normal angle into 8 directions, and search for the center point of the line adjacent to the current starting connection point according to the normal angle in the corresponding direction. At the same time, divide the center point of the line into two parts, the right side and the left side, with the current point as the center and the normal direction. The number of the center points of the line on each side is at most 3; Step S3.3: According to the distance and normal direction information between adjacent contour points, the extended lines are continuously connected according to the method of calculating the center point of the adjacent lines of each point in step S3.2, until there is no line center point at the end that can be connected or connected to the end point or middle point of other lines, thus completing the extraction of the current line; Step S3.4: According to the order of the feature values ​​in step S3.1, starting from the next point that has not been connected to the line as the starting connection point, execute steps S3.2 and S3.3 to complete the extraction of the next line; repeat steps S3.1, S3.2 and S3.3 until all the starting connection points have been traversed; Step S3.5: Remove the lines with less than 2 points to obtain the final screening result.

[0009] Furthermore, the Hessian matrix can be expressed as follows: , in, is the result of convolution of the second-order partial derivative of the two-dimensional Gaussian function with the image in step S1.

[0010] Furthermore, the two eigenvalues ​​of the Hessian matrix It can be expressed as: , The corresponding eigenvector It can be expressed as: .

[0011] Furthermore, the current pixel The second-order Taylor expansion of the image gray value distribution function along the normal direction can be expressed as follows: , in Represents the result of convolution of the first-order partial derivative of the two-dimensional Gaussian function with the image. Represents the current pixel The gray value of is the result of convolution of the second-order partial derivative of the two-dimensional Gaussian function with the image in step S1, are the x and y components of the eigenvector, and t is the center position of the light strip in the cross section.

[0012] Furthermore, the position t of the center point of the light strip in the cross section is expressed as: , in are the x and y components of the eigenvector, t is the center point of the light strip in the cross section, Represents the result of convolution of the first-order partial derivative of the two-dimensional Gaussian function with the image. is the result of convolution of the second-order partial derivative of the two-dimensional Gaussian function with the image in step S1.

[0013] In the present invention, the two-dimensional filter is split into row filter and column filter, and some row filters are split into the same form, which effectively reduces the amount of calculation; image filtering and sub-pixel center point extraction are completed on the GPU, which effectively speeds up the extraction of lines in the image, especially for larger-sized images, and can achieve a significant acceleration effect. For lines whose cross-sectional light intensity approximately follows Gaussian distribution, the present invention can extract them efficiently and accurately, and has high use value in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is a schematic diagram of the process of the present invention; Figure 2 Schematic diagram of the division result of dividing each image after row filtering into four parts in an embodiment of the present invention; Figure 3 A schematic diagram of a process of using four CUDA streams to perform calculations and data transmission in an embodiment of the present invention; Figure 4 Schematic diagram of the search range of adjacent connection points in normal directions of 8 different angles in an embodiment of the present invention; Figure 5 Schematic diagram of bright line extraction results in an embodiment of the present invention. DETAILED DESCRIPTION

[0015] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments. The following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0016] Figure 1 The schematic diagram of the process of the present invention comprises the following steps: Step S1: Calculate the convolution of the first-order partial derivative and the second-order partial derivative of the two-dimensional Gaussian function with the image; Step S1 specifically includes: Step S1.1: Calculate the convolution kernel size according to the input Gaussian smoothing parameter Sigma; Step S1.2: Calculate the first-order partial derivative of the two-dimensional Gaussian function and the second-order partial derivatives , generate the convolution kernel and convolve it with the two-dimensional image to generate the convolution result and .

[0017] Step S2: Calculate the normal direction of the line according to the Hessian matrix of the two-dimensional discrete image, perform Taylor expansion on the grayscale distribution function along the normal direction, and obtain the sub-pixel position of the center point of the line; Step S2 specifically includes: Step S2.1: Calculate the maximum absolute eigenvalue and the corresponding eigenvector of the Hessian matrix of the two-dimensional discrete image, and Whether it is higher than the low threshold, the brightness and darkness information of the extracted lines are used to preliminarily screen possible line center points; Step S2.2: For the current pixel The grayscale value distribution function of the image along the normal direction is expanded by the second order Taylor, and the center point position t of the light strip in the section is calculated, which is the sub-pixel position of the extreme point. The point whose sub-pixel position is still within the current pixel range is taken as the center point of the final line to be connected, and the sub-pixel position coordinates, normal direction, and maximum absolute eigenvalue above the high threshold of the point are stored in a picture of the same size as the original image.

[0018] Step S3: According to information such as the distance between adjacent contour points and the normal direction, the contour points are connected according to the connection rules to extract the lines in the image; Step S3 specifically includes: Step S3.1: traverse each pixel of the eigenvalue image in step S2.2, sort the non-zero eigenvalues ​​in descending order, and start from the first point above the high threshold that has not been connected to the line in this order as the starting connection point of this possible line; Step S3.2: Divide the normal angle into 8 directions, and search for the center point of the line adjacent to the current starting connection point according to the normal angle in the corresponding direction. At the same time, divide the center point of the line into two parts, the right side and the left side, with the current point as the center and the normal direction. The number of the center points of the line on each side is at most 3; Step S3.3: According to the distance and normal direction information between adjacent contour points, the extended lines are continuously connected according to the method of calculating the center point of the adjacent lines of each point in step S3.2, until there is no line center point at the end that can be connected or connected to the end point or middle point of other lines, thus completing the extraction of the current line; Step S3.4: According to the order of the feature values ​​in step S3.1, starting from the next point that has not been connected to the line as the starting connection point, execute steps S3.2 and S3.3 to complete the extraction of the next line; repeat steps S3.1, S3.2 and S3.3 until all the starting connection points have been traversed; Step S3.5: Remove the lines with less than 2 points to obtain the final screening result.

[0019] In a digital image with lines, the grayscale value of the profile along the normal direction of the line approximately follows a Gaussian distribution, and the extreme points of the line profile are potential center points of the lines to be screened. The lines in the image can be obtained by connecting the center points of the lines that meet the requirements according to the rules.

[0020] In order to calculate the direction vector of the line normal, it is necessary to introduce the Hessian matrix of the two-dimensional discrete image to determine the direction. The eigenvector corresponding to the maximum absolute eigenvalue of the Hessian matrix corresponds to the normal vector of the line. The matrix can be expressed as follows: , in, It represents the result of convolution of the second-order partial derivative of the two-dimensional Gaussian function with the image, and the corresponding Hessian matrix is ​​calculated for each point on the image. The two eigenvalues ​​of this matrix and the corresponding eigenvector It can be expressed as: , , for For a pixel point, the corresponding sub-pixel coordinate of the line center is for: , in, are the x and y components of the line normal vector (i.e. eigenvector).

[0021] Current pixel The second-order Taylor expansion of the image gray value distribution function along the normal direction can be expressed as follows: , in, Represents the result of convolution of the first-order partial derivative of the two-dimensional Gaussian function with the image. Represents the current pixel Gray value. The above formula is derived with respect to t. The position t where the derivative is 0 is the center point of the light strip in the section. The position is calculated as follows: , The position t of the center point of the line can be calculated as: , The sub-pixel points at the center of the lines in the entire image that meet the requirements are recorded and connected according to the rules, so that the lines in the entire image can be extracted.

[0022] The following is a specific calculation process of an embodiment: First, according to the input The value determines the filter kernel size of the partial derivative of the two-dimensional Gaussian function: , Since the two first-order partial derivatives and three second-order partial derivative filter kernels are separable, they can be split into row filter and column filter kernels as shown below: , , , , , , , , , , The creation of a one-dimensional filter kernel is completed according to the above formula.

[0023] In this embodiment, CUDA programming is used to implement computing on the GPU. The host processor used in the test is Intel Core i5 13600K, and the device processor is NVIDIA GeForce GTX 1650.

[0024] First, apply for 5 image memories of the same size as the image to be processed on the host side. The pixel type of 4 of the images is float, which respectively stores the x-coordinate, y-coordinate, normal angle, and absolute value of the eigenvalue above the high threshold of the extracted light strip center point; the pixel type of 1 image is unsigned char, which stores the connection information of the extracted points. Then apply for the global memory of the image to be processed and the size of the one-dimensional filter kernel on the host side on the device side, and complete the image and filter kernel transmission from the host to the device. Note that when the filter kernel is split into row and column filtering, dx row With dxy row Same, dy row With dyy row Similarly, the original image may be first subjected to line filtering calculations to reduce the five line filtering calculations to three.

[0025] In the kernel function filter_row for row filtering, the thread block size is set to 128. Each thread corresponds to a pixel to be processed in the image. The boundary is filled with the boundary pixel value. The temporary accumulation variable during filtering is stored in the register memory. After the accumulation calculation is completed, the result is written to the global memory on the device side for storing the row filtering calculation results.

[0026] After completing the three row filters of the image, the five column filters are calculated and the line center position and other information are extracted. These calculations are completed in a kernel function points_cal_stream, and the calculation method used is the above formula. If the absolute values ​​of the two eigenvalues ​​of the Hessian matrix are the same, it means that there is no significant normal direction at this location, that is, there is no line center point here. When calculating the line center point position t, only when the calculated t satisfies the following two formulas at the same time, it indicates that the point is a reasonable line center point: , This indicates that the position of the sub-pixel extreme point is still located in this pixel, and it is a reasonable line center point. If the maximum absolute eigenvalue is greater than the low threshold and meets the sign requirements (the eigenvalue should be negative for bright lines and positive for dark lines), then this point is used as the final line center point that passes the screening. At this time, the x-coordinate, y-coordinate, normal angle, and point connection information are stored in the corresponding global memory on the device side. The normal angle ɑ can be calculated as follows: , If the maximum absolute eigenvalue is greater than the high threshold, it is stored in the eigenvalue image global memory on the device.

[0027] For pixel points that have not been extracted to reasonable extreme points, the x-coordinate, y-coordinate, and corresponding point on the normal angle image are stored as -1 to mark this place, and the connection information of the point and the corresponding point on the eigenvalue image are stored as 0.

[0028] In the calculation of the kernel function points_cal_stream, each thread independently calculates the information of its corresponding pixel position. Since the calculations between different parts of the image are independent, first create N CUDA streams and divide each image after row filtering into N parts. Different parts are processed by different CUDA streams, such as Figure 2 The figure shows the segmentation when N=4. The memory portion where the five images are applied for in the host memory is registered as page-locked memory to achieve synchronous execution of the calculation of the kernel function points_cal_stream and the transmission of the result image from the device to the host between different CUDA streams, as shown in Figure 3 The figure shows the process of 4 CUDA streams performing calculations and data transfers when N=4. It can be seen that the data transfer time is less than the calculation time, and the data transfer time can be covered by the calculation time. The transfer time can be effectively covered by creating an appropriate number of CUDA streams. After all calculations and data transfers are completed, the device-side memory can be released, and the remaining calculation tasks are completed by the host side.

[0029] The eigenvalue image stores eigenvalues ​​higher than the high threshold. Each point with an eigenvalue higher than the high threshold is used as a possible starting point for line connection. Starting from these points, points with eigenvalues ​​higher than the low threshold are connected. The eigenvalue image is traversed and these eigenvalues ​​are sorted from large to small, and connected in this order.

[0030] For each extracted center point, its normal angle is divided into 8 directions. The search direction of the next point connected to the current point is determined according to the normal direction of the current point. The points to be searched are divided into two parts, the right side and the left side, with the current point as the center according to the normal direction. The number of points to be searched on each side is up to 3. Figure 4 As shown, the search range of adjacent connection points at 8 different angles of normal direction. The pixels marked as 1, 2 and 3 in the figure are the connection points to be searched on the right side of the current point P, and the pixels marked as 4, 5 and 6 are the connection points to be searched on the left side of the current point P. If point P is at the edge of the image or there are no connection points above the low threshold in its search range, the number of connection points to be calculated on each side may be less than 3.

[0031] The order of connecting and extending the lines starting from the current point P is: first connect and extend to the right until there is no line center point to connect to at the end or connect to the endpoints and middle points of other lines, and then connect and extend to the left until there is no line center point to connect to at the end or connect to the endpoints and middle points of other lines.

[0032] by Figure 4 For example, when connecting to the right side when -22.5°<ɑ≤22.5°, the point on the right side marked as 1, 2, and 3 that will be finally selected to connect to P should meet the following requirements: The difference between the normal angles of the current point P and the point to be connected Should be less than , select the difference between the distance d and the angle among the points that meet this requirement The point with the smallest weight and sum is used as the final connection point of point P to the right. The weight sum of the angle difference and the distance and angle difference can be calculated by the following formula: , , , The weight coefficient In this embodiment, the value is 1.0.

[0033] After completing the selection of the point to be connected, the corresponding connection logic is executed according to the current connection status of the point to be connected. There are 5 connection states of the point to be connected, and these 5 states and their corresponding connection logic are as follows: 1. The point to be connected has not been added to any line: At this time, add the point to the line where point P is located, and mark the point as the middle point of the added line. Continue to search for the point to be connected. If it does not exist, re-mark the point as the endpoint of the line.

[0034] 2. The point to be connected belongs to the endpoint of a line, and the normal angle meets the requirements and can be connected: At this time, connect the current line with the line without adding new points. When connecting, insert the current line where the current point P is located to the starting position of the new line in sequence, delete the current line and mark the line number of the center point on the line as the new line number.

[0035] 3. The point to be connected is the endpoint of a line, and the normal angle does not meet the requirements: At this time, the new point is added to the current line, and the current line and the found line are not connected together. At the same time, the point is marked as a multi-fork point, indicating that the point is the endpoint of multiple lines.

[0036] 4. The point to be connected is the middle point of a line: At this time, the line should be divided into two from this point, and a new line number should be generated to store the divided half of the line. At the same time, the division point should be marked as a multi-fork point.

[0037] 5. The point to be connected is a multi-fork intersection: add the point to this line and end the search.

[0038] Repeat the above steps until each starting search point has been traversed, and then remove the lines with less than 2 points to obtain the lines extracted in the final image. Figure 5 As shown, the horizontal part of the railing in (a) presents a bright line pattern, and the extracted bright lines are shown in (b). It can be clearly seen from the extraction results that the horizontal railing presenting a bright line pattern is accurately extracted, and the calculation speed is still very fast for the larger-sized image, which fully reflects the high efficiency and accuracy of the present invention in extracting lines on large-sized images.

[0039] The above embodiments are only used to illustrate the technical means adopted by the present invention and are not limited to the above technical means. The technical means derived, modified, or recombined from the above methods are still within the protection scope of the present invention.

Claims

1. A GPU-based image line extraction method, characterized in that: The following steps are involved: Step S1: Calculate the convolution of the first-order partial derivative and the second-order partial derivative of the two-dimensional Gaussian function with the image; Step S2: Calculate the normal direction of the line according to the Hessian matrix of the two-dimensional discrete image, perform Taylor expansion on the grayscale distribution function along the normal direction, and obtain the sub-pixel position of the center point of the line; Step S3: According to the distance between adjacent contour points and normal direction information, the contour points are connected according to the connection rules to extract the lines in the image.

2. The GPU-based image line extraction method according to claim 1, characterized in that: The step S1 specifically includes: Step S1.1: Calculate the convolution kernel size according to the input Gaussian smoothing parameter Sigma; Step S1.2: Calculate the first-order partial derivative of the two-dimensional Gaussian function and the second-order partial derivatives , generate the convolution kernel and convolve it with the two-dimensional image to generate the convolution result and .

3. The GPU-based image line extraction method according to claim 1, characterized in that: The step S2 specifically includes: Step S2.1: Calculate the maximum absolute eigenvalue and the corresponding eigenvector of the Hessian matrix of the two-dimensional discrete image, and preliminarily screen the center points of the lines according to whether the maximum absolute eigenvalue is higher than the low threshold and the brightness and darkness information of the extracted lines; Step S2.2: For the current pixel The grayscale value distribution function of the image along the normal direction is expanded by the second order Taylor, and the center point position t of the light strip in the section is calculated, which is the sub-pixel position of the extreme point. The point whose sub-pixel position is still within the current pixel range is taken as the center point of the final line to be connected, and the sub-pixel position coordinates, normal direction, and maximum absolute eigenvalue above the high threshold of the point are stored in a picture of the same size as the original image.

4. The GPU-based image line extraction method according to claim 3, characterized in that: The step S3 specifically includes: Step S3.1: traverse each pixel of the eigenvalue image in step S2.2, sort the non-zero eigenvalues ​​in descending order, and start from the first point above the high threshold that has not been connected to the line in this order as the starting connection point of this line; Step S3.2: Divide the normal angle into 8 directions, and search for the center point of the line adjacent to the current starting connection point according to the normal angle in the corresponding direction. At the same time, divide the center point of the line into two parts, the right side and the left side, with the current point as the center and the normal direction. The number of the center points of the line on each side is at most 3; Step S3.3: According to the distance and normal direction information between adjacent contour points, the extended lines are continuously connected according to the method of calculating the center point of the adjacent lines of each point in step S3.2, until there is no line center point at the end that can be connected or connected to the end point or middle point of other lines, thus completing the extraction of the current line; Step S3.4: According to the order of the feature values ​​in step S3.1, starting from the next point that has not been connected to the line as the starting connection point, execute steps S3.2 and S3.3 to complete the extraction of the next line; repeat steps S3.1, S3.2 and S3.3 until all the starting connection points have been traversed; Step S3.5: Remove the lines with less than 2 points to obtain the final screening result.

5. The GPU-based image line extraction method according to claim 3, characterized in that: The Hessian matrix can be expressed as follows: ， in, is the result of convolution of the second-order partial derivative of the two-dimensional Gaussian function with the image in step S1.

6. The GPU-based image line extraction method according to claim 3, characterized in that: Two eigenvalues ​​of the Hessian matrix It can be expressed as: ， The corresponding eigenvector It can be expressed as: 。 7. The GPU-based image line extraction method according to claim 3, characterized in that: Current pixel The second-order Taylor expansion of the image gray value distribution function along the normal direction can be expressed as follows: ， in Represents the result of convolution of the first-order partial derivative of the two-dimensional Gaussian function with the image. Represents the current pixel The gray value of is the result of convolution of the second-order partial derivative of the two-dimensional Gaussian function with the image in step S1, , are the x and y components of the eigenvector, and t is the center position of the light strip in the cross section.

8. The GPU-based image line extraction method according to claim 3, characterized in that: The position t of the center point of the light strip in the cross section is expressed as: ， in are the x and y components of the eigenvector, t is the center point of the light strip in the cross section, Represents the result of convolution of the first-order partial derivative of the two-dimensional Gaussian function with the image. is the result of convolution of the second-order partial derivative of the two-dimensional Gaussian function with the image in step S1.

Images