A High-Energy-Efficient FPGA-Implemented Point Cloud Feature Extraction Method and Its Application

Abstract

One technical solution of the present invention is to provide a method for extracting point cloud features implemented by a high-energy-efficiency FPGA, which is mapped to run on the FPGA. Another technical solution of the present invention is to provide an application of the aforesaid method for extracting point cloud features implemented by a high-energy-efficiency FPGA, which is characterized in that it is applied to the extraction of point cloud features for unmanned driving or robots. Compared with the prior art solutions, the innovation of the present invention lies in: a projection method with low complexity to organize disordered and sparse point clouds; a method with high parallelism to extract coarse-grained feature points; and a method with high parallelism to select fine-grained feature points.

Description

Technical Field

[0001] The present invention relates to a point cloud feature extraction algorithm and an application of the point cloud feature extraction algorithm. Background Art

[0002] The point cloud feature extraction algorithm extracts features (such as contours, corner points, and plane points) from each frame of point cloud data generated by a lidar for subsequent calculations, and is widely used in the fields of unmanned driving and intelligent robots. The point cloud feature extraction has two functions: one is to filter out the noise in the original point cloud and improve the subsequent positioning and mapping accuracy; the other is to remove redundant points and improve the running speed of subsequent operations.

[0003] However, the lidar beam has now been upgraded to 128 lines, and the scanning speed has been increased to 20 Hz, which will greatly increase the processing time of traditional feature extraction algorithms. On an edge computing processor, the current state-of-the-art point cloud feature extraction algorithm can only achieve a processing speed of less than 9 Hz, far lower than the real-time requirement. Coupled with the constraint of application power consumption for intelligent vehicles, there is an urgent need to implement a real-time and high-energy-efficiency feature extraction algorithm.

[0004] To solve this problem, relevant experts have made explorations in different aspects. The [1] series of works calculates geometric feature points by calculating the local curvature of each point. However, its calculation is complex and it cannot efficiently process a large amount of point clouds. The [2] series of works maps the three-dimensional point cloud to a two-dimensional range image or a bird's-eye view image, and then uses an image-like method to extract features. Although the speed has been greatly improved, they ignore that the lidar beam is unevenly emitted in the vertical direction, resulting in low feature accuracy. [3] proposed a method for quickly extracting two-dimensional feature points by developing the parallelism of the GPU, and the processing speed reached 300 frames per second. However, the GPU consumes a large amount of power, which will greatly affect the endurance of the intelligent vehicle. [4] implemented a high-performance and low-power feature extraction algorithm on an FPGA platform, but it can only process two-dimensional image data.

[0005] References:

[0006] 【1】J.Zhang and S.Singh, “Low-drift and real-time LiDAR odometry and mapping,” Auton.Robots, vol.41, no.2, pp.401–416, Feb.2017.

[0007] 【2】N. Attarmoghaddam and K. F. Li, “An area-efficient FPGA implementation of a real-time multi-class classifier for binary images,” IEEE Trans. Circuits Syst. II, Exp. Briefs, vol. 69, no. 4, pp. 2306–2310, Apr. 2022.

[0008] 【3】B. Nagy, P. Foehn, and D. Scaramuzza, “Faster than FAST: GPU-accelerated frontend for high-speed VIO,” in Proc. IEEE / RSJ Int. Conf. Intell. Robots Syst. (IROS), Oct. 2020, pp. 4361–4368.

[0009] 【4】Y. Liu et al., “MobileSP: An FPGA-based real-time keypoint extraction hardware accelerator for mobile VSLAM,” IEEE Trans. Circuits Syst. I, Reg. Papers, early access, Jul. 21, 2022, doi: 10.1109 / TCSI.2022.3190300. Summary of the Invention

[0010] The technical problem to be solved by the present invention is that on an edge computing processor, the currently most advanced point cloud feature extraction algorithm can only achieve a processing speed lower than 9 Hz, far lower than the real-time requirement.

[0011] To solve the above technical problem, a technical solution of the present invention is to provide a method for extracting point cloud features implemented by an energy-efficient FPGA, which is mapped to run on an FPGA. The method is characterized by including the following steps:

[0012] Step 1: Obtain unordered point cloud. For each point (x, y, z) in the unordered point cloud, use a laser angle-guided projection method to accurately project the point cloud into a point cloud matrix, including the following steps:

[0013] Step 101: Construct a lookup table for the median of laser emission angles for a lidar. Record the median angle φ of adjacent emission angles of the lidar in the lookup table for the median of laser emission angles.n ,φ n =(ω n +ω n+1 ) / 2,ω n is the emission angle of the nth laser beam emitted by the laser radar, n∈[0,N-1], and the laser radar emits a total of N laser beams;

[0014] Step 102: Multiply the square of the tangent value of each median angle recorded in the laser emission angle median lookup table by the positive and negative coefficients to construct a beam lookup table τ. n , the corresponding bundle search value τ n The calculation of is as follows:

[0015] τ n =sign(φ n )×(tan(φ n )) 2

[0016] In the formula, sign(φ n ) represents φ n The positive and negative signs of

[0017] Step 103: For each point (x, y, z) in the unordered point cloud, p =sign(z)×(z 2 / (x 2 +y 2 ))Get the corresponding search value τ p , and then use to find the value τ p By comparing with the beam lookup table, the laser beam to which each point belongs is obtained, and then the row index v in the point cloud matrix is ​​obtained. Combined with the row index h of the current point, the current point is projected into the point cloud matrix.

[0018] Step 104, repeat step 103 until all points in the unordered point cloud are traversed to obtain a point cloud matrix;

[0019] Step 2: Divide the elements of the point cloud matrix into three categories: starting elements, missing elements, and normal elements. Lost elements refer to the position where no point is mapped to the corresponding element, and those with some mapping are called normal elements. Starting elements refer to the elements belonging to the 0th row or the normal elements in the next row of the missing elements.

[0020] Column scanning traverses the point cloud matrix to detect coarse-grained feature points, including the following steps:

[0021] Step 201: Calculate the local plane curvature of each data point in each point cloud matrix, and filter out unreliable data points or blocking points;

[0022] For reliable data points, data points with local plane curvature c exceeding the corner threshold t edge are marked as coarse-grained corner points. For data points with local plane curvature c lower than the plane point threshold t plane , proceed to step 202 for processing;

[0023] Step 202: Calculate the slope θ of the data points obtained in the previous step. Mark data points with slope θ greater than the threshold t vp as coarse-grained vertical plane points. For data points with slope θ less than the threshold t θ , proceed to step 203 for processing;

[0024] Step 203: Introduce the global average ground point height h g and the local average ground point height h l , where the global average ground point height h g represents the average height of all ground points currently calculated, and the local average ground point height h l represents the average height of ground points within the local range of the current processing point;

[0025] For data points with slope θ less than the threshold t θ :

[0026] If the data point is the starting element, calculate the absolute value of the difference between its height value and the global average ground point height h g . Mark data points with the absolute value less than the global height difference threshold t hg as coarse-grained ground points;

[0027] If the data point is a normal element, calculate the absolute value of the difference between its height value and the global average ground point height h g , and calculate the difference between its height value and the local average ground point height h l . Mark data points with the absolute value of the difference between the height value and the global average ground point height h g less than the global height difference threshold t hg and the difference between the height value and the local average ground point height h l within the threshold range as coarse-grained ground points;

[0028] Step 204: Collectively refer to the obtained coarse-grained ground points and coarse-grained vertical plane points as coarse-grained plane points, and thus obtain coarse-grained feature points composed of coarse-grained plane points and coarse-grained corner points;

[0029] Step 3: Among the coarse-grained feature points, gradually and evenly select fine-grained feature points.

[0030] Preferably, in step 103, the column index h is calculated using the following formula:

[0031]

[0032] Wherein, Δα represents the rotational angle resolution of the lidar.

[0033] Preferably, in step 201, for the i-th data point p in the point cloud matrix i , its local plane curvature c is calculated by the following formula:

[0034]

[0035] Wherein, S represents the set of consecutive adjacent points adjacent to p in the same row in the point cloud matrix i .

[0036] Preferably, in step 201, by calculating the difference in depth distance between each data point and its adjacent points, unreliable data points or blocking points are filtered out.

[0037] Preferably, in step 202, the slope θ is the square of the difference in the z-direction distance between the current data point and the next laser beam divided by the square of the horizontal projection distance between the two points. Let the slope of the data point (x (i,j) , y (i,j) , z (i,j) ) at the (i, j) position in the point cloud matrix be θ (i,j) , then there is:

[0038] Preferably, in step 3, a plane point priority queue and a corner point priority queue are respectively established based on the coarse-grained plane points and the coarse-grained corner points. The elements in the plane point priority queue and the corner point priority queue constitute the fine-grained feature points, where:

[0039] The number of required fine-grained feature corner points is n e , then the corner point priority queue stores n e elements. The coarse-grained corner points are arranged in descending order of curvature in the corner point priority queue, and the elements with a difference in non-coexistent column indices in the corner point priority queue are less than a preset value;

[0040] The number of required fine-grained plane points is n p , then the plane point priority queue stores n p elements. The coarse-grained plane points are arranged in ascending order of curvature in the plane point priority queue, and the elements with a difference in non-coexistent column indices in the plane point priority queue are less than a preset value.

[0041] Another technical solution of the present invention is to provide an application of the above-mentioned point cloud feature extraction method implemented by an energy-efficient FPGA, which is characterized in that it is applied to the point cloud feature extraction of unmanned driving or robots.

[0042] Compared with the existing technical solutions, the innovation of the present invention lies in:

[0043] 1) A low-complexity projection method to organize unordered and sparse point clouds

[0044] The present invention proposes a laser angle-guided projection method to accurately project point clouds into a point cloud matrix, and proposes a look-up table method to reduce the algorithm complexity. At the same time, the present invention also implements a column scan scheduler and a progressive refresh strategy through dynamically adjustable inputs to ensure the high-performance operation of the FPGA hardware pipeline.

[0045] 2) A high-parallelism method to extract coarse-grained feature points

[0046] Based on the organized point cloud matrix, the present invention proposes a low-complexity method based on column scan and local search to stream-extract coarse-grained plane points and feature corner points. At the same time, the present invention optimizes the logic therein, eliminates data dependencies, and improves the implementation performance on the FPGA.

[0047] 3) A high-parallelism method to select fine-grained feature points

[0048] The present invention proposes a high-parallelism conditional priority queue to merge search sorting operations and conditional selection operations. The present invention uses the continuity and progressive nature of the point cloud matrix to uniformly select fine-grained feature points from the coarse-grained feature points, which will significantly reduce the running time and computing resources. Brief Description of the Drawings

[0049] Figure 1 Schematically shows the flowchart of the point cloud feature extraction method of the present invention, and is also the hardware architecture diagram;

[0050] Figure 2 Is a schematic diagram of ground detection. Detailed Embodiments

[0051] The following further elaborates the present invention in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and not to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms also fall within the scope defined by the appended claims of this application.

[0052] A point cloud feature extraction method implemented by an energy-efficient FPGA disclosed in this embodiment has been fully mapped to the FPGA for efficient operation, and the overall block diagram of the process is as Figure 1 shown, and the specific process is as follows:

[0053] Step 1. For a point (x, y, z) in the unordered point cloud, use the laser angle-guided projection method to accurately project the point cloud into the point cloud matrix, including the following steps:

[0054] Step 101. Construct a lookup table for the median laser emission angle for the lidar. Record the median angle φ of adjacent emission angles of the lidar in this lookup table for the median laser emission angle. n , φ n =(ω n +ω n+1 ) / 2, where ω n is the emission angle of the nth laser beam emitted by the lidar, n ∈ [0, N - 1], and the lidar emits a total of N laser beams.

[0055] Step 102. After multiplying the square of the tangent value of each median angle recorded in the lookup table for the median laser emission angle by a positive and negative coefficient, construct a beam lookup table τ. For each median angle φ n , the corresponding beam lookup value τ n is calculated as shown in the following formula:

[0056] τ n =sign(φ n )×(tan(φ n )) 2

[0057] In the formula, sign(φ n ) represents the positive and negative sign of φ n .

[0058] Step 103. For each point (x, y, z) in the unordered point cloud, obtain the corresponding lookup value τ p =sign(z)×(z 2 / (x 2 +y 2 )) and then compare the lookup value τ p with the beam lookup table to obtain the laser beam to which each point belongs, and thus obtain the row index v in the point cloud matrix. Compared with the method described in formula (2) of reference [2], this method can greatly reduce the computational complexity and improve the projection accuracy at the same time. p Step 104. Calculate the column index h based on the formula

[0059] where Δα represents the rotational angle resolution of the lidar. After obtaining the row index v and column index h corresponding to each point (x, y, z), project the point (x, y, z) to the position (v, h) in the point cloud matrix.

[0060] Step 105: Repeat Step 103 and Step 104 until all points in the unordered point cloud are traversed to obtain a point cloud matrix.

[0061] Step 101 and Step 102 are preprocessings for different lidars. To efficiently implement Step 103 to Step 105 on an FPGA, the present invention also designs a column scan scheduler with dynamically adjustable inputs. This column scan scheduler dynamically controls the input according to the difference between the column indices of the input point cloud matrix data and the output data, so as to efficiently implement this projection algorithm in a pipeline with less memory.

[0062] The column scan scheduler traverses the point cloud matrix column by column to detect coarse-grained feature plane points and feature corner points. The present invention divides the elements of the point cloud matrix into three categories, namely start elements, missing elements, and normal elements, where: missing elements refer to positions where no points are mapped, points with mappings are called normal elements, and start elements refer to elements belonging to the 0th row or normal elements in the row below the missing elements.

[0063] The coarse-grained detection process includes the following steps:

[0064] Step 201: Calculate the local plane curvature c of each data point in each point cloud matrix as shown in the following formula:

[0065]

[0066] In the formula, p i represents the i-th data point in the point cloud matrix, and S represents the set of consecutive adjacent points adjacent to p i in the same row in the point cloud matrix.

[0067] At the same time, by calculating the difference in depth distance between each data point and its adjacent points, unreliable data points or blocked points are quickly filtered out.

[0068] For reliable data points, data points with local plane curvature c exceeding the corner point threshold t edge are marked as coarse-grained corner points. For data points with local plane curvature c lower than the plane point threshold t plane , they enter Step 202 for processing.

[0069] Step 202: Calculate the slope θ of the data points obtained in the previous step. The slope θ is the square of the difference in the z-direction distance between the current data point and the next laser beam divided by the square of the horizontal projection distance between the two points. Let the slope of the data point (x (i,j) , y (i,j) , z (i,j) ) at the position (i, j) in the point cloud matrix be θ (i,j) , then there is:

[0070]

[0071] Mark the data points with a slope θ greater than the threshold t vp as coarse-grained vertical plane points. For data points with a slope θ less than the threshold t θ , proceed to step 203 for processing.

[0072] Step 203: Introduce the global average ground point height h g and the local average ground point height h l , where the global average ground point height h g represents the average height of all ground points currently calculated, and the local average ground point height h l represents the average height of ground points within the local range of the currently processed point. This local range is tentatively set as a 7×7 square matrix, as shown in the localregion point area in Figure 2 .

[0073] For data points with a slope θ less than the threshold t θ :

[0074] If the data point is the starting element, calculate the absolute value of the difference between its height value and the global average ground point height h g . Mark the data points with an absolute value less than the global height difference threshold t hg as coarse-grained ground points;

[0075] If the data point is a normal element, calculate the absolute value of the difference between its height value and the global average ground point height h g , and calculate the difference between its height value and the local average ground point height h l . Mark the data points with an absolute value of the difference between the height value and the global average ground point height h g less than the global height difference threshold t hg and the difference between the height value and the local average ground point height h l within the threshold range as coarse-grained ground points.

[0076] Step 204: Collectively refer to the obtained coarse-grained ground points and coarse-grained vertical plane points as coarse-grained plane points, and thus obtain coarse-grained feature points composed of coarse-grained plane points and coarse-grained corner points.

[0077] Among the coarse-grained feature points, gradually and evenly select fine-grained feature points. Different from the previous method of first sorting all points by curvature and then selecting, the present invention designs a conditional selection priority queue to quickly select fine-grained points. The length of the priority queue is determined according to the algorithm requirements. Mark the number of required fine-grained feature corner points and fine-grained plane points as n e and n pTaking fine-grained feature corners as an example, the present invention sets two conditions for the priority queue: one is that this priority queue is arranged in descending order of curvature, with the larger ones in the front and the smaller ones in the back; the other is that the difference in column indices of non-coexistent elements in this priority queue is less than 5. The first condition ensures the orderliness of the priority queue, and the second condition ensures that the feature points will not gather, and evenly distributed feature points can be selected. The specific steps are as follows:

[0078] Step 301: Compare each to-be-processed coarse-grained corner with all elements in the corner priority queue, respectively compare the curvature and column index, and obtain the corresponding flag bit array.

[0079] Step 302: According to the flag bit array, judge the operations of each element in the corner priority queue. There are three operations in total, namely unchanged, shifted, and inserted.

[0080] Step 303: Update the corner priority queue. The elements in the corner priority queue that need to be shifted are shifted one element to the right, and the to-be-processed element is inserted into the position with the insert flag in the corner priority queue.

[0081] Step 304: Traverse all coarse-grained corners. In this way, the elements retained in the corner priority queue are the selected fine-grained feature corners.

[0082] The selection method of fine-grained plane points is the same as that of corners, but the arrangement order of the plane point priority queue is reversed, becoming arranged in ascending order of curvature. Finally, the elements in the corner priority queue and the plane point priority queue constitute the fine-grained feature points.

[0083] The present invention can be applied to point cloud feature extraction in unmanned driving or robots, and the point cloud can be composed of data from lidar. For different types of lidar, the present invention can accurately and quickly complete the task of extracting the characteristics of the point cloud. FPGA acceleration makes the entire algorithm have better real-time performance and consume less energy.

Claims

1. A method for extracting point cloud features implemented with high energy efficiency and mapped to run on an FPGA, characterized in that It includes the following steps: Step 1: Obtain unordered point cloud. For each point (x, y, z) in the unordered point cloud, use the laser angle-guided projection method to accurately project the point cloud into the point cloud matrix, including the following steps: Step 101: Construct a median lookup table for the laser emission angle of the lidar, and record the median angle φ of adjacent emission angles of the lidar in the median lookup table for the laser emission angle n , φ n =(ω n +ω n+1 ) / 2, where ω n is the emission angle of the nth laser beam emitted by the lidar, n ∈ [0, N - 1], and the lidar emits a total of N laser beams; Step 102: Multiply the square of the tangent value of each median angle recorded in the laser emission angle median lookup table by the positive and negative coefficient to obtain each beam search value τ n , and construct a beam search table based on this, as shown in the following formula: τ n = sign(φ n ) × (tan(φ n )) 2 where sign(φ n ) represents the sign of φ n ; Step 103: For each point (x, y, z) in the unordered point cloud, obtain the corresponding lookup value τ through τ p = sign(z) × (z 2 / (x 2 + y 2 )) and obtain the corresponding lookup value τ p . Then, compare the lookup value τ p with the beam search table to obtain the laser beam to which each point belongs, thereby obtaining the row index v in the point cloud matrix. Combining with the row index h of the current point, project the current point into the point cloud matrix; Step 104: Repeat Step 103 until all points in the unordered point cloud are traversed to obtain the point cloud matrix; Step 2: Divide the elements of the point cloud matrix into three categories, namely starting elements, missing elements, and normal elements, where: Missing elements refer to those positions where no points are mapped. Those with points mapped are called normal elements. Starting elements refer to the elements belonging to the 0th row or the normal elements in the next row of the missing elements; Perform column scanning to traverse the point cloud matrix and detect coarse-grained feature points, including the following steps: Step 201: Calculate the local plane curvature of each data point in each point cloud matrix and filter out unreliable data points or blocking points; For reliable data points, data points where the local planar curvature c exceeds the corner threshold t edge are marked as coarse-grained corner points. For data points where the local planar curvature c is lower than the planar point threshold t plane , proceed to step 202 for processing; Step 202: Calculate the slope θ of the data points obtained in the previous step. Mark the data points with a slope θ greater than the threshold t vp as coarse-grained vertical plane points. For the data points with a slope θ less than the threshold t θ , proceed to Step 203 for processing; Step 203: Introduce the global average ground point height h g and the local average ground point height h l , where the global average ground point height h g represents the average height of all ground points currently calculated, and the local average ground point height h l represents the average height of ground points within the local range of the currently processed point; For data points where the slope θ is less than the threshold t θ : If the data point is the starting element, calculate the absolute value of the difference between its height value and the global average ground point height h g , and mark the data points with an absolute value less than the global height difference threshold t hg as coarse-grained ground points; If the data point is a normal element, calculate the absolute value of the difference between its height value and the global average ground point height h g and calculate the difference between its height value and the local average ground point height h l Mark the data points whose absolute value of the difference between the height value and the global average ground point height h g is less than the global height difference threshold t hg and the difference between the height value and the local average ground point height h l is within the threshold range as coarse-grained ground points; Step 204: Collect the obtained coarse-grained ground points and coarse-grained vertical plane points and call them coarse-grained plane points, then obtain the coarse-grained feature points composed of coarse-grained plane points and coarse-grained corner points; Step 3: Gradually and evenly select fine-grained feature points from the coarse-grained feature points.

2. The method for extracting point cloud features implemented by a high-energy efficiency FPGA according to claim 1, wherein In Step 103, the row index h is calculated using the following formula: In the formula, Δα represents the rotational angle resolution of the lidar.

3. The method for extracting point cloud features implemented by a high-energy-efficient FPGA according to claim 1, wherein, In step 201, for the i-th data point p in the point cloud matrix i , its local plane curvature c is calculated using the following formula: where S represents the set of consecutive adjacent points adjacent to p in the same row in the point cloud matrix i in the point cloud matrix 4. The method for extracting point cloud features implemented by a high-energy-efficient FPGA according to claim 1, wherein In Step 201, unreliable data points or blocking points are filtered out by calculating the difference in depth distance between each data point and its neighboring points.

5. The method for extracting point cloud features implemented by a high-energy efficiency FPGA according to claim 1, wherein In step 202, the slope θ is the square of the difference in the z-direction distance between the current data point and the next laser beam divided by the square of the horizontal projection distance between the two points. Let the slope of the data point (x (i,j) , y (i,j) , z (i,j) ) at the (i, j) position in the point cloud matrix be θ (i,j) . Then, we have:

6. The method for extracting point cloud features implemented by a high-energy efficiency FPGA according to claim 1, wherein In Step 3, a plane point priority queue and a corner point priority queue are respectively established based on the coarse-grained plane points and coarse-grained corner points. The elements in the plane point priority queue and the corner point priority queue constitute the fine-grained feature points, where: The number of fine-grained feature corner points required is n e , then the corner point priority queue stores n e elements. The coarse-grained corner points are arranged in the corner point priority queue from largest to smallest curvature, and elements with a difference in column indices that cannot coexist in the corner point priority queue are less than a preset value; The number of fine-grained planar points required is n p , then the planar point priority queue stores n p elements. The coarse-grained planar points are arranged in ascending order of curvature in the planar point priority queue, and elements with a difference in column indices less than a preset value cannot coexist in the planar point priority queue.

7. An application method of the point cloud feature extraction method implemented by a high-energy efficiency FPGA as described in claim 1, characterized in that, It is applied to point cloud feature extraction for unmanned driving or robots.

Images