Orb feature extraction accelerator based on stream processing and non-blocking implemented by FPGA

Abstract

Disclosed in the present invention is an ORB feature extraction accelerator based on stream processing and non-blocking implemented by an FPGA, which mainly comprises the following two innovations: a non-blocking hardware architecture based on stream processing and a buffer management algorithm are proposed, wherein in this technique, the algorithm is used to precisely control and buffer each column of an rBRIEF descriptor computation window, such that a new pixel stream input can be received while a descriptor is computed, thereby achieving non-blocking processing; and an efficient hardware sorting design embedded inside an accelerator is proposed, wherein in this technique, on the basis of a counting sort algorithm, minimal resources are used to implement rBRIEF sorting on hardware and the rBRIEF sorting is embedded inside the accelerator. The present invention achieves high-speed feature point extraction while ensuring the quality of feature points, without significantly reducing the accuracy of algorithms such as ORB_SLAM.

Description

FPGA-based stream processing and non-blocking ORB feature extraction accelerator Technical Field

[0001] The present invention relates to a low-power ORB feature extraction accelerator based on FPGA. Background Art

[0002] ORB feature extraction [1] is a commonly used algorithm in computer vision. Its main function is to extract the pixels (feature points) that remain unchanged in continuous images and calculate the descriptors of the feature points (used for feature point matching). Subsequently, various SLAM algorithms (such as ORB_SLAM) can be applied to estimate the camera's motion trajectory by extracting and comparing the features of the previous and next frames of the image. At the same time, a point cloud map composed of feature points can be obtained. This algorithm is widely used in the fields of visual positioning and 3D reconstruction.

[0003] As camera resolution and frame rates continue to increase, ORB feature extraction speed is facing significant challenges, especially when running on low-power platforms such as small robots or drones. Taking the ARM Cortex-A53 as an example, ORB_SLAM runs at only 6 frames per second, far below the camera's frame rate. ORB feature extraction accounts for over 60% of the processing time per frame. Accelerating ORB feature extraction can significantly improve the efficiency of algorithms like ORB_SLAM.

[0004] In order to solve this problem, relevant scholars have proposed a series of hardware designs to accelerate the operation of ORB feature extraction. One of the difficulties in ORB feature extraction hardware design is that the calculation of the rBRIEF descriptor [5] is complex and difficult to parallelize. eSLAM [2] proposed the RS-BRIEF descriptor, which can significantly reduce the computational complexity, but it leads to a significant reduction in accuracy. FSLAM [3] uses quantization and table lookup methods to speed up the calculation of feature point angles, but the overall calculation speed of rBRIEF is limited. On the other hand, since the calculation of the rBRIEF descriptor is difficult to apply pipeline processing, when calculating a descriptor, the input pixel stream will be blocked until the descriptor is calculated. The designs in [2][3][4] are all based on blocking calculations, and the delay caused by blocking accounts for 30%. In addition, the output descriptors need to be sorted according to their response values. Since the data bit width of the descriptors is large and the number is large, the usual hardware sorting will occupy a lot of on-chip memory and logic resources. eSLAM [3] uses hardware heap sorting, and ac2SLAM [4] adds a ping-pong buffer on the basis of heap sorting to reduce resource usage.

[0005] References

[0006] [1]E.Rublee,V.Rabaud,K.Konolige,and G.Bradski,“Orb:An efficient alternative to sift or surf,”in 2011 International Conference on Computer Vision,2011,pp.2564-2571.

[0007] [2]R.Liu,J.Yang,Y.Chen,and W.Zhao,“eslam:An energy-efficient accelerator for real-time orb-slam on fpga platform,”in Proceedings of the 56th Annual Design Automation Conference 2019,2019,pp.1-6.

[0008] [3]V.Vemulapati and D.Chen,“Fslam:an efficient and accurate slam accelerator on soc fpgas,”in 2022 International Conference on Field-Programmable Technology(ICFPT),2022,pp.1-9.

[0009] [4]C.Wang,Y.Liu,K.Zuo,J.Tong,Y.Ding,and P.Ren,“ac 2slam:Fpga accelerated high-accuracy slam with heapsort and parallel keypoint extractor,”in 2021 International Conference on Field-Programmable Technology(ICFPT),2021,pp.1-9.

[0010] [5] M.Calonder, V.Lepetit, C.Strecha, and P.Fua, "Brief: Binary robust independent elementary features," in Proc.11th European Conference on Computer Vision, 2010, pp.778-792.

[0011] Summary of the Invention

[0012] The purpose of the present invention is to accelerate the speed of ORB feature extraction to significantly improve the operating efficiency of algorithms such as ORB_SLAM.

[0013] In order to achieve the above object, the technical solution of the present invention is to provide an ORB feature extraction accelerator based on stream processing and non-blocking implemented by FPGA, including

[0014] The downsampling module is used to adjust the input image to the required size to obtain the original image;

[0015] Gaussian filter module, used to blur the original image to obtain a blurred image;

[0016] Corner detection module, used to determine the feature points in the blurred image and output the feature point mask;

[0017] The non-maximum suppression module is used to perform non-maximum suppression on the feature point mask, and is characterized by further comprising:

[0018] The rBRIEF calculation module uses a cache management algorithm to parallelize window column selection, rBRIEF calculation, and window column FIFO calculation. The steps include:

[0019] The working area of ​​the rBRIEF descriptor is a 37x37 sliding window, which is updated in each clock cycle. Whenever the sliding window is updated: the middle pixel left pixel of the leftmost column of pixels of the feature point mask and the middle pixel right pixel of the rightmost column of pixels are detected; if the middle pixel right pixel is a feature point, the middle column of the blurred image is regarded as the first column of a window, and the middle column of the blurred image is written into the window column FIFO, and the counter is cleared at the same time, and the next 36 middle columns are also written into the window column FIFO; if the middle pixel left pixel is a feature point, the middle column of the blurred image is regarded as the last column of a window, and a label is attached when each middle column is written into the window column FIFO to indicate whether it is the starting column or the ending column. The rBRIEF calculation module decides whether to perform rBRIEF calculation based on the label;

[0020] The rBRIEF calculation module reads data from the window column FIFO and splices it into another window matrix. A new column is inserted at the end of the window matrix in each clock cycle, and the other columns of the window matrix are shifted left in turn. The window splicing runs at a throughput of 1 column per cycle until a certain column becomes the end column; then the reading of the window column FIFO is stopped, and then the rBRIEF calculation is performed. When calculating rBRIEF, the original image is used to calculate the center of mass direction, and then the BRIEF coordinates are rotated according to the direction angle to calculate the rBRIEF descriptor;

[0021] The sorting module is a counting sort implemented by hardware.

[0022] Preferably, the sorting module internally uses 256 containers to cache the indexes of feature points, and the indexes corresponding to feature points with the same response value are cached in the same container.

[0023] Preferably, the sorting module is integrated with the rBRIEF calculation module. When the calculation of an rBRIEF descriptor is completed, its index is immediately stored in the container; when the calculation of all rBRIEF descriptors is completed, the indexes are output in sequence.

[0024] The above technical solution mainly includes the following two aspects of innovation:

[0025] 1) A non-blocking hardware architecture and buffer management algorithm based on stream processing is proposed. This technology uses an algorithm to precisely control and buffer each column of the rBRIEF descriptor calculation window, allowing the descriptor to receive new pixel stream inputs while it is being calculated, thus achieving non-blocking processing.

[0026] 2) We propose an efficient hardware sorting design embedded in the accelerator. This technology is based on a counting sort algorithm, uses minimal resources to implement rBRIEF sorting on hardware, and embeds it in the accelerator.

[0027] The technical solution disclosed in this invention is primarily used in visual SLAM on low-power platforms, achieving high-speed, low-power ORB feature extraction through a unique architecture. The non-blocking rBRIEF descriptor calculation in this invention significantly improves data throughput, while the integrated hardware sorting module further reduces overall latency and resource usage. While achieving high-speed feature point extraction, this invention also ensures feature point quality without significantly reducing the accuracy of algorithms such as ORB_SLAM. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 illustrates the system architecture of the accelerator;

[0029] Figure 2 illustrates the hardware architecture of rBRIEF descriptor computation;

[0030] Figure 3 illustrates the hardware architecture of the sorting module;

[0031] Figure 4 shows the comparison of experimental data;

[0032] Figure 5 illustrates the verification of non-blocking computation. DETAILED DESCRIPTION

[0033] Below in conjunction with specific embodiment, further set forth the present invention.Should be understood that these embodiments are only used to illustrate the present invention and are not used in limiting the scope of the present invention.In addition, should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall equally within the scope limited by the appended claims of the application.

[0034] The accelerator disclosed in the present invention has been fully implemented and runs on FPGA. The system framework of the entire accelerator is shown in Figure 1. The specific working process of the accelerator is as follows:

[0035] Image data is input from the ARM processor (PS in Figure 1) via AXIDMA in the form of a pixel stream to the FPGA (PL in Figure 1). The FPGA accelerator includes six submodules: downsampling, Gaussian filtering, corner detection (FAST), non-maximum suppression (NMS), descriptor calculation (rBRIEF), and sorting (Sort).

[0036] The downsampling module (Resizer) uses linear interpolation to resize the input image to the desired size.

[0037] The resized image is input to the Gaussian filter module, which blurs the image using a fixed Gaussian convolution kernel.

[0038] The blurred image and the original image are synchronously input to the corner detection module (FAST). This module uses the FAST algorithm to determine the location of the feature points in the blurred image. The feature point mask (KP mask), the blurred image, and the original image are synchronously output. The data value on the feature point mask is the Harris response value of the point.

[0039] The non-maximum suppression module (NMS) performs 3x3 non-maximum suppression on the feature point mask and simultaneously outputs the blurred image, the original image, and the feature point mask after non-maximum suppression.

[0040] The calculation of the rBRIEF descriptor (rBRIEF in Figure 1) is divided into three parts: window selection, rBRIEF calculation, and window column FIFO. This invention proposes a new cache management algorithm that allows these three parts to be calculated in parallel. The entire algorithm structure is shown in Figure 2. The following is a description of the algorithm flow:

[0041] 1) Window column selection

[0042] The rBRIEF descriptor computation operates on a 37x37 sliding window, which is updated every clock cycle. Whenever the sliding window is updated, we examine the middle two pixels between the leftmost and rightmost columns of the feature mask (the left pixel and right pixel in Figure 2) and decide whether to write the middle column of the blurred image into the FIFO. If the right pixel is a feature point, the middle column of the blurred image (the middle column) is considered the first column of a window (from left to right), indicated by the dashed box 3 in Figure 2. This middle column is written to the window column FIFO, the counter is reset, and the next 36 middle columns are also written to the window column FIFO. If the left pixel is a feature point, the middle column is considered the last column of a window (indicated by the dashed box 1 in Figure 2). Each middle column is appended with a tag when it is written to the window column FIFO, indicating whether it is the starting column (first column) or the ending column (last column). The rBRIEF computation module uses this tag to determine whether to perform rBRIEF computation.

[0043] b) rBRIEF calculation.

[0044] As shown in area 5 in Figure 2, the rBRIEF calculation module reads data from the window column FIFO and concatenates it into another window matrix. A new column is inserted at the end of the window matrix every clock cycle, and the remaining columns of the window matrix are shifted left in sequence. Window concatenation operates at a throughput of one column per cycle until a column reaches the end. It then stops reading from the window column FIFO and performs rBRIEF calculation, which takes approximately 300 cycles. To calculate rBRIEF, the original image is used to calculate the centroid orientation. The BRIEF coordinates are then rotated according to this orientation angle to calculate the rBRIEF descriptor.

[0045] c) Window column FIFO.

[0046] The existence of the window column buffer (area 4 in Figure 2) allows different production and consumption rates of the window column. By controlling the depth of the window column FIFO, it is possible to achieve that the rBRIEF calculation is carried out while the new pixel input is not interrupted.

[0047] The final step is sorting. This design uses hardware-implemented counting sorting, as shown in Figure 3. The hardware sorting uses 256 internal containers to cache feature point indices. The indices corresponding to feature points with the same response value are cached in the same container. The sorting module is actually integrated with the rBRIEF module. When a descriptor is calculated, its index is immediately stored in a container. When all descriptors are calculated, the indices are output sequentially. Since only the feature point indices are cached, additional resources are only required for one BRAM and a few hundred LUTs.

[0048] Figure 4 compares the proposed method with the original ORB_SLAM3 and existing accelerator designs (eSLAM, ac2SLAM, and FSLAM). Experimental data shows that the proposed method has the lowest latency and resource usage, the highest accuracy, and the lowest energy consumption. Figure 5 plots the accelerator latency as the density of feature points changes, demonstrating that the proposed method achieves non-blocking computation of the rBRIEF descriptor under actual operating conditions.

Claims

1. FPGA-based stream processing and non-blocking ORB feature extraction accelerator, including A downsampling module is used to adjust the input image to the required size to obtain the original image; Gaussian filter module, used to blur the original image to obtain a blurred image; Corner point detection module, used to determine the feature points in the blurred image and output feature point masks; The non-maximum suppression module is used to perform non-maximum suppression on the feature point mask, and is characterized by further comprising: The rBRIEF calculation module uses a cache management algorithm to enable window column selection, rBRIEF calculation, and window column FIFO to be calculated in parallel, including the following steps: The working area of ​​the rBRIEF descriptor is a 37x37 sliding window, which is updated in each clock cycle. Whenever the sliding window is updated: the middle pixel left pixel of the leftmost column of pixels of the feature point mask and the middle pixel right pixel of the rightmost column of pixels are detected; if the middle pixel right pixel is a feature point, the middlemost column of the blurred image is regarded as the first column of a window, and the middlemost column of the blurred image is written into the window column FIFO, and the counter is reset, and the next 36 middlemost columns are also written into the window column FIFO; if the middle pixel left pixel is a feature point, the middlemost column of the blurred image is regarded as the last column of a window, and when each middlemost column is written into the window column FIFO, a label is attached to indicate whether it is the starting column or the ending column. The rBRIEF calculation module decides whether to perform rBRIEF calculation based on the label; The rBRIEF calculation module reads data from the window column FIFO and splices it into another window matrix. A new column is inserted at the end of the window matrix in each clock cycle, and the other columns of the window matrix are shifted left in turn. The window splicing runs at a throughput of 1 column per cycle until a column is the end column; then the reading of the window column FIFO is stopped, and then the rBRIEF calculation is performed. When calculating rBRIEF, the original image is used to calculate the centroid direction, and then the BRIEF coordinates are rotated according to the direction angle to calculate the rBRIEF descriptor; The sorting module is a counting sort implemented by hardware.

2. The stream processing and non-blocking ORB feature extraction accelerator implemented by FPGA as claimed in claim 1, characterized in that: The sorting module internally uses 256 containers to cache the indexes of feature points, and the indexes corresponding to feature points with the same response value are cached in the same container.

3. The stream processing and non-blocking ORB feature extraction accelerator implemented by FPGA as claimed in claim 1, characterized in that: The sorting module is integrated with the rBRIEF calculation module. When the calculation of an rBRIEF descriptor is completed, its index is immediately stored in the container; when the calculation of all rBRIEF descriptors is completed, the indexes are output in sequence.

Images