A deep learning-based visual slam acceleration system and method

Abstract

The application discloses a visual SLAM acceleration system and method based on deep learning, wherein the acceleration strategy design for each key module in the deep learning visual SLAM, such as a feature extraction neural network, a feature extraction module, a feature matching module and a pose estimation module, and the visual SLAM acceleration prototype system based on an FPGA platform are constructed to realize the application. The application achieves the near real-time operation rate of 15 FPS while ensuring the calculation precision, and a better balance is achieved in the system operation rate and precision.

Description

Technical Field

[0001] This invention relates to a deep learning-based visual SLAM acceleration system and method, belonging to the field of visual SLAM accelerator design technology. Background Technology

[0002] With the continuous development of deep learning in the field of computer vision, methods to improve the performance and accuracy of visual SLAM using convolutional neural networks have gradually emerged. Detone et al. proposed the SuperPoint fully convolutional neural network, which directly detects all feature points of the entire image and the descriptor of each point in a single forward inference. SuperPoint trains an interest point extraction network for self-supervised training, thereby avoiding the limitations of manually designed feature points and the drawbacks of local block-based methods. Compared with traditional feature extraction algorithms, SuperPoint greatly simplifies the process complexity and improves the robustness of feature extraction. This method was published in the paper [DeTone D, Malisiewicz T, Rabinovich A. SuperPoint: Self-supervised interest point detection and description[C] / / Proceedings of the IEEE conference on computer vision and pattern recognition workshops.2018:224-236].

[0003] Convolutional neural networks typically require computation on three-dimensional or even four-dimensional data, which, from an algorithmic perspective, is equivalent to multiple loop processing. For example, the inference computation of convolution operations can be represented as a six-fold loop computation logic, including the height and width dimensions of features, the height and width dimensions of the convolution kernel, and the input and output channel dimensions of the data. The innermost layer of these loops is a standard multiply-accumulate (MAC) operation. By reasonably adjusting the nesting order between loops, the data dependency of each multiply-accumulate operation in the innermost loop can be eliminated. This method was published in the paper [Skrimponis P, Pissadakis E, Alachiotis N, et al. Accelerating Binarized Convolutional Neural Networks with Dynamic Partial Reconfiguration on Disaggregated FPGAs[M] / / ParallelComputing:Technology Trends.IOS Press,2020:691-700]. The so-called unfolding refers to copying multiple identical hardware computing units, placing the innermost loop operation on multiple hardware units for parallel execution, achieving the effect of increasing hardware resources to reduce execution latency by a factor of two. The outermost loop layer, which is not expanded, can be designed for pipelined execution to maximize hardware utilization. Generally, the amount of hardware resources consumed by hardware expansion has a positive linear relationship with the acceleration effect, which involves the trade-off between resource usage and acceleration performance. The method was published in [Zhang C, Prasanna V. Frequency domain acceleration of convolutional neural networks on CPU-FPGA shared memory system[C] / / Proceedings of the 2017 ACM / SIGDA International Symposium on Field-Programmable Gate Arrays.2017:35-44]. A fast exp function calculation method can reduce the clock cycle and required hardware logic resources for exp function calculation at the cost of a small amount of precision loss. The method was published in [Schraudolph N NA fast, compact approximation of the exponential function[J].Neural Computation,1999,11(4):853-862].

[0004] FPGA platforms, through dynamic hardware circuit updates, enable rapid deployment and verification of SLAM acceleration technologies, facilitating the implementation and iteration of research ideas. Therefore, many research results are based on FPGA platforms. In the field of visual SLAM, Fang et al. designed an ORB-SLAM acceleration module on an FPGA to address the computational bottleneck in visual SLAM algorithms. This module accelerated the feature extraction part of the feature point visual SLAM algorithm, achieving nearly double the speed while reducing power consumption by 83% compared to desktop CPUs. This method was published in the paper [Fang W, Zhang Y, Yu B, et al. FPGA-based ORB feature extraction for real-time visual SLAM[C] / / 2017 International Conference on Field Programmable Technology (ICFPT). IEEE, 2017:275-278].

[0005] The introduction of deep learning convolutional neural networks has improved the performance of visual SLAM algorithms, and the accelerated design of deep learning has always been an important research direction in the field of system architecture. Therefore, some research has focused on combining deep learning acceleration design with visual SLAM algorithms. Xu et al. proposed using the Xilinx DPU deep learning accelerator on an FPGA platform to accelerate the computation of the SuperPoint neural network based on CNN feature extraction, and designed Softmax and Normalization computation kernels for network post-processing. This system verified the feasibility of applying neural network acceleration to the front-end feature extraction part of visual SLAM, achieving an average processing speed of 15 fps. This method and the design of the Xilinx DPU were published in the papers [Xu Z, Yu J, Yu C, et al. CNN-based Feature-point Extraction for Real-time Visual SLAM on Embedded FPGA[C] / / 2020 IEEE 28th Annual International Symposium on Field-Programmable Custom Computing Machines (FCCM). IEEE, 2020: 33-37] and [[Xilinx, DPU for convolutional neural network[EB / OL]. https: / / www.xilinx.com / products / intellectual-property / dpu.html, 2021-10-07].

[0006] Currently, a large amount of research has focused on accelerating the computational tasks in visual SLAM algorithms, leveraging the parallel computing capabilities and low power consumption of FPGAs to achieve significant improvements in computational speed and energy efficiency compared to CPUs. Some studies have also introduced the use of deep learning processors in feature extraction tasks, improving the accuracy of visual SLAM algorithms while achieving real-time computation. However, most studies only focus on one or a few parts of the SLAM algorithm, without simultaneously considering both the acceleration performance of the entire system and the accuracy of the algorithm. Summary of the Invention

[0007] The technical problem solved by this invention is to overcome the shortcomings of the prior art and provide a deep learning-based visual SLAM acceleration system and method. It can be fully deployed on an FPGA computing platform and achieves a near real-time running speed of 15 FPS while ensuring computing accuracy, thus achieving a better balance between system running speed and accuracy.

[0008] One of the technical solutions of this invention:

[0009] A deep learning-based visual SLAM acceleration system includes: a deep learning accelerator, a feature extraction module accelerator, a feature matching module accelerator, a pose estimation module accelerator, external storage space, and an external controller.

[0010] Image data acquired by a depth camera or from a dataset is stored in external storage and then transmitted to a deep learning accelerator via scheduling by an external controller. The deep learning accelerator utilizes a multi-combination acceleration strategy for the feature extraction neural network (LEN) to improve inference speed, maintain inference accuracy, save platform hardware resources, and reduce actual system power consumption when the LEN extracts features from the input image. This multi-combination acceleration strategy includes: block operation strategy, quantization calculation strategy, operator design, storage space arrangement, memory access function design, and scheduling control scheme design. The image confidence tensor and descriptor tensor extracted by the LEN are transferred to external storage. The block operation strategy divides the data calculated by the LEN into blocks to reduce the hardware resources consumed by the LEN's computation.

[0011] The feature extraction module accelerator includes a post-processing unit computation acceleration strategy and Softmax and Normalization post-processing unit size settings. Image feature point tensors and descriptor tensors are passed from external storage space to the feature extraction module accelerator. The image feature point tensors are used to calculate feature point confidence using the Softmax post-processing unit acceleration strategy. The descriptor tensors are used to calculate descriptor vectors using the Normalization post-processing unit size settings. The feature point confidence vector and descriptor vector are then transferred to external storage space.

[0012] The feature point confidence vector and descriptor vector are passed from external storage space to the feature matching module accelerator. The feature matching module accelerator uses the feature matching acceleration strategy to accelerate the matrix multiplication calculation of the descriptor matrix and reduce the hardware resource requirements. The descriptor matrix is ​​composed of the feature point confidence vector and the descriptor vector. The resulting set of matching points is transferred to external storage space.

[0013] The matching point set and image depth data are fed into the pose estimation module accelerator through the control of the external controller. The pose estimation module accelerator uses a pose estimation-oriented acceleration strategy to accelerate the pose calculation process of the depth camera or platform and reduce the hardware resource requirements. The resulting pose data of the depth camera or platform is transferred to external storage space.

[0014] Image data and pose data in the external storage space are transmitted to the host computer through the control of the external controller to realize the visualization of SLAM information. The visualization of SLAM information includes the visualization of the running trajectory of the depth camera or platform, the visualization of the corresponding 3D sparse map, the statistics of running rate per frame, and the local saving function of pose trajectory text.

[0015] External controllers are used for communication, driver invocation, and data transfer operations.

[0016] Furthermore, in the deep learning accelerator, the block operation strategy performs block operations on the computation data of the feature extraction neural network to reduce the hardware resources occupied by the computation of the feature extraction neural network.

[0017] The block operation strategy includes the block-based loop method and block size selection, involving parameters including: (1) the high dimension H and wide dimension W of the block-based features in an operator layer of the feature extraction neural network; (2) the input channel dimension CIN and output channel dimension COUT of the block-based features; (3) the number of blocks in the high dimension TH and wide dimension TW of the features; (4) the number of blocks in the input channel dimension TCIN and output channel dimension TCOUT of the features; and (5) the high dimension KH and wide dimension KW of the weight convolution kernel after block-based operation.

[0018] The block-based looping method involves splitting the feature extraction neural network into four dimensions: width, height, input channel, and output channel, respectively, thus transforming the loop calculations in the convolution operation of the feature extraction neural network into:

[0019] TH×TW×TCIN×TCOUT×(H×W×CIN×COUT×KH×KW)

[0020] The block size selection is as follows: H=30, W=40, CIN=16, COUT=64;

[0021] The quantization calculation strategy is used to reduce the hardware resource usage and structural complexity of deep learning accelerators. Integer quantization data is used in all operations. The weights and intermediate layer results of the feature extraction neural network use an 8-bit width, and the sum of the convolution operators uses a 32-bit width. The quantization calculation strategy uses the following methods: (1) General conversion of floating-point numbers: Set the quantization scaling factor to establish the mapping relationship between the floating-point number to be converted and the converted integer number; (2) Integer quantization in convolution operations: Take the integer number after the floating-point number is converted into the convolution calculation, and then convert the newly appearing floating-point number in the convolution calculation into an integer and perform an arithmetic right shift operation to realize the overall quantization of the convolution operation.

[0022] The operator design is used to accelerate the operation units in the neural network, including the core acceleration strategy for convolution calculation, the core acceleration strategy for BAQ calculation, and the core acceleration strategy for max pooling calculation.

[0023] The core acceleration strategy for convolution calculation is used to perform accelerated convolution calculation of block data. The core acceleration strategy for convolution calculation is as follows: on the CIN and COUT loops in the convolution operation of weight matrix and feature vector, hardware unrolling operation is performed to convert the convolution operation into parallel dot product operation in hardware. This is used to improve the parallelism of convolution operation in convolution layer, and at the same time, the convolution operation process is divided into pipeline segments according to the clock cycle frequency to further reduce the calculation cycle of parallel operation.

[0024] The BAQ computation core acceleration strategy is used to accelerate the calculation of bias value, activation function, and quantization scaling in the convolutional layer. The BAQ computation core acceleration strategy is as follows: (1) Perform hardware unrolling operation on the COUT loop in the bias value, activation function, and quantization scaling operations to improve computational parallelism; (2) Set the activation function operation after the last convolution layer of the feature extraction neural network structure to be bypassable; (3) Perform pipeline segmentation of the BAQ computation process according to the clock cycle frequency to reduce the computation cycle of the parallel operation process.

[0025] The core strategy for max pooling computation is used to accelerate max pooling computation in convolutional layers. This strategy does not perform pooling operation directly on a block. Instead, it waits for four adjacent convolutional blocks in the [H,W] dimension to be scheduled and completed before performing pooling operation on these four blocks simultaneously. During the pooling operation, hardware unrolling is performed in the COUT dimension of the block data, and pipelined operation is performed by splitting according to the clock cycle frequency.

[0026] The three core acceleration strategies for convolution, BAQ, and max pooling form a continuous acceleration flow, and the entire internal operation of the convolutional layer, "convolution - activation function - max pooling", does not involve additional memory access overhead for reading and writing external storage.

[0027] The storage space is divided into input space, output space, and temporary storage space;

[0028] The input space includes Input, Weight, Bias, and M spaces, which are used to store input features, weight parameters, bias parameters, and quantization scaling parameters, respectively.

[0029] The Input space has a unit width of 8 bits, a unit size of (H+2)×(W+2)×CIN, a number of units of 4, and a split dimension of CIN.

[0030] The weight space has a cell width of 8 bits, a cell size of 3×3×CIN×COUT, a cell count of 4, and a split dimension of CIN and COUT.

[0031] The Bias space has a unit width of 32 bits, a unit size of [value], a unit number of COUT, a unit number of 2, and a split dimension of COUT;

[0032] The M-space has a unit width of 32 bits, a unit size of , a number of units COUT, a number of units of 2, and a split dimension of COUT;

[0033] The output space includes Output and Pooling spaces, which are used to store the calculation results of the convolutional and pooling layers to be output, respectively.

[0034] The Output space has a unit width of 8 bits, a unit size of H×W×COUT, a number of units of 4, and a split dimension of COUT.

[0035] The Pooling space has a cell width of 8 bits, a cell size of H×W×COUT, a cell count of 1, and a split dimension of COUT.

[0036] The temporary storage space Accum is only used to temporarily store the accumulated results of the convolution calculation core;

[0037] The Accum space has a cell width of 32 bits, a cell size of H×W×COUT, a cell count of 4, and a split dimension of COUT.

[0038] The memory access unit design includes internal transmission design and external transmission design;

[0039] The memory access unit is internally designed to divide the memory access function into the following modules: Load_Input, Load_Weight, Load_Bias, Load_M, and Store modules;

[0040] The Load_Input and Load_Weight modules are used to read 8 bits of data into the Input and Weight spaces of the storage space;

[0041] The Load_Bias and Load_M modules are used to read the 32-bit Bias and M spaces;

[0042] The Store module is used to write 8-bit Output and Pooling data back to external storage;

[0043] The external design of the memory access unit includes a maximum of four 128-bit AXI bus lines between the external storage and the memory access unit, and a low-bit-width data splicing transmission method. The memory access unit reads 128 bits at a time from the external storage via the AXI bus, which is 16 consecutive 8-bit data. The memory access module corresponding to each storage space uses hardware expansion to split the 128-bit data into 8 bits and put them into 16 RAMs respectively. The feature data dimension arrangement order of the memory access unit in the external storage and the storage unit is as follows: first arrange the data according to the channel dimension, and then arrange it according to the width dimension and the height dimension in sequence.

[0044] The control and scheduling scheme includes inter-layer scheduling and intra-layer scheduling;

[0045] The inter-layer scheduling is used to control the execution state of the feature extraction neural network computation and to store the parameter data of each layer. The inter-layer scheduling adopts a dual-buffer design, with the specific read-write alternation steps as follows: when buffer A reads data, data in buffer B is processed; when buffer B reads data, buffer A is processed.

[0046] The in-layer scheduling is used to control the computation module in the feature extraction neural network. The operation sequence is as follows: (1) Select feature and weight blocks; (2) Schedule and complete the computation process of "convolution calculation - BAQ calculation - max pooling calculation" of the convolutional layer.

[0047] Furthermore, in the feature extraction module accelerator:

[0048] The post-processing unit acceleration strategy adopts a fast exp function calculation method to accelerate the exp function calculation process involved in the feature point tensor Softmax calculation. The method is as follows: 1) The input float32 type data is represented as an integer, and the data is stored in little-endian byte order in the hardware; 2) The converted integer is directly represented as a floating-point number as the result of the exp function.

[0049] The post-processing unit is configured with 8 Softmax and 8 Normalization operation units, and the AXI transmission bus consists of two 128-bit wide lines.

[0050] Furthermore, the feature matching acceleration strategy in the feature matching module accelerator includes matrix multiplication calculation unit design and comparison unit design.

[0051] The descriptor matrix consists of descriptor vectors corresponding to the feature point confidence vectors;

[0052] The matrix multiplication calculation unit is designed to adjust the loop order of vector dimension D, previous image descriptor matrix dimension M, and current image descriptor matrix dimension N in the feature matching operation, so that dimension D is placed at the outermost layer of the triple loop, and hardware expansion operation is performed on dimensions M and N to realize parallel dot product operation and improve the parallelism of the overall loop calculation.

[0053] The matrix multiplication calculation unit is designed to have a scale of 16×16, consisting of 256 multiply-accumulator units.

[0054] The comparison unit is designed to store the local optimal results of two adjacent images in both directions, i.e., the maximum value of the local dot product calculated by the matrix multiplication unit, in a single on-chip storage unit; the size of this storage unit is capable of storing temporary nearest neighbor results that are twice the number of the maximum feature points.

[0055] The feature matching module accelerator's internal computation logic adopts the integer quantization design in deep learning accelerators, quantizing all input descriptor vectors into int8 type data.

[0056] Furthermore, in the pose estimation module accelerator, the acceleration strategy for pose estimation adopts a parameter upper limit setting method;

[0057] The parameter upper limit settings include: upper limit setting for matching feature point pairs and upper limit setting for the number of iterations during pose solving;

[0058] The upper limit for matching feature point pairs is set to 200 pairs for each calculation, selected in ascending order of descriptor distance;

[0059] The upper limit of the number of iterations in the pose solving process is set to 10 iterations. When the incremental magnitude in the iterative solving process is less than 10e-6, the loop will automatically exit and the solution will end.

[0060] Furthermore, the deep learning accelerator, feature extraction module accelerator, feature matching module accelerator, and pose estimation module accelerator are deployed on the programmable logic (PL) of the FPGA embedded platform. The ARM in the FPGA embedded platform acts as an external controller to control the deep learning-based visual SLAM acceleration system, handling operations such as communication between the deep learning-based visual SLAM acceleration system and the host computer, driver calls, and data transfer. This external controller is connected to each accelerator in the programmable logic via a 32-bit AXI-Lite bus. The external storage is the 4GDDR4 on-chip memory space on the FPGA embedded platform. Under the DMA control of the ARM in the FPGA embedded platform, the on-chip storage is connected to the data interfaces of each accelerator through six 128-bit AXI buses.

[0061] The second technical solution of this invention: a deep learning-based visual SLAM acceleration method, comprising the following steps:

[0062] Step 1: The image data acquired by the depth camera or in the dataset is scheduled and fed into the deep learning accelerator. In the deep learning accelerator, a multi-combination acceleration strategy is adopted to accelerate the inference process of the feature extraction neural network. The multi-combination acceleration strategy includes block operation strategy, quantization calculation strategy, operator design, storage space arrangement, memory access function design and scheduling control scheme design, to obtain the image confidence tensor and descriptor tensor.

[0063] Step 2: Based on the image feature point tensor and descriptor tensor obtained in Step 1, the input is fed into the feature extraction module accelerator. In the feature extraction module accelerator, the post-processing unit acceleration strategy and post-processing unit size setting are adopted to accelerate the calculation of the Softmax and Normalization post-processing units of the image feature point tensor and descriptor tensor, and the image confidence vector and descriptor vector are obtained.

[0064] Step 3: In the feature matching module accelerator, the matrix multiplication calculation unit design and comparison unit design are used to accelerate the operation process of the image confidence vector and descriptor vector matrix obtained in Step 2, so as to obtain the matching point set;

[0065] Step 4: Based on the matching point set obtained in Step 3, an acceleration strategy oriented towards pose estimation is adopted in the pose estimation module accelerator to accelerate the pose calculation of the depth camera or platform and obtain the pose data of the depth camera or platform.

[0066] The advantages of this invention compared to the prior art are:

[0067] (1) Unlike existing technologies, the deep learning-based visual SLAM acceleration system and method proposed in this invention first proposes a deep learning accelerator for feature extraction neural networks. This accelerator achieves accelerated inference, preserved computational accuracy, reduced platform hardware resources, and lower system operating power through the use of multiple combined acceleration strategies. Based on this, feature extraction module accelerator, feature matching module accelerator, and pose estimation module accelerator are proposed respectively. Based on the image feature point tensor and descriptor tensor output by the deep learning accelerator, the feature extraction module accelerator uses a post-processing unit acceleration strategy and post-processing unit size setting to accelerate the computation of the Softmax and Normalization post-processing units for the image feature point tensor and descriptor tensor. Based on the image confidence vector and descriptor vector output by the feature extraction module accelerator, the feature matching module accelerator accelerates the matrix operation process of the image confidence vector and descriptor vector through matrix multiplication calculation unit design and comparison unit design. Based on the matching point set output by the feature matching module accelerator, the pose estimation module accelerator accelerates the pose calculation of the depth camera or platform through a pose estimation-oriented acceleration strategy. This achieves overall acceleration of deep learning-based visual SLAM algorithms on the FPGA platform, balancing the acceleration performance of the entire system with the accuracy of the algorithm.

[0068] (2) The deep learning-based visual SLAM acceleration system and method proposed in this invention breaks through the bottleneck of computation and hardware resource consumption of deep learning-based visual SLAM algorithm on resource-limited embedded platforms, and solves the problem of its difficult deployment on resource-limited embedded platforms.

[0069] (3) The deep learning-based visual SLAM acceleration system fully implemented on the FPGA platform can reduce the computational load, storage and power consumption of the system while taking into account the accuracy of the algorithm. It provides key technologies and optimization ideas for the development and hardware implementation of small unmanned systems, robots and intelligent embedded systems.

[0070] (4) The deep learning-based visual SLAM acceleration system fully implemented on the FPGA platform in this invention can achieve an energy efficiency ratio of less than 2.5 fps / W while maintaining a system operating speed of 15 FPS and pose estimation accuracy. Compared with the Xilinx DPU, a related accelerator in the world, this invention achieves a significant reduction in RPE error while maintaining the same computing speed, and has better usability.

[0071] (5) The deep learning accelerator implemented on the FPGA platform in this invention can run independently and is used to accelerate the inference of feature extraction neural networks, maintain computational accuracy, save platform hardware resources, and reduce the actual operating power of the system. It provides key technologies and optimization ideas to support the development and hardware implementation of deep learning-based computer vision. Attached Figure Description

[0072] Figure 1 This invention comprises a deep learning-based visual SLAM acceleration system.

[0073] Figure 2 This describes the execution order of each operation in the main workflow of this invention;

[0074] Figure 3 This is the overall architecture of the deep learning accelerator of the present invention;

[0075] Figure 4 The continuous accelerated computational flow designed for the operator of this invention;

[0076] Figure 5 This is a schematic diagram of the core acceleration strategy for convolution calculation in this invention;

[0077] Figure 6 This is a schematic diagram of the BAQ computation core acceleration strategy of the present invention;

[0078] Figure 7 This is a schematic diagram of the grid-shaped block data arrangement method of the present invention;

[0079] Figure 8 This is a schematic diagram of the in-layer scheduling pipeline design of the present invention;

[0080] Figure 9 Schematic diagram of matrix multiplication calculation and comparison unit;

[0081] Figure 10 This describes the workflow and conversion logic of the external controller program in this invention. Detailed Implementation

[0082] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0083] The design ideas and methods for a deep learning-based visual SLAM acceleration system are as follows:

[0084] like Figure 1As shown, the deep learning-based visual SLAM acceleration system of this invention comprises: a deep learning accelerator, a feature extraction module accelerator, a feature matching module accelerator, a pose estimation module accelerator, external storage space, and an external controller. The deep learning accelerator and the feature extraction module accelerator are used to perform computation and acceleration tasks for feature extraction; the feature matching module accelerator is used to perform computation and acceleration tasks for feature matching; the pose estimation module accelerator is used to perform computation and acceleration tasks for pose estimation; the external controller is used to control the deep learning-based visual SLAM acceleration system, mainly including handling communication with the host computer, driver calls, and data transfer operations; the external storage space is used to store the input and output data of each component in the deep learning-based visual SLAM acceleration system.

[0085] In the main workflow of a deep learning-based visual SLAM acceleration system, the specific execution order of feature extraction, feature matching, and pose estimation operations is as follows: Figure 2 As shown, the image data acquired by the depth camera first undergoes feature extraction, then feature matching, and finally pose estimation. The data is then visualized on the host computer based on the deep learning visual SLAM acceleration system, showing the output depth camera trajectory, 3D sparse map, and running rate statistics.

[0086] Figure 1 The arrows in the diagram indicate the direction of data transmission. The deep learning-based visual SLAM acceleration method and operation process are summarized as follows:

[0087] Step 1: Image data acquired by the depth camera or from the dataset is stored in external storage and then transmitted to the deep learning accelerator via scheduling by an external controller. The deep learning accelerator employs multiple combination acceleration strategies to accelerate the inference process of the feature extraction neural network, and the resulting image confidence tensor and descriptor tensor are transferred to external storage.

[0088] Step 2: The image feature point tensor and descriptor tensor obtained in Step 1 are passed to the feature extraction module accelerator through scheduling by the external controller. The feature extraction module accelerator adopts a post-processing unit acceleration strategy and post-processing unit size setting to accelerate the calculation of the Softmax and Normalization post-processing units for the image confidence tensor and descriptor tensor. The resulting image confidence vector and descriptor vector are transferred to external storage space.

[0089] Step 3: The image confidence vector and descriptor vector obtained in Step 2 are transmitted to the feature matching module accelerator through the scheduling of the external controller. The matrix operation process of the image confidence vector and descriptor vector matrix is ​​accelerated through the matrix multiplication calculation unit design and comparison unit design. The resulting matching point set is transferred to the external storage space.

[0090] Step 4: The matching point set obtained in Step 3 and the image depth data acquired by the depth camera are transmitted to the pose estimation module accelerator through the scheduling of the external controller. The pose calculation of the depth camera or platform is accelerated through the acceleration strategy oriented towards pose estimation. The resulting pose data of the depth camera or platform is transferred to the external storage space.

[0091] Step 5: Image data and pose data in the external storage space are transmitted to the host computer through the control of the external controller, which is used to generate a visualization of the depth camera's running trajectory, a 3D sparse map, and running rate statistics.

[0092] The design concept and acceleration method of this invention for deep learning accelerators are as follows:

[0093] As shown in the figure, the overall architecture of a deep learning accelerator is divided into four parts: control unit, computation unit, storage unit, and memory access unit. The control unit receives control commands from the external CPU and schedules the various units within the deep learning accelerator according to the established process. The computation unit handles most of the computational functions in the feature extraction neural network and is the most energy-intensive part of the deep learning accelerator. The storage unit is on-chip storage space, consisting of SRAM or BRAM on the FPGA. The memory access unit is mainly used for data read and write operations within the deep learning accelerator.

[0094] The feature extraction neural network (LEN) consists of 12 convolutional layers, followed by ReLU activation and three max-pooling layers. To maximize the inference speed of the LNN while minimizing accuracy loss and hardware resource consumption, the main design principle for deep learning accelerators is defined as "hardware scheduling-modular operation." "Hardware scheduling" means fixing the inference process of the LNN within the hardware control unit, thus avoiding the instruction parsing and control latency caused by software control. "Modular operation" means implementing different computational and memory access operations using individual modules and combining them for execution according to a scheduling scheme, thereby improving the reusability of the hardware design and avoiding excessive hardware resource consumption caused by complex logic.

[0095] Based on the overall architecture and design concept described above, multiple acceleration strategies are designed for deep learning accelerators. These strategies include: block-based computation strategy, quantization computation strategy, operator design, storage space arrangement, memory access function design, and scheduling control scheme design. Their specific implementation methods are as follows.

[0096] Quantization calculation strategy

[0097] To reduce the hardware resource consumption and structural complexity of deep learning accelerators, integer quantization is used throughout the entire computation process. Specifically, the inputs and weights of the feature extraction neural network use an 8-bit width, while the summation of convolutional layers uses a 32-bit width to prevent overflow. The formula for converting floating-point numbers to integers in the quantization calculation strategy is shown below, where q is an integer, r is a floating-point number, and S is the quantization scaling factor:

[0098]

[0099] Let y be the output of the convolution operation, w be the weights of the convolution operation, x be the input of the convolution operation, b be the bias of the convolution operation, and S be the weights of the convolution operation. y S w S b The quantization scaling factors for y, w, and x are respectively, and q y q w q b These are the integers converted from y, w, and x, respectively. The formula for quantizing the convolution operation in the quantization strategy is shown below, that is, by quantizing [M... i (Q+q b Perform a right shift operation to obtain the integer q after the convolution operation output. y M i floating-point number and The converted integer, Q = q w q b If n is greater than 16, it is the number of right shift steps:

[0100] q y =[M i (Q+q b )]>>n

[0101] By implementing a quantization calculation strategy, the hardware multiply-accumulator (DSP) resource consumption required during feature extraction neural network inference can be effectively reduced, thereby improving the computing power of deep learning accelerators and reducing the latency of feature extraction neural network inference execution under limited hardware platform resources.

[0102] Block operation strategy

[0103] Since the on-chip BRAM size of the ZCU102 FPGA embedded platform is only 32.1Mb, a block-based computation strategy is needed to divide the computational data in the feature extraction neural network according to size in order to perform on-chip computation. The block size is H=30, W=40, CIN=16, COUT=64. The operation method is to perform block processing on the width, height, input channel, and output channel of the feature, so that the width of the feature of a convolutional layer operation becomes W×TW, and the other dimensions are processed similarly. Taking the block-based computation of the convolution operation in the feature extraction neural network as an example, after block processing, the original six-layer loop adds four block loop dimensions, becoming a total of ten-layer loop operation, and the calculation order becomes [(TH×H)×(TW×W)×(TCIN×CIN)×(TCOU×COUT)×KH×KW]. To improve the speed of block-based computation in convolution operations, the block-based operation strategy adjusts the computation order to [TH×TW×TCIN×TCOUT×(H×W×CIN×COUT×KH×KW)], where the value within parentheses represents the computational cost of a single block, and the value outside represents the total number of blocks. The purpose of this is to move data memory access operations outside the block-based computation of convolution operations while maintaining the total computational cost. This reduces data read / write operations during loop operations and avoids excessive data memory access latency. The parameters and symbols in the above block operation strategy are as follows. In the following description, the data used in a single calculation are all block-based data: 1) The height (H) and width (W) dimensions of the block-based features in an operator layer of the neural network; 2) The number of input (CIN) and output (COUT) channels of the block-based features; 3) The number of blocks in the height (TH) and width (TW) dimensions of the features; 4) The number of blocks in the input (TCIN) and output (TCOUT) channels of the features; 5) The height (KH) and width (KW) dimensions of the block-based weight convolution kernel.

[0104] Operator design

[0105] In the convolutional layers of the feature extraction neural network, convolution, bias, activation function, quantization scaling, and max pooling calculations are accelerated by executing core acceleration strategies for convolution, BAQ, and max pooling, respectively. The core acceleration strategy for convolution is used for convolution calculation and acceleration; the core acceleration strategy for BAQ is used for bias, activation function, and quantization scaling calculations and acceleration; and the core acceleration strategy for max pooling is used for max pooling calculation and acceleration. Figure 4As shown, the three core acceleration strategies for convolution, BAQ, and max pooling in the operator design form a continuous accelerated computation flow, so that the entire internal operation of the convolutional layer, "convolution calculation - BAQ calculation - max pooling calculation", does not involve additional memory access overhead of reading and writing external storage.

[0106] The core acceleration strategy for convolution computation employs an optimized design based on matrix-vector accumulation units. According to the computational principle of two-dimensional convolution operators, the input feature vector needs to be multiplied by a vector of length CIN in the COUT dimension of the weight matrix. Therefore, to maximize the parallel processing of the dot product calculation, the core acceleration strategy for convolution computation is as follows: Figure 5 As shown in the diagram. The hardware unpacking operation along the COUT dimension involves broadcasting the input feature vector and copying it to COUT copies. The copied feature vectors are then mapped to each vector in the weight matrix, and dot product operations are performed simultaneously in multiple dot product units. The hardware unpacking operation along the CIN dimension involves performing the unpacking operation along the CIN dimension of the vector computation in each dot product unit, and then accumulating the parallel computation results of the vectors using an addition tree structure. To further improve the hardware computational acceleration, the parallel processing of the dot product computation in the core convolution computation acceleration strategy is pipelined at a frequency of 100MHz.

[0107] The BAQ computational core acceleration strategy is a multi-task acceleration strategy that includes executing bias value, activation function, and quantization scaling computation tasks in convolutional layers. The computational process within the BAQ computational core acceleration strategy is as follows: Figure 6 As shown, it performs hardware unrolling on the COUT cyclic dimension of its input tensor to achieve parallel BAQ operations and performs pipeline segmentation adapted to the core acceleration strategy of convolution computation. The BAQ operation process is as follows: 1) Calculate the bias value of the convolution computation result; 2) Calculate the activation function of the result after bias value calculation, where the activation function is the ReLU function; 3) When the activation function calculation result is greater than 0, perform quantization scaling operation on the calculation result; otherwise, the activation function calculation result is equal to 0. In addition, since the last convolution layer of the feature extraction neural network structure does not perform activation function operation, the activation function part in the computation flow of the core acceleration strategy of BAQ computation is designed to be bypassable: if the control unit does not enable the activation function function according to the definition of the feature extraction neural network, the BAQ operation will not perform the activation function operation.

[0108] The maximum pooling core acceleration strategy is used to calculate the three-layer pooling operator with a stride of 2 and a window size of 2 in the feature extraction neural network, and accelerate this calculation process. Since there is no arithmetic operation in the maximum pooling calculation, only the size comparison of integer values, hardware unfolding is performed on the COUT dimension of its input tensor to realize the parallel operation of the comparison tree. The implementation of the comparison tree requires the introduction of an additional small part of storage space for storing comparison results, which is the only storage space located inside the arithmetic unit. All input and output data required for other core calculations are stored in the storage unit according to the storage space arrangement. In addition, in order to reduce the storage fragmentation generated during the pooling process, the maximum pooling core acceleration strategy stipulates that four BAQ calculation results adjacent in the [H,W] dimension and arranged in a "field" shape shall be acquired at the same time before performing the maximum pooling calculation, wherein the arrangement in the "field" shape is as Figure 7 shown.

[0109] Storage Space Arrangement

[0110] The storage space is generally divided into three categories: input space, output space, and temporary storage space. Input, Weight, Bias, and M belong to the input space and are used to store input features, weight parameters, bias parameters, and quantization scaling parameters. Output and Pooling belong to the output space and are used to store the computation results of the convolutional and pooling layers to be output. Accum is a special on-chip temporary storage space used only to temporarily store the accumulated results of the core convolutional computation. The Input space has an 8-bit width, a unit size of (H+2)×(W+2)×CIN, 4 units, and a split dimension of CIN; the Weight space has an 8-bit width, a unit size of 3×3×CIN×COUT, 4 units, and a split dimension of CIN and COUT; the Bias space has a 32-bit width, a unit size of COUT, 2 units, and a split dimension of COUT; the M space has a 32-bit width, a unit size of COUT, 2 units, and a split dimension of COUT; the Output space has an 8-bit width, a unit size of H×W×COUT, 4 units, and a split dimension of COUT; the Pooling space has an 8-bit width, a unit size of H×W×COUT, 1 unit, and a split dimension of COUT; the Accum space has a 32-bit width, a unit size of H×W×COUT, 4 units, and a split dimension of COUT. Furthermore, because a storage space partitioning strategy is used to enable the simultaneous reading of large amounts of data, the number of storage spaces varies. For example, the Input space is partitioned according to the CIN dimension, so that data belonging to different CIN dimensions is stored in different RAMs. In this way, when the hardware processing unit retrieves data, it actually retrieves data from different RAM storage devices, thus meeting the requirement of simultaneous reading. Other storage spaces also have similar partitioning designs.

[0111] Memory access function design

[0112] The memory access function is used to read a block of data from external storage into on-chip storage, or to write data from on-chip storage back to a specific location in external storage. The memory access function is designed to accelerate the data access process.

[0113] Based on the data read / write direction and data bit width, the memory access function is divided into the following modules: Load_Input, Load_Weight, Load_Bias, Load_M, and Store. Load_Input and Load_Weight read 8-bit wide data into the Input and Weight spaces of the memory, respectively. Load_Bias and Load_M are responsible for reading the 32-bit wide Bias and M parameters. Store writes the 8-bit wide Output and Pooling data back to external storage. To quickly locate the data position within the entire feature space, the index of the memory access unit is divided into intra-block index (intra_index) and inter-block index (inter_index). Taking the Load_Input module, which reads input features, as an example, the inter-block index is defined as:

[0114] inter_index=layer_offset+th_index×w_len×cin_len+tw_index×cin_len+tcin_index

[0115] Where `layer_offset` represents the memory offset of this feature layer. `th_index` represents the number of blocks in the H dimension, and `w_len` refers to the length of this feature layer in the W dimension; other parameters are defined similarly. The intra-block offset, based on the inter-block offset, locates the position of each specific data point.

[0116] intra_index=h_index×w_len×cin_len+w_index×cin_len+cin

[0117] During the chunked data reading process, the index within each data block changes. Since w_len represents the length along the W dimension of the entire feature layer, and the block length is only a part of this length, the change in the intra-block index is not continuous but jumps with each loop read operation. Ultimately, the position of a data point within the entire external storage space is determined by both the intra-block offset and the inter-block offset.

[0118] index=inter_index+intra_index

[0119] During the reading and writing of the entire block of data, the triple loop traverses each of the three dimensions, and the index scans each data position in the block. Furthermore, by setting up to four 128-bit AXI buses between the external storage space and the memory access unit of the deep learning accelerator, the throughput of data access is improved. Since the input and weight data of the feature extraction neural network are quantized to 8 bits, and there are up to four 128-bit AXI buses between the external storage space and the memory access unit of the deep learning accelerator, a low-bit-width data concatenation transmission method is designed to allow the memory access unit to read 128 bits of data at once from the external storage via the AXI bus. These 128 bits consist of 16 consecutive 8-bit feature data. The corresponding memory access module uses hardware decomposition to divide the 128-bit data into 8-bit segments and place them into 16 RAM blocks.

[0120] When accessing memory, data is arranged according to the HWC order, meaning data is first arranged along the channel dimension (C), followed by the width dimension (W) and the height dimension (H). Unlike the CHW data arrangement used on GPUs, HWC allows for the reading of continuous data along the C direction during memory access. This enables direct concatenation of C-direction data during low-width data transmission, allowing for hardware expansion within the memory access unit and achieving parallel storage.

[0121] Dispatch and control scheme design

[0122] The scheduling control scheme consists of two parts: inter-layer scheduling and intra-layer scheduling. It handles the scheduling tasks during the inference process of the feature extraction neural network, including the number of scheduling operations, the order of scheduling, and the enabling status. Inter-layer scheduling includes two components: a feature extraction neural network structure register set and a finite state machine (FSM). The feature extraction neural network structure register set stores the parameter data of each layer in the feature extraction neural network. The FSM performs cyclic scheduling of the parameter data of each layer in the feature extraction neural network structure register based on the execution state of the feature extraction neural network. Since inter-layer scheduling only involves two levels of "read-operation," a double-buffering design is used for acceleration. The specific implementation steps of the double-buffering design are: when cache 1 reads data, data in cache 2 is processed; when cache 2 reads data, cache 1 is processed. Double-buffering designs generally require two cache spaces of the same size; however, by combining the quantization calculation strategy and memory access optimization design proposed in this invention, the storage space requirements of the double-buffering design can be effectively reduced. In-layer scheduling controls the specific computations of each layer in a feature extraction neural network, specifically the convolutional layers. Its task can be summarized as: selecting blocks of data containing feature vectors and weight matrices in sequence, and completing the computation flow of "convolution computation – BAQ computation – max pooling computation". Since memory access time for data reading and writing back during in-layer scheduling significantly impacts the inference efficiency of the feature extraction neural network, increasing the proportion of convolution computation during the execution of convolutional layers is the main optimization direction for in-layer scheduling. Therefore, in the design of the scheduling control scheme, the pipeline design of in-layer scheduling is as follows: Figure 8 As shown, during the computation of the convolutional layer in the feature extraction neural network, except for the initial data reading stage and the final BAQ, max pooling, and data write-back stages, the convolutional computations in other stages are all operating at full capacity. By employing a scheduling control scheme, the overall operating efficiency of the feature extraction neural network is accelerated.

[0123] The design ideas and methods for the feature extraction module accelerator are as follows:

[0124] The feature extraction module accelerator is used to accelerate the post-processing of the outputs of the two branches of the feature extraction neural network and generate feature point confidence scores and descriptor vectors for the image. Specifically, the first branch of the feature extraction neural network is the Softmax post-processing unit, whose output is the feature confidence score of each pixel in the image, represented by a float32 type value. Using an 8×8 pixel region as the unit, each region also contains a confidence score representing a "no-feature point" in that region, resulting in a total of 65 confidence scores. The second branch of the feature extraction neural network is the Normalization post-processing unit, which consists of descriptors corresponding to the 8×8 regions. These descriptors are vectors represented by 256 float32 floating-point numbers. To ensure the similarity between different images when calculating distances, each descriptor vector undergoes a Normalization operation. Since the Softmax operation unit involves natural exponentiation, a fast exp function is used to accelerate the computation of natural exponentiation on the hardware; while the Normalization operation unit performs multiplication and accumulation functions on 8-bit integer data.

[0125] The fast exp function calculation method can reduce the clock cycle and required hardware logic resources of the exp function calculation at the cost of a small loss of precision. The calculation of exp is represented by the following formula. Since the exponentiation process with base 2 is closely related to the IEEE-754 floating-point representation, the exponentiation calculation of the natural number e is first converted to exponentiation with base 2. Float32 data can be represented in computer storage as 1 sign bit S, 8 exponent bits E, and 23 mantissa bits M. Therefore, the floating-point number can be represented as:

[0126] y = S × M × 2^E

[0127] Since the exponent E only represents the order of magnitude of the data, and the actual significant bits are represented by M, placing the significant data M from the floating-point number in the exponent bits and then re-representing it as a floating-point number completes the calculation of 2^M. The mantissa M is shifted left by a logical left shift of 23 bits. Because the IEEE-754 exponent E uses offset binary representation, an additional offset of 127 is needed on top of the logical left shift. Let x be the input float32 type data, and y be the result obtained using the fast exp function. The above calculation process can then be reorganized into a formula usable in a computer:

[0128] y = (1 << 23) × (x / ln2 + 127)

[0129] y = 12102202x + 1064866804

[0130] The overall steps can be simply described as follows: First, represent the input float32 type data x as an integer, calculate the result y using the two formulas mentioned above, and then directly represent y as a floating-point number to obtain the result of the exp function. The prerequisite for the 23-bit logical left shift is that the data is stored in little-endian byte order. This ensures that after shifting left by 23 bits, the highest bit of the data is stored precisely at the exponent.

[0131] Using the fast exp function to compute the logic has two advantages, both of which fully utilize the characteristics of hardware acceleration technology and facilitate implementation on the FPGA platform:

[0132] (1) All data in the accelerator is stored in registers and represented in binary 0s and 1s. The above-mentioned integer-to-floating-point type conversion is only a logical conversion. In actual hardware, no changes or data shifts are required. Only basic shift and multiply-accumulate operations are needed.

[0133] (2) This method converts complex floating-point calculations into a single 32-bit integer multiplication and a single 32-bit integer addition, which greatly reduces hardware resource consumption and clock cycle count.

[0134] In the post-processing unit of the feature extraction module accelerator, the number of computation units determines the upper limit of the computation speed achievable by the feature extraction module accelerator: the number of computation units is directly proportional to the hardware resource usage and inversely proportional to the computation clock cycle. Considering the remaining resources of other modules in the deep learning visual SLAM accelerator, a scale of 8 computation units each for Softmax and Normalization is adopted. The computation flow of this module is as follows: First, from the network result data of the deep learning accelerator inference computation, feature confidence data and descriptor data of 8 8×8 pixel regions are taken and sent to the post-processing unit. The 8 Softmax and 8 Normalization computation units in the post-processing unit simultaneously process the data of these 8 regions in parallel, thus obtaining the post-processing results of 8 regions at once. Finally, the processed data is written back to the corresponding address of external storage. Data is transmitted through two 128-bit wide AXI buses to ensure sufficient throughput bandwidth so that the overall computation speed is not affected by memory access bottlenecks.

[0135] The design ideas and methods for the feature matching module accelerator are as follows:

[0136] The feature matching module accelerator is used to accelerate the computation of the feature matching algorithm on hardware. To maximize the accuracy of visual SLAM hardware acceleration technology, this invention employs the following brute-force matching algorithm and bidirectional nearest neighbor criterion to obtain a high-precision feature point matching set:

[0137] (1) Brute-force matching algorithm: Use two-dimensional loop to compare the distance between every two pairs of points in two point sets.

[0138] (2) Bidirectional nearest neighbor criterion: A set of matching points is found only when two points are the points that are closest to each other in their own point sets.

[0139] However, the brute-force matching algorithm has high computational complexity, and frequent memory accesses and descriptor data read / write operations can negatively impact the hardware module's computational efficiency. To achieve parallel computation based on a hardware unfolding method in this module, while minimizing the time consumption caused by memory accesses, this invention optimizes the vector dot product as follows: Since each descriptor is a 256-dimensional vector, n descriptors can form an n×256-dimensional matrix. Therefore, if two images yield two descriptor matrices N and M respectively, and matrix multiplication is performed, each coordinate position in the resulting matrix K is the dot product of the descriptor vectors contained in N and M. Thus, the accelerator computation process can read two matrices at once and utilize the multiply-accumulate matrix unit to perform parallel matrix multiplication, thereby completing all computational operations for the feature matching task in one go.

[0140] The feature matching module accelerator consists of two main parts: a matrix multiplication unit and a comparison unit, such as... Figure 9 As shown. The design concept of the matrix multiplication unit is to combine the above-mentioned optimization strategies for algorithm computation by replicating multiple multiply-accumulate units to form a multiply-accumulate unit matrix, thereby calculating the vector dot product in matrix multiplication in parallel. The multiply-accumulate unit of the feature matching module accelerator has a size of 16×16, totaling 256 units. The computation process of the comparison unit follows immediately after the matrix multiplication unit. Its input is the 16×16 local multiplication result matrix obtained from the parallel computation in the previous step. This result matrix reflects the distance between two sets of 16 feature points each. Only a local maximum dot product value can be obtained through this set of distances. Since the optimal matching result needs to wait for the completion of the full-scale cyclic comparison before it can be determined, the comparison unit of the feature matching module accelerator has an on-chip storage unit, denoted as the temporary nearest neighbor space, used to store the current optimal calculation result of two adjacent frames. The operation steps of the comparison unit can be simply summarized as follows: first, determine the coordinates of this 16×16 local result matrix in the entire result matrix; then, select the maximum dot product value in each row and each column through the comparison tree structure, and update the temporary nearest neighbor space according to its coordinates. In addition, to save hardware resources and data transmission time as much as possible, the internal computing logic of the feature matching hardware acceleration module has been fully implemented with integer quantization, that is, all input descriptor vectors are quantized into int8 type data.

[0141] The design ideas and methods for the pose estimation module accelerator are as follows:

[0142] The pose estimation module accelerator accelerates the calculation process of pose data from depth cameras or platforms and reduces hardware resource requirements through a pose estimation-oriented acceleration strategy. Pose data includes the depth coordinates and pose matrix of feature points in the matching point set. For the pose estimation calculation method, firstly, the depth information corresponding to the feature points in the matching point set can be directly obtained from the image depth data through a binocular parallax or structured light sensor in the depth camera, thus obtaining the depth coordinates of each feature point in the 2D image perpendicular to the image plane. Secondly, since the depth camera introduces observation errors, the pose estimation calculation process can be summarized as solving a pose matrix that minimizes the observation error, i.e., obtaining a globally optimal solution through multiple iterative optimizations. Therefore, the pose estimation module accelerator mainly adopts a nonlinear optimization method based on the Gauss-Newton method to solve the PnP (Positioning-Nearest Phenomenon) problem. The specific incremental equation for the optimization correction objective is:

[0143] J(x)J T (x)Δx=-J(x)f(x)

[0144] HΔx=g

[0145] Where J is the Jacobian matrix represented by Lie algebra, which describes the first-order transformation relationship of the reprojection error with respect to the depth camera pose Lie algebra; H is considered as the Hessian matrix obtained by multiplying J and its transpose; and Δx is the optimization increment of the pose matrix T represented by Lie algebra. The specific process is as follows: through multiple iterative calculations, the increment is updated to the current pose transformation matrix, thereby reducing the sum of observation errors of the current frame image's pose for all landmark points.

[0146] Overall, the pose estimation module accelerator consists of two parts: a Jacobian construction unit and a linear equation solving unit. Control parameters such as depth camera intrinsics and iteration counts are stored in the accelerator's configuration register. The Jacobian construction unit retrieves 3D and 2D matching point pairs from external storage and stores them in the on-chip cache. Each point pair contains 3D and 2D coordinate information represented by five float32 type data points, totaling 20 bytes. Therefore, to improve the hardware speed of pose estimation calculations and limit their computational scale, the accelerator strategy for the pose estimation module stipulates: 1) To control the scale of the pose estimation task and avoid the influence of mismatched points, the upper limit for the number of matching point pairs used in each pose estimation calculation is 200 pairs, selected according to the descriptor distance from smallest to largest; 2) The Jacobian construction unit cyclically reads the coordinate data of each matching feature point pair and simultaneously calculates the values ​​of 10 positions in parallel; 3) The calculation results are accumulated onto the Hessian matrix of the linear equation. After traversing all matching point pairs, the linear equation solving unit performs matrix decomposition and elimination inversion based on the constructed linear equation. To control the computational scale, this invention limits the number of iterations to a maximum of 10, and automatically exits the loop when the magnitude of the increment Δx is less than 10e-6.

[0147] The deployment and control program design for a deep learning-based visual SLAM acceleration system on an FPGA embedded platform is as follows:

[0148] The deep learning accelerator, feature extraction module accelerator, feature matching module accelerator, and pose estimation module accelerator are deployed on the programmable logic (PL) of the FPGA embedded platform. The ARM in the FPGA embedded platform serves as the external controller for the deep learning-based visual SLAM acceleration system, and the external storage space is the 4G DDR4 on-chip storage space of the FPGA. Figure 1 The arrows indicate the direction of data transmission. The external controller is connected to the deep learning accelerator, feature extraction module accelerator, feature matching module accelerator, and pose estimation module accelerator via a 32-bit AXI-Lite bus for transmitting status and control commands. The data transmission interface between the external storage space and the deep learning accelerator, feature extraction module accelerator, feature matching module accelerator, and pose estimation module accelerator is connected via six 128-bit AXI buses. The implementation logic of the deep learning-based visual SLAM acceleration system is written in C++ and converted to Verilog using High-Level Synthesis (HLS) technology for hardware synthesis and implementation of the programmable logic section. It supports the use of an Intel RealSense D435i depth camera as an image acquisition device, which can simultaneously acquire 480×640 resolution visible light and depth images at a rate of 30fps.

[0149] The control program design for a deep learning-based visual SLAM acceleration system includes the design of the external controller program on the FPGA embedded platform and the host computer program design.

[0150] External controller programming

[0151] The external controller program sets up four different SLAM algorithm system states to handle scheduling tasks in different scenarios. For example... Figure 10 As shown, the workflow and transition logic of the external controller program in different states are as follows: In the initialization state, since there is no previous frame image, only feature extraction is performed on the first input frame image, and the pose transformation matrix of the first frame is initialized to the map coordinate origin; in the normal tracking state, when the matching effect between two frames is good, the hardware module is called to perform feature extraction, feature matching, and pose estimation tasks respectively, and then the current frame is updated to the previous frame for subsequent processes; in the bad match state, the number of matching point pairs between two frames is less than 100 pairs. Too few matching point pairs are insufficient to support the pose estimation task, so the previous frame is not updated; in the lost tracking state, if 10 consecutive frames are in the lost tracking state, the entire SLAM algorithm's view tracking is lost, and the pose localization task fails. Furthermore, the control bus uses a 32-bit AXI-Lite bus connected to the three hardware module IPs on the PL side. External storage is provided by the FPGA's 4GB DDR4 on-chip memory, which is connected to the accelerator's data interface via six 128-bit AXI buses under the ARM's DMA control. The accelerator's implementation logic is written in C++ and converted to Verilog using High-Level Synthesis (HLS) technology for hardware synthesis and implementation of the programmable logic section.

[0152] Host computer programming

[0153] The host computer program is designed with four threads. The main thread controls the program's startup and multi-threaded task management. The data loading thread acquires images from the depth camera or reads images sequentially from the dataset, transmitting them as a byte stream to the FPGA embedded platform via TCP. The data receiving thread also receives pose transformation matrix data and feature point information returned by the computing platform via TCP. The last thread handles system visualization and converts the pose transformation matrix into quaternion format, writing it to the trajectory result text. This thread calculates the computing platform's operating speed by subtracting the current time from the program's start time and dividing by the total number of frames processed, for performance comparison across multiple platforms.

[0154] The parts of this invention not disclosed in detail are well-known technologies in the field.

[0155] Those skilled in the art will understand that all or part of the methods in the various embodiments described above can be implemented by program instructions and related hardware. The program can be stored in a computer-readable storage medium, which may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0156] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A deep learning-based visual SLAM acceleration system, characterized in that... comprising: a deep learning accelerator, a feature extraction module accelerator, a feature matching module accelerator, a pose estimation module accelerator, an external storage space, and an external controller; image data collected by a depth camera or image data in a data set is stored in the external storage space, and is transmitted to the deep learning accelerator under the scheduling of the external controller; the deep learning accelerator, comprises a multi-combination acceleration strategy for a feature extraction neural network; the multi-combination acceleration strategy comprises: a blocking operation strategy, a quantization calculation strategy, operator design, storage space arrangement, memory access unit design, and scheduling control scheme design; the image feature point tensor and descriptor tensor extracted by the feature extraction neural network are transferred and stored to the external storage space; the blocking operation strategy performs a blocking operation on calculation data of the feature extraction neural network, so as to reduce hardware resources occupied by calculation of the feature extraction neural network, and involved parameters include: (1) the height dimension H and width dimension W of the blocked features in one operator layer of the feature extraction neural network; (2) the input channel dimension CIN and output channel dimension COUT of the blocked features; (3) the number of blocks in the feature height dimension TH and feature width dimension TW; (4) the number of blocks in the feature input channel dimension TCIN and output channel dimension TCOUT; (5) the height dimension KH and width dimension KW of the blocked weight convolution kernel; the blocking operation strategy includes a post-blocking loop mode and blocking size selection, the post-blocking loop mode performs blocking processing on the four dimensions of the feature: width, height, input channel and output channel respectively, so that the width of the feature for one convolution layer operation becomes W×TW, and the other dimensions follow the same rule; the quantization calculation strategy is configured to reduce the hardware resource consumption and structural complexity of the deep learning accelerator, and integer quantized data is adopted in the entire operation process; the operator design is configured to accelerate operation units in the neural network, and comprises a convolution calculation core acceleration strategy, a BAQ calculation core acceleration strategy, and a max pooling calculation core acceleration strategy; the convolution calculation core acceleration strategy is configured to execute convolution acceleration calculation on blocked data; the BAQ calculation core acceleration strategy is configured to perform acceleration calculation on bias values, activation functions and quantization scaling in a convolution layer; the max pooling calculation core acceleration strategy is configured to perform max pooling acceleration calculation operation in the convolution layer, and after four adjacent "field"-shaped convolution blocks in the [H,W] dimension are all scheduled and complete calculation, pooling operations are performed on the four "field"-shaped convolution blocks at the same time, wherein H and W are the height dimension and width dimension of the blocked features; the convolution calculation core acceleration strategy, the BAQ calculation core acceleration strategy, and the max pooling calculation core acceleration strategy form a continuous accelerated calculation flow, and the internal operation process of the entire convolution layer "convolution - activation function - max pooling" does not involve additional memory access overhead for reading and writing the external storage; the storage space arrangement is divided into an input space, an output space and a temporary storage space; the memory access unit design includes internal transmission design and external transmission design; The scheduling control scheme design includes inter-layer scheduling and intra-layer scheduling; the inter-layer scheduling is used to control the execution state of the feature extraction neural network and store the parameter data of each layer; the intra-layer scheduling is used to control the computing modules in the feature extraction neural network. The feature extraction module accelerator includes a post-processing unit computation acceleration strategy and Softmax and Normalization post-processing unit size settings. Image feature point tensors and descriptor tensors are passed from external storage space to the feature extraction module accelerator. The image feature point tensors are used to calculate feature point confidence vectors through the Softmax post-processing unit acceleration strategy. The descriptor tensors are used to calculate descriptor vectors through the Normalization post-processing unit size settings. The feature point confidence vectors and descriptor vectors are then transferred to external storage space. The feature point confidence vector and descriptor vector are passed from the external storage space to the feature matching module accelerator. The feature matching module accelerator uses the feature matching acceleration strategy to accelerate the matrix multiplication calculation of the descriptor matrix and reduce the hardware resource requirements. The descriptor matrix is ​​composed of the feature point confidence vector and the descriptor vector. The resulting set of matching points is transferred to the external storage space. The matching point set and image depth data are fed into the pose estimation module accelerator through the control of the external controller. The pose estimation module accelerator uses a pose estimation-oriented acceleration strategy to accelerate the pose calculation process of the depth camera or platform and reduce the hardware resource requirements. The resulting pose data of the depth camera or platform is transferred to external storage space. Image data and pose data in the external storage space are transmitted to the host computer through the control of the external controller to realize the visualization of SLAM information. The visualization of SLAM information includes the visualization of the running trajectory of the depth camera or platform, the visualization of the corresponding 3D sparse map, the statistics of running rate per frame, and the local saving function of pose trajectory text. External controllers are used for communication, driver invocation, and data transfer operations.

2. The deep learning-based visual SLAM acceleration system according to claim 1, characterized in that: In the deep learning accelerator, among the multiple combined acceleration strategies: The block-based looping method involves splitting the features in the feature extraction neural network into four dimensions: width, height, input channel, and output channel. This transforms the loop calculations in the convolution operation of the feature extraction neural network into: TH×TW×TCIN×TCOUT×(H×W×CIN×COUT×KH×KW) The block size selection is as follows: H=30, W=40, CIN=16, COUT=64; The quantization calculation strategy is used to reduce the hardware resource usage and structural complexity of the deep learning accelerator, and integer quantization data is adopted in the entire operation process, wherein the weights and intermediate layer results of the feature extraction neural network use 8-bit width, and the accumulated sum result of the convolution operator uses 32-bit width; the methods used in the quantization calculation strategy are: (1) General conversion of floating-point numbers: setting a quantization scaling factor to establish a mapping relationship between the floating-point number to be converted and the converted integer; (2) Integer quantization in convolution operation: substituting the integers converted from floating-point numbers into convolution calculation, then converting the newly appeared floating-point numbers in the convolution calculation into integers and performing arithmetic right shift operations to realize the overall quantization of the convolution operation; The convolution computing core acceleration strategy is: performing hardware unrolling on the two layers of loops of CIN and COUT in the convolution operation of the weight matrix and the feature vector, and converting the convolution operation into a parallel dot product operation in hardware; it is used to improve the parallelism of the convolution operation of the convolution layer, and at the same time, the convolution operation process is divided into pipeline stages according to the clock cycle frequency, so as to further reduce the calculation cycle in the parallel operation process; The described BAQ computing core acceleration strategy is: (1) Performing hardware unrolling on the COUT loop in the offset value, activation function, and quantization scaling operation to improve calculation parallelism; (2) Setting the activation function operation after the last layer of convolution of the feature extraction neural network structure to a bypassable form Bypass; (3) Dividing the BAQ calculation process into pipeline stages according to the clock cycle frequency to reduce the calculation cycle in the parallel operation process; The maximum pooling computing core acceleration strategy is used for the maximum pooling accelerated calculation operation in the convolution layer. This strategy does not directly perform the pooling operation on one block, but waits until four adjacent convolution blocks in the [H,W] dimension arranged in a "field" shape are all scheduled and the calculation is completed, then performs the pooling operation on these four blocks at the same time. During the pooling operation, hardware unrolling is performed on the COUT dimension of the block data, and a pipelining operation divided according to the clock cycle frequency is executed; The described input space includes Input, Weight, Bias, and M spaces, which are respectively used to store input features, weight parameters, offset parameters, and quantization scaling parameters; The input space has a unit width of 8 bits and a unit size of [missing information]. The number of units is 4, and the split dimension is CIN; The weight space has a unit width of 8 bits and a unit size of [missing information]. The number of cells is 4, and the splitting dimensions are CIN and COUT; The unit bit width of the Bias space is 32bit, the unit size is COUT, the number of units is 2, and the splitting dimension is COUT; The unit bit width of the M space is 32bit, the unit size is COUT, the number of units is 2, and the splitting dimension is COUT; The described output space includes Output and Pooling spaces, which are respectively used to store the calculation results of the convolution layer and the pooling layer to be output; The output space has a unit width of 8 bits and a unit size of [missing information]. The number of units is 4, and the split dimension is COUT; The pooling space has a cell width of 8 bits and a cell size of [missing information]. The number of units is 1, and the split dimension is COUT; The described temporary storage space Accum is only used for temporarily storing the accumulated results of the convolution computing core; The Accum space has a unit width of 32 bits and a unit size of [missing information]. The number of units is 4, and the split dimension is COUT; The described internal transmission design divides the memory access function into the following modules: Load_Input, Load_Weight, Load_Bias, Load_M, and Store modules; The described Load_Input and Load_Weight modules are used to read 8bit data into the Input and Weight spaces of the storage space; The Load_Bias and Load_M modules are used to read the 32-bit Bias and M spaces; The Store module is used to write 8-bit Output and Pooling data back to external storage; The external transmission design includes a maximum of four 128-bit AXI bus lines between the external storage and the memory access unit, and a low-bit-width data splicing transmission method. The memory access unit reads 128 bits at a time from the external storage via the AXI bus, which is 16 consecutive 8-bit data segments. The memory access module corresponding to each storage space uses hardware expansion to split the 128-bit data into 8-bit segments and place them into 16 RAM segments. The feature data dimension arrangement order of the memory access unit in the external storage and the storage unit is as follows: first, the data is arranged according to the channel dimension, and then according to the width dimension and the height dimension in sequence. The inter-layer scheduling adopts a dual-buffer design. The specific read-write alternation steps are as follows: when buffer A reads data, data in buffer B is processed; when buffer B reads data, data in buffer A is processed. The in-layer scheduling is used to control the order of operations in the computation module of the feature extraction neural network: (1) select feature and weight blocks; (2) schedule and complete the computation process of "convolution calculation - BAQ calculation - max pooling calculation" of the convolutional layer.

3. The deep learning-based visual SLAM acceleration system according to claim 1, characterized in that: In the feature extraction module accelerator: The post-processing unit acceleration strategy employs a fast... The function computation method accelerates the exp function calculation process involved in the Softmax calculation of feature point tensors. The method is as follows: 1) Represent the input float32 type data as an integer, and store the data in little-endian byte order in the hardware; 2) Directly represent the converted integer as a floating-point number as the result of the exp function. The post-processing unit is configured with 8 Softmax and 8 Normalization operation units, and the AXI transmission bus consists of two 128-bit wide lines.

4. The deep learning-based visual SLAM acceleration system according to claim 1, characterized in that: The acceleration strategy for feature matching in the feature matching module accelerator includes matrix multiplication calculation unit design and comparison unit design. The descriptor matrix consists of feature point confidence vectors and descriptor vectors; The matrix multiplication calculation unit is designed to adjust the loop order of vector dimension D, previous image descriptor matrix dimension M, and current image descriptor matrix dimension N in the feature matching operation, so that dimension D is placed at the outermost layer of the loop, and hardware expansion operations are performed on dimensions M and N to realize parallel dot product operation and improve the parallelism of the overall loop calculation. The matrix multiplication calculation unit is designed to have a scale of 16×16, consisting of 256 multiply-accumulator units. The comparison unit is designed to store the local optimal results of two adjacent images in both directions, i.e., the maximum value of the local dot product calculated by the matrix multiplication unit, in an on-chip storage unit; the size of the on-chip storage unit is capable of storing twice the number of temporary nearest neighbor results of the maximum feature points. The feature matching module accelerator's internal computation logic adopts the integer quantization design in deep learning accelerators, quantizing all input descriptor vectors into int8 type data.

5. The deep learning-based visual SLAM acceleration system according to claim 1, characterized in that: In the pose estimation module accelerator, the acceleration strategy for pose estimation adopts a parameter upper limit setting method. The parameter upper limit settings include: upper limit setting for matching feature point pairs and upper limit setting for the number of iterations during pose solving; The upper limit for matching feature point pairs is set to 200 pairs for each calculation, selected in ascending order of descriptor distance; The upper limit of the number of iterations in the pose solving process is set to 10 iterations. When the incremental magnitude in the iterative solving process is less than 10e-6, the loop will automatically exit and the solution will end.

6. The deep learning-based visual SLAM acceleration system according to claim 1, characterized in that: The deep learning accelerator, feature extraction module accelerator, feature matching module accelerator, and pose estimation module accelerator are deployed on the programmable logic (PL) of the FPGA embedded platform. The ARM in the FPGA embedded platform acts as an external controller to control the deep learning-based visual SLAM acceleration system, handling communication, driver calls, and data transfer operations between the deep learning-based visual SLAM acceleration system and the host computer. This external controller is connected to each accelerator in the programmable logic via a 32-bit AXI-Lite bus. The external storage is 4GB of DDR4 on-chip memory on the FPGA embedded platform. Under the DMA control of the FPGA embedded platform ARM, this on-chip memory is connected to the data interfaces of each accelerator through six 128-bit AXI buses.

7. A deep learning-based visual SLAM acceleration method based on the system described in any one of claims 1-6, characterized in that: Step 1: The image data acquired by the depth camera or in the dataset is scheduled and fed into the deep learning accelerator. In the deep learning accelerator, a multi-combination acceleration strategy is adopted to accelerate the inference process of the feature extraction neural network. The multi-combination acceleration strategy includes block operation strategy, quantization calculation strategy, operator design, storage space arrangement, memory access unit design and scheduling control scheme design, to obtain image feature point tensor and descriptor tensor. Step 2: Based on the image feature point tensor and descriptor tensor obtained in Step 1, the input is passed to the feature extraction module accelerator. In the feature extraction module accelerator, the post-processing unit acceleration strategy and post-processing unit size setting are adopted to accelerate the calculation of the Softmax and Normalization post-processing units of the image feature point tensor and descriptor tensor, and the feature point confidence vector and descriptor vector are obtained. Step 3: In the feature matching module accelerator, the matrix multiplication calculation unit design and comparison unit design are used to accelerate the operation process of the feature point confidence vector and descriptor vector matrix obtained in Step 2, so as to obtain the matching point set; Step 4: Based on the matching point set obtained in Step 3, an acceleration strategy oriented towards pose estimation is adopted in the pose estimation module accelerator to accelerate the pose calculation of the depth camera or platform and obtain the pose data of the depth camera or platform.

Images