A kind of suitable for SLAM nonlinear parallelization chip computing architecture and working method

Abstract

This invention discloses a chip computing architecture suitable for nonlinear parallelization of SLAM, including at least one block-structured parallelized matrix multiply-accumulate unit based on a systolic array, used to decompose a large-scale matrix into block-structured matrix multiply-accumulate operations of up to 6×6 size; at least one iterative solver for a preprocessing conjugate gradient method to solve large-scale symmetric positive definite matrix equations; and a hardware mapping module for processing and analyzing complex data flows in SLAM back-end optimization. This invention provides an accelerated architecture for SLAM back-end optimization, implementing a hardware accelerator for back-end optimization based on bundle adjustment, which can be flexibly applied to the computation of back-end optimization components of various SLAM algorithms, featuring flexible configuration, high computation speed, and low power consumption.

Description

Technical Field

[0001] This invention belongs to the field of computing acceleration chip technology, and in particular relates to a nonlinear parallel chip computing architecture and working method suitable for SLAM. Background Technology

[0002] Existing technologies typically implement SLAM algorithms using general-purpose processors, which cannot meet the demands of common real-time SLAM operations. This approach usually reduces the frequency of backend optimizations to meet real-time requirements, which significantly reduces backend optimization performance. Another way to implement SLAM algorithms is to perform the computation on a graphics processing unit (GPU). However, due to the complexity of SLAM operations, this method cannot fully support subsequent calculations, thus failing to achieve a globally effective acceleration.

[0003] To address the bottleneck of computational power, manufacturers and research institutions are seeking to accelerate robotic algorithms through hardware design. eSLAM proposes an energy-efficient architecture for real-time ORBSLAM on an FPGA platform, accelerating the feature extraction and matching stages, thus enabling real-time SLAM algorithms on a low-power platform. However, eSLAM only slows down the front-end, neglecting the more computationally demanding back-end computations.

[0004] Intel has proposed a general-purpose multi-robot system that incorporates multiple functions such as SLAM and path planning, enabling tasks in search and rescue, precision agriculture, and industrial automation. This system uses a general-purpose processor for robot computation, integrating a host processor for sensor data acquisition and preprocessing; a TensilicaDSP processor for localization / mapping, collision avoidance, and collaborative intelligent decision-making; dedicated path planning and motion control hardware accelerators; an audio accelerator for human speech detection; and a CNN accelerator for object detection and recognition. However, there is no dedicated accelerator specifically designed for SLAM. The University of Michigan has proposed a parallel processor to accelerate the semi-global matching process. It enables dense, real-time 3D depth and 3D motion perception for pose design, achieving neighbor-guided semi-global matching at full HD (1920×1080, FHD) resolution, thus enabling real-time autonomous flight of UAVs at full HD resolution. However, the problem is that the accelerator only accelerates the pose estimation part, which is a very small module in a SLAM system.

[0005] Furthermore, existing SLAM-related patents, such as SLAM computing devices and methods, implement a SLAM hardware accelerator device comprising three main parts: storage, computation, and control. These devices disclose acceleration mechanisms for vector and matrix operation units, effectively accelerating SLAM algorithms according to different needs. They are applicable to various SLAM algorithms and diverse input data types, meeting diverse computational requirements and offering advantages such as high flexibility, high configurability, fast computation speed, and low power consumption. However, their matrix operations are designed for 16-dimensional square matrices and lack specific backend optimizations, representing an acceleration of general SLAM computations.

[0006] The patent titled "An FPGA Accelerator for Known Self-Position Beam Adjustment in SLAM" discloses an FPGA accelerator for known pose update. It implements two hardware modules: a rotation matrix processor and a reprojection processor. By dividing the computation stages, the two modules achieve a balance in computation time and reuse of intermediate variables, thereby improving the computation speed and saving hardware resources.

[0007] However, this invention is based on scenarios where the pose is known, and its acceleration method is only one step in the bundle adjustment method. The more computationally intensive key steps are not included, so its applicability is relatively limited. Summary of the Invention

[0008] The purpose of this invention is to address the poor adaptability of existing standalone acceleration front-ends in dedicated SLAM hardware acceleration structures used for back-end optimization processes, and to propose a chip computing architecture and working method suitable for SLAM nonlinear parallelization.

[0009] To achieve the above objectives, the present invention adopts the following technical solution:

[0010] This invention provides a nonlinear parallel chip computing architecture suitable for SLAM, including at least one block-structured parallel matrix multiply-add unit based on a systolic array, used to decompose a large-scale matrix into a block-structured matrix parallelization multiply-add operation with a maximum size of 6×6.

[0011] At least one iterative solver for a preprocessing conjugate gradient method for solving large-scale symmetric positive definite matrix equations;

[0012] And a hardware mapping module is used for processing and analyzing complex data flows in SLAM backend optimization.

[0013] Preferably, the iterative solver solves the positive definite matrix based on the preprocessed conjugate gradient method by performing block-structured parallel computation on the matrix equations constructed by Schul in a block-structured manner, thereby obtaining the optimized change in camera pose parameters and thus the optimized change in map points.

[0014] Preferably, the iterative solver uses the parallelization of the Schur complement matrix to construct an acceleration unit to quickly perform Schur resolution on the Hessian matrix of the observation projection coordinate error for m points on the 3D map and n poses of the 6D camera, thereby reducing the large-scale matrix equation to a size of 6n×6n.

[0015] Preferably, the number of map points m is greater than the number of camera poses n. The parallelization of the Schur complement matrix construction acceleration unit reduces the size of the computation matrix. Matrix operations will be performed and accelerated by the parallelized matrix operation unit, and the number of such units is determined by the accelerator resources.

[0016] This invention provides a working method suitable for SLAM nonlinear parallelized chip computing architecture, specifically including the following steps.

[0017] Step 1: Pre-calculate the correlation matrix. This step will be actually calculated and accelerated by the matrix operation unit.

[0018] Step 2: The iterative solver constructs the Schur complement matrix in parallel using the Schur complement construction acceleration unit;

[0019] Step 3: Use the dedicated matrix iterative calculation acceleration unit of the preprocessed conjugate gradient method to iteratively solve the matrix equation and obtain the changes in camera pose parameters;

[0020] Step 4: The matrix operation unit participates in accelerating the calculation of spatial changes of map points.

[0021] The present invention also provides a computer storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in the first aspect.

[0022] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0023] 1. An accelerated architecture for SLAM back-end optimization is proposed, which implements a hardware computing accelerator for back-end optimization based on the bundle adjustment method. It can be flexibly applied to the computing of back-end optimization components of various SLAM algorithms and features flexible configuration, fast computing speed and low power consumption.

[0024] 2. Hardware accelerators can reduce the computational cost of SLAM algorithms, thereby improving their performance. Simultaneously, dedicated computing architectures designed at both the hardware and software levels will increase computing power and reduce hardware costs, thus lowering the barriers to robot design and application, and enhancing the performance and scalability of SLAM systems.

[0025] 3. The camera pose parameters and map point information were updated. The computation of the bundle adjustment method for the SLAM backend optimization algorithm was accelerated through a hardware parallel acceleration computing framework. This improved the computing power of the SLAM system hardware, increased the computing speed, and saved computing time. This laid the foundation for extending the SLAM algorithm to more complex algorithms, achieving high performance of the SLAM system and expanding the application scenarios of the SLAM system. Attached Figure Description

[0026] Figure 1 This is a schematic diagram of the overall structure of a nonlinear parallel chip computing architecture for SLAM proposed in this invention.

[0027] Figure 2 This invention provides a flowchart for multiplying and adding a 6-dimensional square matrix.

[0028] Figure 3 This is a schematic diagram of the structure of a Schulp construction unit proposed in this invention.

[0029] Figure 4 This is a schematic diagram of the preprocessing gradient descent matrix solver unit proposed in this invention.

[0030] Figure 5 This is a schematic diagram of the SLAM backend optimization hardware acceleration structure proposed in this invention. Detailed Implementation

[0031] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0032] A classic and effective method for solving the nonlinear optimization problem in the SLAM backend is bundle adjustment. In bundle adjustment, the number of map points can reach tens of thousands compared to the number of camera pose parameters. Even using keyframes and sparsity features, each keyframe typically observes hundreds of keypoints. Furthermore, the computation involves numerous matrix operations and complex matrix equation solving. Specifically, the SLAM algorithm's backend optimization computation consumes a significant amount of time in constructing the Schur complement equation, solving large-scale positive definite matrices, and performing other matrix operations. This also consumes substantial computational resources on current mainstream CPUs, placing high demands on CPU computing power.

[0033] In this embodiment, the Levenberg-Marquardt (LM) algorithm is used for optimization during the bundle adjustment method solution process, as shown in the following equation.

[0034]

[0035] Where x represents the parameters to be optimized, divided into two parts: camera pose parameters (sets) and map point parameters (m sets). f(x) is the perspective projection function, J(x) is the Jacobian matrix of parameters x, D(x) is the diagonal matrix of the Jacobian matrix, ∈ represents the error between the actual map points and the camera model predicted projection points, λ represents the LM algorithm parameters, and δ * Let the calculated optimization change be... The matrix equation can be obtained, as shown in formula (2):

[0036]

[0037] Based on the above problems and their solution methods, the following special properties of SLAM back-end linear optimization are studied:

[0038] First, matrix operations are currently the main bottleneck affecting computational performance in backend optimization. In the pre-computation of Jacobi matrix and Hessian matrix, the large-scale equations can be resolved by the Shure resolution method as shown in formula (3). The matrix equations are solved by using the preprocessing conjugate gradient method. Matrix operations are required in all these key steps.

[0039] Second, since the Jacobian matrix has a sparse structure, the first term on the left side of formula (2) has a special sparse block structure matrix, and each block structure can be operated in parallel in each step of the calculation without affecting each other. Therefore, there are methods and devices for designing parallel operation modules to accelerate the calculation.

[0040]

[0041]

[0042] The computing architecture for hardware acceleration of the SLAM algorithm backend optimization in this embodiment is as follows: Figure 1 As shown, it is mainly divided into a matrix operation acceleration unit, a preprocessing conjugate gradient method matrix equation solving acceleration unit, and other parts supporting SLAM algorithm data storage and algorithm control. Data transmission between the bus, storage module, general-purpose processor, and dedicated operation module is controlled and scheduled using a general-purpose processor. The matrix operation acceleration unit implements fast multiplication and addition operations between matrices, vectors, and scalars. The Schur complement matrix parallelization construction acceleration unit solves large-scale matrix equations to construct Schur complement matrix equations. The preprocessing conjugate gradient method matrix equation solving acceleration unit solves the matrix equations in parallel using various block structures.

[0043] like Figure 2As shown, computation is performed using a pulsating array approach. The computational units form a computational network based on a data flow relationship. Data is read from the starting addresses of the two matrices to be computed according to instructions, and then sequentially input into the designated units and passed to the multidimensional computational units. In each clock cycle, data from the multidimensional computational units is passed to the right / downwards. During computation, each computational unit performs multiplication on the input data, adds the intermediate value stored in that unit, and obtains the computation result. The result is stored in the current unit and simultaneously passed to the adjacent right / downward computational unit. Data already computed in a computational unit can be directly passed to the multidimensional computational unit as an intermediate value, utilizing data reusability to significantly reduce the number of data transfers. The above calculation and transmission steps are repeated until all matrices to be computed are input into the multidimensional computational unit and computation is completed. The final result is then output to the storage destination address specified by the instructions, thus completing the entire computational instruction flow. Furthermore, this device can change the size of the data stream input via instructions, thereby enabling operations such as vector multiplication and matrix multiplication and addition between 1×3 vectors, 1×6 vectors, 3D square matrices and 3×6 matrixes.

[0044] like Figure 3 As shown, the Schul complement construct unit structure provided in this embodiment solves the equation with a 3m+6n square matrix on the left side. The size can be reduced to a 3m square matrix by Schul complement elimination, which greatly simplifies and accelerates the subsequent matrix equation solving process.

[0045] The unit input includes six types of matrices, such as projection error, camera parameter Jacobian matrix, and map point parameter Jacobian matrix. Based on the data dependency of the calculation, it is divided into five parts, totaling 12 calculation stages. At the same time, several different parts are further divided into five calculation stages according to the amount of computation and data dependency to balance the calculation delay and increase the calculation speed.

[0046] The shaded area represents a dedicated matrix multiplication and addition unit.

[0047] The first part calculates the Jacobian matrix J of the map points. p The transpose of the matrix multiplied by itself, V = ∑J pT J p And calculate the inverse V of its matrix. -1 ;

[0048] The second part calculates the Jacobian matrix J of the map points. p and projection error e p Matrix multiplication ∈ p Then, sequentially with matrix V -1 J p and J cPerform matrix multiplication and addition operations to obtain J. c J p V -1 ∈ p ;

[0049] The third part calculates the Jacobian matrix J with respect to the camera parameters. c and projection error e c Matrix multiplication ∈ c Subtracting the result from the first part gives b. schur =∈ c -∑J cT J p V -1 ∈ p ;

[0050] The fourth part calculates the Jacobian matrix J of the camera parameters. c The transpose of the map point and the Jacobian matrix J c The transpose and matrix multiplication and addition operations W T =∑J pT J c After calculating its transpose, it is then sequentially compared with matrix V. -1 J p and J c Perform matrix multiplication and addition operations to obtain J. cT J p V -1 W T ;

[0051] The fifth part calculates the Jacobian matrix J of the camera parameters. c The transpose of the matrix multiplied by itself, U = ∑J cT J c Then, perform matrix subtraction with the result from Part 4, H. schur =U-∑J cT J p V -1 W T .

[0052] The intermediate variables generated in the five computational stages are stored in on-chip RAM or in a register file. Simultaneously, the RAM size between computational stages is expanded to form a Ping-pong buffer structure to improve computational parallelism. This completes the construction of the solution equations after Schur complement elimination.

[0053] like Figure 4 As shown, the acceleration unit in this embodiment includes four operations: matrix multiplication (represented by dashed box), vector dot product (represented by double solid box), vector axpy operation (represented by shaded box), and scalar operation (represented by single solid box).

[0054] Detailed explanation:

[0055] The preprocessing gradient descent matrix solution method is divided into three computational stages to balance computational latency and increase computational speed, based on the computational complexity and data dependencies. Before using this unit for solution, the data for each part, such as the first row of Algorithm 1, is initialized according to the algorithm. The equation Ax = b corresponds to the matrix in this embodiment with the following relationship: A := H schur b:=b schur .

[0056] The first stage calculates rows 8, 9, and 10 in Algorithm 1;

[0057] The second step calculates steps 5 and 4 of the algorithm, and the calculation is performed in parallel in two parts.

[0058] In the third stage of the computation algorithm, rows 6 and 7 can also be calculated in parallel as two parts.

[0059] The matrices x, r, w, and p generated in each iteration will be used as data to be updated in the next iteration until the exit conditions satisfied by rows 2 and 3 in Algorithm 1 are met.

[0060] The optimized matrix x is finally obtained. Intermediate variables generated in the three computational stages are stored in on-chip RAM or a register file. Simultaneously, the RAM size between computational stages is expanded to form a Ping-pong buffer structure to improve computational parallelism. At this point, the unit has completed the preprocessing gradient descent matrix solution, obtaining the optimized matrix x, which is the optimized matrix of the camera parameters.

[0061] Algorithm 1: Preprocessing Conjugate Gradient Algorithm

[0062]

[0063] like Figure 5 As shown, this embodiment aims to accelerate the back-end optimization process of the SLAM algorithm in parallel, reduce data exchange, and save storage space.

[0064] To control the SLAM backend optimization process more efficiently, the preprocessing matrix operation part, the construction of the matrix equation by Schur complement, and the calculation of the positive definite matrix equation by PCG are mainly accelerated through bus connection. Therefore, each unit is executed in a pipeline manner, or it can be executed concurrently.

[0065] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A nonlinear parallel chip computing architecture suitable for SLAM, characterized in that: It includes at least one block-structured parallelized matrix multiply-add unit based on a pulsating array, used to decompose a large-scale matrix into a block-structured matrix of up to 6×6 size for parallelized multiply-add operations, and to perform matrix operations for pre-computing the correlation matrix and accelerating the calculation of spatial changes in map points; at least one iterative solver for solving large-scale symmetric positive definite matrix equations using a preprocessing conjugate gradient method. This iterative solver integrates a Schur complement construction acceleration unit and a dedicated matrix iterative calculation acceleration unit for the preprocessing conjugate gradient method. The Schur complement construction acceleration unit performs fast Schur resolution on the Hessian matrix of the observation projection coordinate errors of 3D map points and 6D camera poses and constructs the Schur complement matrix in parallel. The dedicated matrix iterative calculation acceleration unit iteratively solves the block-structured Schur construct matrix equations in a block-structured parallel operation to obtain the changes in camera pose parameters. The iterative solver is optimized based on the Levenberg-Marquart (LM) algorithm, and the core calculation follows the formula: (1), make The matrix equation is derived as follows: (2), and the formula was obtained through the Schuler elimination method: , (3), of which The parameters to be optimized include camera pose parameters and map point parameters. For perspective projection functions, For parameters Jacobian matrix, It is the diagonal matrix of the Jacobian matrix. This represents the error between the actual map points and the camera model's predicted projection points. For LM algorithm parameters, To calculate the optimized change, The change in camera pose parameters. The amount of change in map point parameters; It also includes a hardware mapping module, which is used for processing and analyzing complex data flows in SLAM backend optimization, providing data flow support for matrix operations and equation solving in each unit.

2. The chip computing architecture suitable for SLAM nonlinear parallelization according to claim 1, characterized in that, The iterative solver solves the positive definite matrix based on the preprocessed conjugate gradient method by performing block-structured parallel computation on the matrix equations constructed by Shure, thereby obtaining the optimized changes in camera pose parameters and thus the optimized changes in map points.

3. The chip computing architecture suitable for SLAM nonlinear parallelization according to claim 1, characterized in that, The iterative solver uses a parallelized Schur complement matrix to construct an acceleration unit that processes 3D map points. Individual and six-dimensional camera pose One method is used to quickly perform Schuler elimination on the Hessian matrix of the observed projection coordinate error, reducing the large-scale matrix equation to a size of 6n×6n.

4. A chip computing architecture suitable for SLAM nonlinear parallelization according to claim 3, characterized in that, Number of map points Greater than the number of camera poses The parallelization of the Schur complement matrix reduces the size of the computation matrix. Matrix operations are performed and accelerated by the parallelized matrix operation units, and the number of these units is determined by the accelerator resources.

5. The working method for a nonlinear parallel chip computing architecture of SLAM according to claim 1, characterized in that, Specifically, the steps include the following: Step 1: Pre-calculate the relevant matrix, which will be actually calculated and accelerated by the matrix operation unit; Step 2: The Schur complement construction acceleration unit of the iterative solver performs parallel construction of the Schur complement matrix; Step 3: Use the dedicated matrix iterative calculation acceleration unit of the preprocessed conjugate gradient method to iteratively solve the matrix equation and obtain the changes in camera pose parameters; Step 4: The matrix operation unit participates in accelerating the calculation of spatial changes of map points.

6. A computer storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the method as described in claim 5.

Images