A digital image processing acceleration method and system for bundle adjustment algorithm

Abstract

The invention discloses an acceleration method and system for digital image processing of a bundle adjustment algorithm, which belongs to the field of digital image processing and includes the steps of storing input observation data and camera parameters; using the observation information of the camera number corresponding to the three-dimensional point in the data, distributing the observation data to be calculated to each calculation pipeline indiscriminately, and realizing the parallel calculation acceleration of multi-camera data pre-classification based on the time balance strategy; the calculation of the Jacobian matrix includes a reprojection calculation module unit and an element calculation module unit; according to the position of the partial derivative element in the Jacobian matrix, the calculated Jacobian matrix data is distributedly stored; using the common view information of the observation point and the observation sequence number, the calculation of the Hessian matrix is ​​accelerated by multi-parallel block calculation; using the sparse distribution of the matrix blocks in the Hessian matrix, the calculated Hessian matrix data is stored in blocks. The invention optimizes the digital image processing calculation speed and data reading and writing speed.

Description

Technical Field

[0001] The present invention relates to the field of digital image processing, and more specifically, to an acceleration method and system for digital image processing of a bundle adjustment algorithm. Background Art

[0002] The bundle adjustment algorithm is a part of the final optimization of the 3D point position and camera parameters in 3D reconstruction, and is an important technology for obtaining good 3D reconstruction results. The bundle adjustment algorithm inputs the preliminarily determined camera pose and 3D point coordinates into the algorithm for solution, and then obtains the best camera pose and 3D point coordinates through optimization methods. And because it can obtain accurate camera poses and coordinates of 3D points in space, the 3D reconstruction technology with the bundle adjustment algorithm as the backend processing is gradually applied to various fields of life.

[0003] The intensive mathematical operations in the bundle adjustment algorithm greatly slow down the execution speed of the algorithm, especially on the x86 platform and ARM platform, where the current optimization methods mainly accelerate the algorithm by calculating multiple mathematical formulas in parallel or increasing the system operation frequency. However, the acceleration effect based on these methods has great limitations. Summary of the invention

[0004] The purpose of the present invention is to overcome the shortcomings of the prior art and provide an acceleration method and system for digital image processing of a bundle adjustment algorithm, which combines the hardware structure and optimizes the calculation speed and data reading and writing speed of the digital image processing of the bundle adjustment algorithm.

[0005] The object of the present invention is achieved through the following solutions:

[0006] An acceleration method for digital image processing of a bundle adjustment algorithm comprises the following steps:

[0007] Prepare data sources;

[0008] Jacobian matrix calculation unit acceleration: the data source data is transmitted to the Jacobian matrix calculation unit of the bundle adjustment algorithm digital image processing system through the input data cache, and multiple observation point calculation pipelines are set in the Jacobian matrix calculation unit to calculate the elements in the Jacobian matrix of the current observation point, and the observation point calculation pipeline includes: a process of reprojecting the three-dimensional point through camera parameters and three-dimensional point coordinates, and a process of calculating each element in the Jacobian matrix using the reprojected coordinates and camera parameters.

[0009] Further, the step of storing Jacobian matrix data using an optimized Jacobian matrix storage structure is included:

[0010] Jacobian matrix storage structure optimization: The Jacobian matrix element data calculated by the Jacobian matrix calculation unit will be grouped and spliced ​​through the output data cache unit. After the single calculation of all Jacobian matrix observation point calculation pipelines is completed, the results will be rearranged according to the RAM storage number, and then written into the subsequent RAM storage array in parallel through multiple RAM bus groups; the Jacobian matrix data will be stored in the Jacobian matrix data storage array after rearrangement.

[0011] Furthermore, the method comprises the steps of: storing the camera's visual information ISEEC of the three-dimensional point extracted from the data source and the index value information COBC of the three-dimensional point into the RAM.

[0012] Further, the method includes the steps of accelerating the Hessian matrix calculation unit of the digital image processing system of the bundle adjustment algorithm:

[0013] The matrix blocks E, matrix blocks B and matrix blocks C in the Hessian matrix H are respectively calculated by the Hessian matrix calculation unit and the group calculation method. After the data calculation of the Hessian matrix calculation unit is completed, the calculated data of the Hessian matrix H are grouped and classified and stored in multiple RAMs through the Hessian matrix storage unit.

[0014] Furthermore, in the step of quasi-data source, the camera parameter storage part stores the parameters of all cameras. Each camera external parameter has 12 parameters, which are the 9 elements r of the rotation matrix R. 1 , ..., r 9 , the three elements t of the translation vector t 1 , t 2 , t 3 Each camera internal parameter has three parameters, namely the conversion ratio parameter k of the pixel coordinate system and the camera distortion parameters s1, s2; the 3D point coordinate data storage part stores the coordinate data of all 3D points in this pipeline, and groups and stores the X, Y, and Z in the coordinates in three RAMs respectively; the observation information INDEXPC data storage part uses the observation point number as the address to store the camera label data and 3D point label data of the observation point.

[0015] Furthermore, in the Jacobian matrix calculation unit acceleration, the sub-steps are included:

[0016] The reprojection link calculation of the three-dimensional point coordinates to the camera coordinate system. During the reprojection calculation process, the coordinates P of the three-dimensional space point after reprojection to the camera coordinate system are set c , R represents the rotation matrix in the camera extrinsic parameters, t represents the translation vector in the camera extrinsic parameters, and Pw represents the coordinates of the real three-dimensional point in the world coordinate system, then calculate P cThe formula is expressed as:

[0017] P c =RP W +t

[0018] During the calculation of each partial derivative element in the Jacobian matrix, the partial derivative F of the objective function with respect to the camera external parameter is set to P c =(X c , Y c , Z c ) T Represents the coordinates of the three-dimensional point after reprojection, expressed as f x and f y Represents the scaling ratio of the coordinate axis, then the formula for calculating the partial derivative F of the objective function with respect to the camera extrinsic parameters is expressed as:

[0019]

[0020] In the process of calculating each partial derivative element in the Jacobian matrix, the partial derivative E of the target function with respect to the three-dimensional point is set, and the formula for calculating the partial derivative E of the target function with respect to the camera extrinsic parameter is expressed as:

[0021]

[0022] Further, the steps include:

[0023] Distributed storage of Jacobian matrix data is achieved by constructing distributed RAM; a storage area is constructed to store the camera's visual information ISEEC of three-dimensional points and the index value information COBC of three-dimensional points extracted from the data source, which is used to provide the required data for the subsequent calculation of the Hessian matrix.

[0024] Further, the method comprises the following sub-steps:

[0025] The JP block calculation unit is used to calculate the E matrix block in the Hessian matrix. After the calculation is completed, the data of the E matrix block is stored in the E matrix storage RAM of the subsequent stage through the RAM bus;

[0026] A plurality of binary tree calculation multiplication tree structures are used to parallelly calculate the B matrix block in the Hessian matrix, and after the calculation is completed, the data of the B matrix block is stored in the subsequent B matrix storage RAM through the RAM bus;

[0027] A plurality of JP block computing units are used to perform parallel computing of the C matrix block in the Hessian matrix, and after the computing is completed, the data of the C matrix block is stored in the C matrix storage RAM of the subsequent stage through the RAM bus;

[0028] The Jacobian matrix data requirements generated by the E matrix calculation unit, the B matrix calculation unit, and the C matrix calculation unit are decoded, and then the Jacobian matrix data is read from the previous Jacobian matrix storage array through the RAM bus.

[0029] Further, the method comprises the following sub-steps:

[0030] Three storage areas are used to store Hessian matrix data, wherein the three storage areas are B matrix data storage area, C matrix data storage area and E matrix data storage area.

[0031] An acceleration system for digital image processing of a bundle adjustment algorithm, comprising:

[0032] The input data cache unit includes the coordinate data storage area of ​​the three-dimensional point, the internal parameter matrix and external parameter matrix data storage area of ​​each camera, and the observation information INDEXPC data storage area. The observation data is divided into pipelines according to the principle of the same calculation time, so that the calculation time of each pipeline is the same, and the data source is prepared for the calculation of the Jacobian matrix;

[0033] A Jacobian matrix calculation pipeline unit, including an input data cache control unit, a Jacobian matrix calculation reprojection calculation unit and a partial derivative element calculation unit, for calculating each element data of the Jacobian matrix;

[0034] An output data cache unit, comprising a pipeline data output cache control unit and a data grouping and rearranging unit, for grouping and rearranging the Jacobian matrix data after calculation;

[0035] The Jacobian matrix data and observation information storage unit includes a Jacobian matrix data distributed storage area, a camera visual information ISEEC storage area for three-dimensional points, and a three-dimensional point index value information COBC storage area, which is used to store Jacobian matrix data and observation information data;

[0036] A Hessian matrix calculation unit, comprising a B matrix block calculation unit, a C matrix block calculation unit, an E matrix block calculation unit and an address decoding unit, for calculating Hessian matrix data;

[0037] The Hessian matrix storage unit includes a B matrix block storage area, a C matrix block storage area and an E matrix block storage area, and is used to store the calculated Hessian matrix data.

[0038] The beneficial effects of the present invention include:

[0039] In terms of the calculation speed of digital image processing in the bundle adjustment algorithm, it is mainly accelerated by accelerating the Jacobian matrix update stage and the Hessian matrix calculation stage, which are the most time-consuming stages in the bundle adjustment algorithm.

[0040] In terms of data reading and writing speed, the algorithm is accelerated mainly by optimizing the Jacobi matrix storage structure to achieve distributed storage of data and improve the data reading and writing speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0042] Figure 1 A flowchart of an acceleration method for digital image processing using a bundle adjustment algorithm provided by an embodiment of the present invention;

[0043] Figure 2 A schematic diagram of the structure of an acceleration system for digital image processing using a bundle adjustment algorithm provided by an embodiment of the present invention;

[0044] Figure 3 A schematic diagram of the design of an input data cache unit involved in a specific implementation manner provided by an embodiment of the present invention;

[0045] Figure 4 A schematic diagram of the design of a Jacobi calculation pipeline unit involved in a specific implementation manner provided in an embodiment of the present invention;

[0046] Figure 5 A schematic diagram of the design of a data allocation unit of an observation point calculation pipeline involved in a specific implementation manner provided by an embodiment of the present invention;

[0047] Figure 6 A schematic diagram of a design of a reprojection calculation unit involved in a specific implementation manner provided by an embodiment of the present invention;

[0048] Figure 7 A schematic diagram of a design of a JC matrix element calculation unit involved in a specific implementation manner provided by an embodiment of the present invention;

[0049] Figure 8 A schematic diagram of the design of a JP matrix element calculation unit involved in a specific implementation manner provided in an embodiment of the present invention;

[0050] Fig. 9 A schematic diagram of the design of an output data cache unit involved in a specific implementation manner provided by an embodiment of the present invention;

[0051] Fig.10A schematic diagram of the design of J and an observation information storage unit involved in a specific implementation manner provided in an embodiment of the present invention;

[0052] Fig.11 A schematic diagram of the design of an ISEEC and COBC observation information storage unit involved in a specific implementation manner provided in an embodiment of the present invention;

[0053] Fig.12 A schematic diagram of a design of a Jacobian matrix storage unit involved in a specific implementation manner provided in an embodiment of the present invention;

[0054] Fig.13 A schematic diagram of the design of a Hessian matrix calculation unit involved in a specific implementation manner provided by an embodiment of the present invention;

[0055] Fig.14 A schematic diagram of the design of a JP block calculation unit involved in a specific implementation manner provided by an embodiment of the present invention;

[0056] Fig.15 A schematic diagram of a design of a binary tree multiplier unit involved in a specific implementation manner provided by an embodiment of the present invention;

[0057] Fig.16 A schematic diagram of the design of a B matrix calculation unit involved in a specific implementation manner provided by an embodiment of the present invention;

[0058] Fig.17 A schematic diagram of a design of a C matrix calculation unit involved in a specific implementation manner provided by an embodiment of the present invention;

[0059] Fig.18 A schematic diagram of the design of a Hessian matrix address decoding unit involved in a specific implementation manner provided in an embodiment of the present invention;

[0060] Fig.19 A schematic diagram of a design of a Hessian matrix storage unit involved in a specific implementation manner provided in an embodiment of the present invention;

[0061] Fig. 20 A calculation residual result of an acceleration system implemented in a specific implementation manner provided in an embodiment of the present invention;

[0062] Fig.21 The normalized statistical result of the calculation error of the acceleration system implemented in a specific implementation manner provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0063] The invention is further described below in conjunction with the accompanying drawings and embodiments. All features disclosed in all embodiments in this specification, or steps in all methods or processes disclosed implicitly, except for mutually exclusive features and / or steps, can be combined and / or expanded or replaced in any manner.

[0064] In the process of seeking to solve the problem in the background, the present invention fully investigates the principle of the bundle adjustment algorithm, analyzes the amount of calculation therein in detail, and proposes an acceleration method and system based on digital circuits (hardware structure characteristics), respectively optimizing the digital image processing calculation speed and data reading and writing speed of the bundle adjustment algorithm. In the embodiment, the steps include: storing the input observation data and camera parameters; using the observation information of the camera number corresponding to the three-dimensional point in the data, the observation data to be calculated is indiscriminately distributed to each calculation pipeline, so as to realize the parallel calculation acceleration of the multi-camera data classified in advance based on the time balance strategy; the calculation of the Jacobian matrix includes two parts: the reprojection calculation module unit and the element calculation module unit; according to the position of the partial derivative element in the Jacobian matrix, the Jacobian matrix data calculated is distributedly stored; using the common view information and the observation sequence number of the observation point, the calculation of the Hessian matrix is ​​accelerated by multi-parallel block calculation; using the sparse distribution of the matrix blocks in the Hessian matrix, the calculated Hessian matrix data is stored in blocks.

[0065] In terms of the calculation speed of digital image processing of the bundle adjustment algorithm, the most time-consuming Jacobian matrix update phase and Hessian matrix calculation phase in the digital image processing of the bundle adjustment algorithm are accelerated. In terms of data reading and writing speed, the Jacobian matrix storage structure is optimized to achieve distributed storage of data and improve the data reading and writing speed, thereby accelerating the digital image processing of the bundle adjustment algorithm.

[0066] As a specific embodiment of the present invention, Figure 1 As shown, the acceleration method for digital image processing of the bundle adjustment algorithm provided by the present invention comprises the following steps:

[0067] Step 1. First, the coordinate data of the three-dimensional points, the intrinsic and extrinsic matrix data of each camera, and the observation information INDEXPC data are input into the input data cache unit through the RAM bus. The observation data is pipelined according to the principle of the same calculation time, so that the calculation time of each pipeline is the same, and the data source is prepared for the subsequent calculation of the Jacobian matrix.

[0068] Step 2: The data is transmitted to the Jacobian matrix calculation unit through the input data buffer unit. The calculation unit starts to calculate the values ​​of the elements in the Jacobian matrix, using the coordinate data of the three-dimensional points given by the previous stage, as well as the intrinsic and extrinsic matrix data of each camera, to calculate each element in the Jacobian matrix.

[0069] Step 3: The calculated Jacobian matrix element data will be grouped and spliced ​​through the output data cache unit. After all the Jacobian matrix calculation pipelines are completed in a single calculation, the results will be rearranged according to the RAM storage number, and then written into the subsequent RAM storage array in parallel through multiple RAM bus groups.

[0070] Step 4: After rearrangement, the Jacobian matrix data will be stored in the Jacobian matrix data storage array, and the camera's visual information of the three-dimensional point ISEEC and the index value information COBC of the three-dimensional point extracted from the data source will be stored in the RAM.

[0071] Step 5: After the Jacobian matrix data is stored, the Hessian matrix calculation unit will use the group calculation method to calculate the matrix block B, matrix block C and matrix block E in the Hessian matrix H respectively.

[0072] Step 6: After the calculation of the Hessian matrix data is completed, the calculated Hessian matrix H data is grouped and classified and stored in multiple RAMs through the Hessian matrix storage unit.

[0073] As another specific embodiment of the present invention, Figure 2 As shown, the acceleration system for digital image processing of the bundle adjustment algorithm provided by the present invention comprises:

[0074] The input data cache unit contains the coordinate data storage area of ​​the three-dimensional point, the data storage area of ​​the intrinsic parameter matrix and extrinsic parameter matrix of each camera, and the observation information INDEXPC data storage area. The observation data is pipelined according to the principle of the same calculation time, so that the calculation time of each pipeline is the same, and the data source is prepared for the subsequent calculation of the Jacobian matrix.

[0075] The Jacobian matrix calculation pipeline unit includes an input data cache control unit, a Jacobian matrix calculation reprojection calculation unit, and a partial derivative element calculation unit. It is mainly responsible for calculating each element data of the Jacobian matrix.

[0076] The output data cache unit includes a pipeline data output cache control unit and a data grouping and rearrangement unit, which is responsible for grouping and rearranging the Jacobian matrix data after calculation.

[0077] The Jacobian matrix data and observation information storage unit includes a Jacobian matrix data distributed storage area, a camera visual information ISEEC storage area for three-dimensional points, and a three-dimensional point index value information COBC storage area, and is mainly responsible for storing Jacobian matrix data and observation information data.

[0078] The Hessian matrix calculation unit includes a B matrix block calculation unit, a C matrix block calculation unit, an E matrix block calculation unit and an address decoding unit, and is mainly responsible for calculating the Hessian matrix data.

[0079] The Hessian matrix storage unit includes a B matrix block storage area, a C matrix block storage area, and an E matrix block storage area, and is mainly responsible for storing the calculated Hessian matrix data.

[0080] The technical solution of the embodiment of the present invention is further described below in conjunction with the accompanying drawings.

[0081] like Figure 3 As shown in the figure, before the Jacobian matrix update calculation begins, the data required for the Jacobian matrix update calculation must first be input. The input data cache unit is responsible for receiving the external input 3D point coordinate data, camera parameter data, and observation information INDEXPC data through the RAM bus, and transmitting these data to the next level through the RAM bus when the latter level needs them.

[0082] like Figure 4 As shown in the figure, after the data source is sorted by the data input cache unit, it will be input into the Jacobian matrix J calculation pipeline unit to start calculating each element in the Jacobian matrix. The Jacobian matrix calculation pipeline unit is designed based on the parallel pipeline method. In order to achieve the highest acceleration ratio, multiple observation point calculation pipelines are set in the Jacobian matrix calculation pipeline unit. The function of the pipeline is to calculate each element in the Jacobian matrix of the current observation point.

[0083] Each observation point calculation pipeline is mainly divided into two units. One is a calculation unit for reprojecting three-dimensional points using camera parameters and three-dimensional point coordinates. The other is a calculation unit for each element in the Jacobian matrix using the reprojected coordinates and camera parameters.

[0084] In addition, this unit also has a data cache control unit, which is mainly responsible for the management of cache data and the generation of enable signals for the subsequent calculation part.

[0085] like Figure 5 As shown, the data cache control part in the Jacobian matrix calculation pipeline unit mainly implements the function of grouping the data input from the previous stage and transmitting the data groups to multiple observation point calculation pipelines when the Jacobian matrix calculation starts.

[0086] like Figure 6 As shown in the figure, the function of the reprojection calculation unit in the Jacobian matrix calculation pipeline unit is to reproject the coordinate data of the three-dimensional point from the world coordinate system to the camera coordinate system using the rotation matrix R and translation vector t in the current camera external parameters. The reprojection calculation process requires calculating the coordinates P of the three-dimensional space point after reprojection to the camera coordinate system. c , with P w Represents the coordinates of the real three-dimensional point in the world coordinate system, then calculate P c The formula is expressed as:

[0087] P c =RP W +t

[0088] like Figure 7 As shown in the figure, the design of the element calculation unit in the Jacobian matrix J calculation pipeline unit is divided into two parts. The first part is the calculation of the partial derivative F of the objective function with respect to the camera extrinsic parameters, that is, the JC calculation unit. The calculation unit has 10 parallel pipelines, and each pipeline calculates one element in the matrix JC separately using the reprojected coordinates and camera parameters transmitted by the previous stage. c =(X c , Y c , Z c )T 表 The coordinates of the three-dimensional point after reprojection are shown in f x and f y Represents the scaling ratio of the coordinate axis, then the formula for calculating the partial derivative F of the objective function with respect to the camera extrinsic parameters is expressed as:

[0089]

[0090] like Figure 8 As shown, another part of the element calculation unit in the Jacobian matrix J calculation pipeline unit is the calculation of the partial derivative E of the objective function to the three-dimensional point, that is, the matrix JP calculation unit. The formula for calculating the partial derivative E of the objective function to the camera extrinsic parameter is expressed as:

[0091]

[0092] like Fig. 9As shown in the figure, after the data calculation of the Jacobian matrix is ​​completed, due to the limitation of the number of data storage buses, the data needs to be grouped and rearranged before it can be stored in the Jacobian matrix storage array. The output data cache unit caches and organizes the output data of the previous Jacobian matrix calculation pipeline unit, rearranges it according to the position of the Jacobian matrix data elements, and outputs it to the next level Jacobian storage array through the RAM bus for storage. The output data cache unit is mainly divided into two units, one is the Jacobian matrix calculation pipeline data cache unit, and the other is the data rearrangement unit.

[0093] The function of the Jacobian matrix calculation pipeline data cache unit in the output data cache unit is to store and manage the Jacobian matrix data after the calculation of the previous observation point calculation pipeline unit. In the Jacobian matrix calculation pipeline unit, the calculation of the Jacobian matrix data is performed by dividing into multiple observation point calculation pipelines, and the output data is also performed according to the number of the observation point calculation pipeline. The output data is relatively complicated, and the Jacobian matrix data output bus of the latter stage is grouped according to the position of the elements in the Jacobian matrix. The two cannot directly correspond. The data output after the observation point calculation pipeline is calculated needs to be sorted and rearranged before it can be transmitted to the next level for storage. The Jacobian matrix calculation pipeline data cache unit first performs the first-level classification of the observation point calculation pipeline data according to the number of the Jacobian calculation pipeline and the position of the element in the Jacobian matrix, eliminating the influence of the observation point calculation pipeline number on the grouping, with the purpose of sorting the data calculated by the scattered observation point calculation pipeline into a data format in the form of data groups, and then storing and managing these data through registers.

[0094] The data rearrangement unit in the output data cache unit performs secondary grouping on the data output by the Jacobian calculation pipeline data cache unit, and organizes it into a grouping form with the position of the element in the Jacobian matrix as the index value. This data management format facilitates writing data to the Jacobian matrix storage array later. After passing through the Jacobian matrix calculation data cache unit, the data is organized into a grouping form based on the number of the Jacobian calculation pipeline and the position of the element in the Jacobian matrix. At this time, the data still cannot be directly written into the Jacobian matrix storage unit of the subsequent stage. It is still necessary to remove the influence of the Jacobian matrix calculation pipeline number on the data grouping, and change the data into a grouping form with only the position of the element in the Jacobian matrix as the index value, and then write the calculated data into the Jacobian matrix storage array.

[0095] If it is assumed that there are K Jacobian matrix calculation pipelines in total, and there are S observation point calculation pipelines in each Jacobian matrix calculation pipeline, then after a single Jacobian matrix calculation is completed, 16KS 64-bit Jacobian matrix element data will be generated. After the first data grouping and sorting by the Jacobian matrix calculation pipeline data cache unit in the output cache unit, the data is changed from 16KS 64-bit Jacobian matrix element data to 16K groups of Jacobian matrix element data, where each group of data has S Jacobian matrix element data, and then it is further grouped by the data rearrangement unit using the position of the element in the Jacobian matrix as the index value. At this time, the data is changed from 16K groups of Jacobian matrix element data to 16 groups of Jacobian matrix element data, where each group of data has KS Jacobian matrix element data. At this time, the requirement of transmitting Jacobian matrix data to 16 groups of RAM buses of the next stage has been met, and the data calculated by the previous stage can be transmitted to the Jacobian matrix storage array of the next stage through the RAM bus for data storage.

[0096] like Fig.10 As shown in FIG. 1 , after the Jacobian matrix data is calculated, the Jacobian matrix data and observation information storage unit constructs a distributed storage array of the Jacobian matrix data in the manner of a distributed RAM, and constructs a storage area to store the camera's visual information of the three-dimensional point ISEEC and the index value information of the three-dimensional point COBC extracted from the data source, and is responsible for providing the required data when calculating the Hessian matrix at the subsequent stage. The storage method of the camera's visual information of the three-dimensional point ISEEC and the index value information of the three-dimensional point COBC is as follows: Fig.11 The Jacobian matrix data is stored as shown in Fig.12 shown.

[0097] like Fig.13 As shown, after the Jacobian matrix data is stored, the Jacobian matrix data can be read by the Hessian matrix calculation unit to calculate the Hessian matrix data. After the calculation is completed, the Hessian matrix data is stored in the next-level Hessian matrix storage unit through the RAM bus.

[0098] The Hessian matrix calculation unit consists of four parts. The first part is the E matrix calculation part in the Hessian matrix calculation. Its function is to use the data of the Jacobian matrix to calculate the E matrix block in the Hessian matrix, and after the calculation is completed, the data of the E matrix block is stored in the E matrix storage RAM of the subsequent stage through the RAM bus.

[0099] The second part is the B matrix calculation part in the Hessian matrix calculation. Its function is to use the data of the Jacobian matrix to calculate the B matrix block in the Hessian matrix. The B matrix calculation part is realized by using multiple binary tree calculation multiplication tree structures for parallel calculation. After the calculation is completed, the data of the B matrix block is stored in the B matrix storage RAM of the subsequent stage through the RAM bus.

[0100] The third part is the C matrix calculation part in the Hessian matrix calculation. Its function is to use the data of the Jacobian matrix to calculate the C matrix block in the Hessian matrix. The C matrix calculation part is realized by using multiple JP block calculation units for parallel calculation. After the calculation is completed, the data of the C matrix block is stored in the C matrix storage RAM of the subsequent stage through the RAM bus.

[0101] The fourth part is the address decoding unit, which is responsible for decoding the Jacobian matrix data requirements generated by the E matrix calculation unit, the B matrix calculation unit, and the C matrix calculation unit, and then reading the Jacobian matrix data from the previous Jacobian matrix storage array through the RAM bus. When calculating the C matrix in the Hessian matrix, it is necessary to determine the location of the required JP block based on the previous ISEEC and COBC observation information, so it is necessary to read the previous Jacobian matrix data and the ISEEC and COBC observation information in the observation information storage unit for judgment.

[0102] like Fig.14 As shown, the E matrix calculation unit in the Hessian matrix calculation unit is responsible for implementing a simple 2-dimensional vector multiplication operation. The mathematical description of the multiplication operation of the JP block in the Hessian matrix calculation unit is also a simple 2-dimensional vector multiplication operation, so in the present invention, the E matrix calculation unit in the Hessian matrix calculation unit and the multiplication operation unit of the JP block adopt the same structure.

[0103] like Fig.16 As shown, the B matrix calculation unit in the Hessian matrix calculation unit is responsible for using the JC matrix block in the Jacobian matrix to realize high-dimensional vector multiplication operations. The dimension of the vector in the multiplication realized is determined by the number of observations in the corresponding camera. The present invention adopts the structure of the binary tree calculation multiplication tree to perform

[0104] High-dimensional vector multiplication operation. In the structure of this binary tree multiplier, there are 8 input parameters, which can realize vector multiplication operation with a dimension of 4. The structure of the binary tree calculation multiplication tree is as follows Fig.15 shown.

[0105] If the vector multiplication operation in the B matrix calculation is implemented with only one binary tree multiplier structure, the speed of the algorithm will be greatly slowed down. Therefore, four binary tree multipliers are used in the present invention to perform simultaneous parallel operations to accelerate the algorithm to a greater extent. Because the single binary tree multiplier structure designed in the present invention can realize vector multiplication operations with a dimension of 4, the B matrix calculation unit designed in this article can support vector multiplication operations with a maximum dimension of 16.

[0106] like Fig.17 As shown, the C matrix calculation unit in the Hessian matrix calculation unit is responsible for realizing vector multiplication using the JP matrix block in the Jacobian matrix. In the C matrix calculation unit designed by the present invention, the calculation of the C matrix block with a maximum common view number of 16 can be satisfied, that is, the vector multiplication operation with a maximum dimension of 32 can be satisfied.

[0107] like Fig.18 As shown, when calculating the elements of the Hessian matrix, the address decoding unit is responsible for responding to the needs of the Jacobian matrix data generated in the calculation unit of the E matrix, the calculation unit of the B matrix, and the calculation unit of the C matrix, and reading these data from the corresponding addresses of the Jacobian matrix storage array of the previous stage, and then allocating them to the calculation units for calculating the Hessian matrix.

[0108] The address decoding unit is mainly designed in three parts. The first part is the addressing of Jacobian matrix data when calculating the B matrix. When calculating the B matrix, the dimension of the vector involved is huge. When calculating the B matrix once, generally, all Jacobian matrix data cannot be simultaneously imported into the B matrix calculation unit for calculation. In the B matrix calculation unit designed in this paper, at most vector multiplication operations with a dimension of 16 can be performed simultaneously. If the number of observation points in the camera corresponding to the B matrix calculation this time is less than 16, that is, the vector dimension in the B matrix calculated this time is just less than 16, then the address of all Jacobian matrix data required for calculating the B matrix this time can be obtained through one operation of the address decoding unit. If the number of observation points in the camera corresponding to the B matrix calculation this time is greater than 16, it is necessary to count the vector multiplication in the B matrix calculation unit, and multiply the count by the ratio factor 16 and add the serial index value of the B matrix to correctly address the Jacobian matrix data required by the B matrix calculation unit this time.

[0109] The second part of the address decoding unit is the addressing of the Jacobian matrix data when calculating the E matrix. The E matrix calculation unit is relatively simple. The calculated vector multiplication is always a 2-dimensional vector multiplication operation, and the number of calculations when calculating the E matrix corresponds to the observation point index value one by one. The data in the previous Jacobian matrix storage array is also stored according to the index value of the observation point as the address. Therefore, when reading the Jacobian matrix data used for the E matrix calculation, the serial number of the calculated E matrix can be directly used as the address index value.

[0110] The third part of the address decoding unit is the addressing of Jacobian matrix data when calculating the C matrix. Because the dimension of the vector in the vector multiplication in the C matrix calculation unit depends on the number of co-views of the three-dimensional point in each camera, the auxiliary operation of ISEEC observation information and COBC observation information is required when addressing the Jacobian matrix data required for calculating the C matrix. When addressing the Jacobian matrix data required for calculating the C matrix, it is first necessary to read the ISEEC information of the current three-dimensional point to determine the number of co-views, that is, to determine the number of JP matrix blocks required for calculating the C matrix, and then determine the position of these JP matrix blocks in the Jacobian matrix storage array through the COBC information, that is, the address of the JP matrix data required for calculating the C matrix can be obtained.

[0111] like Fig.19 As shown, after the calculation of the Hessian matrix elements is completed, the calculation results are stored through the Hessian matrix storage unit, which is mainly divided into three storage areas for storing the Hessian matrix data, namely the B matrix data storage area, the C matrix data storage area, and the E matrix data storage area.

[0112] The number of B matrices calculated by the Hessian matrix is ​​the same as the number of cameras, so the B matrix data storage area can be grouped according to the camera number. Each time the B matrix calculation unit completes the calculation, it will output 6x6 B matrix element data. However, because the data of the 1st row and 2nd column of the B matrix and the data of the 2nd row and 1st column are fixed to 0, and the B matrix is ​​symmetrical, only 20 non-0 element data of the B matrix are saved here. Assuming there are M cameras at this time, the storage unit of the B matrix will be divided into M storage areas, each with 20 B matrix element data.

[0113] The number of C matrices calculated by the Hessian matrix is ​​the same as the number of three-dimensional points, so the C matrix data storage area can be grouped and stored according to the three-dimensional point number. Each time the C matrix calculation unit completes a calculation, it will output 3×3 C matrix element data. However, because the matrix C is symmetrical, it is concluded that there are 6 C matrix element data in each area. Assuming that there are N three-dimensional points at this time, the storage unit of the C matrix will be divided into N storage areas, and each area has 6 C matrix element data.

[0114] The number of E matrices finally calculated by the Hessian matrix is ​​the same as the number of observation points generated in this observation. Therefore, the E matrix data storage area can be grouped and stored according to the observation point number. Each time the E matrix calculation unit completes a calculation, it will output 6×3 E matrix element data, so it is concluded that there are 18 E matrix element data in each area. Assuming that this observation generates Q observation points, the storage unit of the E matrix will be divided into Q storage areas, each with 18 E matrix element data.

[0115] The technical effects of the embodiments of the present invention are described in detail below in conjunction with experiments.

[0116] The embodiment of the present invention implements the acceleration system on the FPGA platform in principle, and the Jacobian matrix update phase requires 1.472ms-3.871ms, an average of 2.677ms, and the Hessian matrix calculation phase requires 4.659ms-9.351ms, an average of 6.867ms. The calculation speed of the Jacobian matrix update phase is 13 times and 390 times higher than that of the Inteli5-8400 platform and the ARMCortex-A9 platform, respectively. The calculation of the Hessian matrix is ​​2.6 times higher than that of the Inteli5-9400 platform.

[0117] The residual error of the simulation result of the acceleration system implemented in the embodiment of the present invention is as follows: Fig. 20 As shown, the error statistics are Fig.21 As shown in Figure 1, it can be seen that the error of 92% of the calculation results is stable within 10 -5 The following satisfies the computational requirements for bundle adjustment.

[0118] Example 1

[0119] An acceleration method for digital image processing of a bundle adjustment algorithm comprises the following steps:

[0120] Prepare data sources;

[0121] Jacobian matrix calculation unit acceleration: the data source data is transmitted to the Jacobian matrix calculation unit of the bundle adjustment algorithm digital image processing system through the input data cache, and multiple observation point calculation pipelines are set in the Jacobian matrix calculation unit to calculate the elements in the Jacobian matrix of the current observation point, and the observation point calculation pipeline includes: a process of reprojecting the three-dimensional point through camera parameters and three-dimensional point coordinates, and a process of calculating each element in the Jacobian matrix using the reprojected coordinates and camera parameters.

[0122] Example 2

[0123] On the basis of Example 1, the method includes the following steps of storing Jacobian matrix data using an optimized Jacobian matrix storage structure:

[0124] Jacobian matrix storage structure optimization: The Jacobian matrix element data calculated by the Jacobian matrix calculation unit will be grouped and spliced ​​through the output data cache unit. After the single calculation of all Jacobian matrix observation point calculation pipelines is completed, the results will be rearranged according to the RAM storage number, and then written into the subsequent RAM storage array in parallel through multiple RAM bus groups; the Jacobian matrix data will be stored in the Jacobian matrix data storage array after rearrangement.

[0125] Example 3

[0126] Based on the second embodiment, the method includes the following steps: storing the camera's visual information ISEEC of the three-dimensional points extracted from the data source and the index value information COBC of the three-dimensional points into RAM.

[0127] Example 4

[0128] On the basis of Example 2, the step of accelerating the Hessian matrix calculation unit of the bundle adjustment algorithm digital image processing system is included:

[0129] The matrix blocks E, matrix blocks B and matrix blocks C in the Hessian matrix H are respectively calculated by the Hessian matrix calculation unit and the group calculation method. After the data calculation of the Hessian matrix calculation unit is completed, the calculated data of the Hessian matrix H are grouped and classified and stored in multiple RAMs through the Hessian matrix storage unit.

[0130] Example 5

[0131] On the basis of Example 1, in the step of quasi-data source, the camera parameter storage part stores the parameters of all cameras, and each camera external parameter has 12 parameters, which are 9 elements r of the rotation matrix R 1 , ..., r 9 , the three elements t of the translation vector t 1 , t 2 , t 3 Each camera internal parameter has three parameters, namely the conversion ratio parameter k of the pixel coordinate system and the camera distortion parameters s1, s2; the 3D point coordinate data storage part stores the coordinate data of all 3D points in this pipeline, and groups and stores the X, Y, and Z in the coordinates in three RAMs respectively; the observation information INDEXPC data storage part uses the observation point number as the address to store the camera label data and 3D point label data of the observation point.

[0132] Example 6

[0133] On the basis of Example 1, the Jacobian matrix calculation unit acceleration includes the following sub-steps:

[0134] The reprojection link calculation of the three-dimensional point coordinates to the camera coordinate system. During the reprojection calculation process, the coordinates P of the three-dimensional space point after reprojection to the camera coordinate system are set c , R represents the rotation matrix in the camera extrinsic parameters, t represents the translation vector in the camera extrinsic parameters, and P w Represents the coordinates of the real three-dimensional point in the world coordinate system, then calculate P c The formula is expressed as:

[0135] P c =RP W +t

[0136] During the calculation of each partial derivative element in the Jacobian matrix, the partial derivative F of the objective function with respect to the camera external parameter is set to P c =(X c , Y c , Z c ) T Represents the coordinates of the three-dimensional point after reprojection, expressed as f x and f y Represents the scaling ratio of the coordinate axis, then the formula for calculating the partial derivative F of the objective function with respect to the camera extrinsic parameters is expressed as:

[0137]

[0138] In the process of calculating each partial derivative element in the Jacobian matrix, the partial derivative E of the target function with respect to the three-dimensional point is set, and the formula for calculating the partial derivative E of the target function with respect to the camera extrinsic parameter is expressed as:

[0139]

[0140] Example 7

[0141] On the basis of Example 2, the method comprises the following steps:

[0142] Distributed storage of Jacobian matrix data is achieved by constructing distributed RAM; a storage area is constructed to store the camera's visual information ISEEC of three-dimensional points and the index value information COBC of three-dimensional points extracted from the data source, which is used to provide the required data for the subsequent calculation of the Hessian matrix.

[0143] Example 8

[0144] On the basis of Example 4, the method comprises the following sub-steps:

[0145] The JP block calculation unit is used to calculate the E matrix block in the Hessian matrix. After the calculation is completed, the data of the E matrix block is stored in the E matrix storage RAM of the subsequent stage through the RAM bus;

[0146] A plurality of binary tree calculation multiplication tree structures are used to parallelly calculate the B matrix block in the Hessian matrix, and after the calculation is completed, the data of the B matrix block is stored in the subsequent B matrix storage RAM through the RAM bus;

[0147] A plurality of JP block computing units are used to perform parallel computing of the C matrix block in the Hessian matrix, and after the computing is completed, the data of the C matrix block is stored in the C matrix storage RAM of the subsequent stage through the RAM bus;

[0148] The Jacobian matrix data requirements generated by the E matrix calculation unit, the B matrix calculation unit, and the C matrix calculation unit are decoded, and then the Jacobian matrix data is read from the previous Jacobian matrix storage array through the RAM bus.

[0149] Example 9

[0150] On the basis of Example 4, the method comprises the following sub-steps:

[0151] Three storage areas are used to store Hessian matrix data, wherein the three storage areas are B matrix data storage area, C matrix data storage area and E matrix data storage area.

[0152] Example 10

[0153] An acceleration system for digital image processing of a bundle adjustment algorithm, comprising:

[0154] The input data cache unit includes the coordinate data storage area of ​​the three-dimensional point, the internal parameter matrix and external parameter matrix data storage area of ​​each camera, and the observation information INDEXPC data storage area. The observation data is divided into pipelines according to the principle of the same calculation time, so that the calculation time of each pipeline is the same, and the data source is prepared for the calculation of the Jacobian matrix;

[0155] A Jacobian matrix calculation pipeline unit, including an input data cache control unit, a Jacobian matrix calculation reprojection calculation unit and a partial derivative element calculation unit, for calculating each element data of the Jacobian matrix;

[0156] An output data cache unit, comprising a pipeline data output cache control unit and a data grouping and rearranging unit, for grouping and rearranging the Jacobian matrix data after calculation;

[0157] The Jacobian matrix data and observation information storage unit includes a Jacobian matrix data distributed storage area, a camera visual information ISEEC storage area for three-dimensional points, and a three-dimensional point index value information COBC storage area, which is used to store Jacobian matrix data and observation information data;

[0158] A Hessian matrix calculation unit, comprising a B matrix block calculation unit, a C matrix block calculation unit, an E matrix block calculation unit and an address decoding unit, for calculating Hessian matrix data;

[0159] The Hessian matrix storage unit includes a B matrix block storage area, a C matrix block storage area and an E matrix block storage area, and is used to store the calculated Hessian matrix data.

[0160] The units involved in the embodiments of the present invention may be implemented by software or hardware, and the units described may also be arranged in a processor. The names of these units do not, in some cases, limit the units themselves.

[0161] According to one aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the methods provided in the above-mentioned various optional implementations.

[0162] As another aspect, the present application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiment; or may exist independently without being assembled into the electronic device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by an electronic device, the electronic device implements the method described in the above embodiment.

[0163] The parts not involved in the present invention are the same as the prior art or can be implemented by using the prior art.

[0164] The above technical solution is only one implementation mode of the present invention. For those skilled in the art, it is easy to make various types of improvements or modifications based on the application methods and principles disclosed in the present invention, and it is not limited to the method described in the above specific implementation mode of the present invention. Therefore, the method described above is only preferred and does not have a restrictive meaning.

[0165] In addition to the above examples, those skilled in the art may obtain other embodiments based on the above disclosure or by using the knowledge or technology in the relevant field to make changes. The features of each embodiment may be interchangeable or replaced. The changes and modifications made by those skilled in the art do not depart from the spirit and scope of the present invention and should be within the scope of protection of the claims attached to the present invention.

Claims

1. An acceleration method for digital image processing of bundle adjustment algorithm, characterized in that, it includes the steps: Prepare data sources; Acceleration of Jacobian matrix calculation unit: Transmit the data of the data source to the Jacobian matrix calculation unit of the digital image processing system of the bundle adjustment algorithm through the input data cache. Set multiple observation point calculation pipelines in the Jacobian matrix calculation unit to calculate the elements in the Jacobian matrix of the current observation point. And the observation point calculation pipeline includes: a process of re-projecting the three-dimensional point through camera parameters and three-dimensional point coordinates, and a process of calculating each element in the Jacobian matrix using the re-projected coordinates and camera parameters; It also includes the step of storing the Jacobian matrix data using an optimized Jacobian matrix storage structure: Optimization of Jacobian matrix storage structure: The Jacobian matrix element data calculated by the Jacobian matrix calculation unit will be grouped and combined through the output data cache unit. After all the Jacobian matrix observation point calculation pipelines are calculated once, the results will be rearranged according to the labels stored in the RAM, and then written into the subsequent RAM storage array in parallel through multiple RAM buses; The Jacobian matrix data will be stored in the Jacobian matrix data storage array after rearrangement; It also includes the step of accelerating the Hessian matrix calculation unit of the digital image processing system of the bundle adjustment algorithm: Through the Hessian matrix calculation unit, use the method of grouped calculation to calculate the matrix block E, matrix block B, and matrix block C in the Hessian matrix H respectively. After the data calculation of the Hessian matrix calculation unit is completed, the calculated data of the Hessian matrix H will be grouped and classified and stored in multiple RAMs through the Hessian matrix storage unit; Use the JP block calculation unit to calculate the E matrix block in the Hessian matrix. After the calculation is completed, store the data of the E matrix block in the subsequent E matrix storage RAM through the RAM bus; Use multiple binary tree calculation multiplication tree structures to calculate the B matrix block in the Hessian matrix in parallel. After the calculation is completed, store the data of the B matrix block in the subsequent B matrix storage RAM through the RAM bus; Use multiple JP block calculation units to calculate the C matrix block in the Hessian matrix in parallel. After the calculation is completed, store the data of the C matrix block in the subsequent C matrix storage RAM through the RAM bus; Decode the Jacobian matrix data requirements generated by the E matrix calculation unit, B matrix calculation unit, and C matrix calculation unit, and then read the Jacobian matrix data from the previous Jacobian matrix storage array through the RAM bus; Use three storage areas to store the Hessian matrix data, where the three storage areas are the B matrix data storage area, the C matrix data storage area, and the E matrix data storage area; In the acceleration of the Jacobian matrix calculation unit, it includes sub-steps: The reprojection link calculation of the three-dimensional point coordinates to the camera coordinate system. During the reprojection calculation process, the coordinates P of the three-dimensional space point after reprojection to the camera coordinate system are set c , R represents the rotation matrix in the camera extrinsic parameters, t represents the translation vector in the camera extrinsic parameters, and P w Represents the coordinates of the real three-dimensional point in the world coordinate system, then calculate P c The formula is expressed as: P c =RP W +t During the calculation of each partial derivative element in the Jacobian matrix, the partial derivative F of the objective function with respect to the camera external parameter is set to P c =(X c ,Y c ,Z c ) T Represents the coordinates of the three-dimensional point after reprojection, expressed as f x and f y Represents the scaling ratio of the coordinate axis, then the formula for calculating the partial derivative F of the objective function with respect to the camera extrinsic parameters is expressed as: During the calculation process of each partial derivative element in the Jacobian matrix, set the partial derivative E of the calculation objective function with respect to the three-dimensional point. The formula for calculating the partial derivative E of the calculation objective function with respect to the external camera parameters is:

2. The acceleration method for digital image processing of bundle adjustment algorithm according to claim 1, characterized in that, Including the steps of storing the visible information ISEEC of the camera with respect to the three-dimensional points and the index value information COBC of the three-dimensional points extracted from the data source into the RAM.

3. The method for accelerating digital image processing of the bundle adjustment algorithm according to claim 1, characterized in that In the step of quasi-data source, the camera parameter storage part stores the parameters of all cameras. Each camera external parameter has 12 parameters, which are the 9 elements r of the rotation matrix r. 1 ,…,r 9 , the three elements t of the translation vector t 1 ,t 2 ,t 3 ,Each camera internal parameter has three parameters, namely, the conversion ratio parameter k of the pixel coordinate system and the camera distortion parameters s1, s2; the 3D point coordinate data storage part stores the coordinate data of all 3D points in this pipeline, and groups and stores the X, Y, and Z in the coordinates in three RAMs respectively; the observation information INDEXPC data storage part stores the camera label data and the three-dimensional point label data of the observation points using the observation point numbers as addresses.

4. The method for accelerating digital image processing of the bundle adjustment algorithm according to claim 1, characterized in that including the steps of: realizing distributed storage of the Jacobian matrix data by constructing a distributed RAM; constructing a storage area to store the visible information ISEEC of the camera with respect to the three-dimensional points and the index value information COBC of the three-dimensional points extracted from the data source, for providing the required data when calculating the Hessian matrix at the subsequent stage.

5. An acceleration system for digital image processing of the bundle adjustment algorithm, characterized in that it includes: an input data cache unit, which includes a coordinate data storage area of three-dimensional points, an internal parameter matrix and external parameter matrix data storage area of each camera, and an observation information INDEXPC data storage area, and divides the observation data into pipelines according to the principle of the same calculation time, so that the calculation time of each pipeline is the same, and also prepares the data source for the calculation of the Jacobian matrix; a Jacobian matrix calculation pipeline unit, which includes an input data cache control unit, a reprojection calculation unit for calculating the Jacobian matrix, and a partial derivative element calculation unit, for calculating each element data of the Jacobian matrix; an output data cache unit, which includes a pipeline data output cache control unit and a data grouping and rearrangement unit, for grouping and rearranging the calculated Jacobian matrix data; a Jacobian matrix data and observation information storage unit, which includes a distributed storage area of Jacobian matrix data, a visible information ISEEC storage area of the camera with respect to the three-dimensional points, and an index value information COBC storage area of the three-dimensional points, for storing the data of the Jacobian matrix and the observation information data; a Hessian matrix calculation unit, which includes a B matrix block calculation unit, a C matrix block calculation unit, an E matrix block calculation unit and an address decoding unit, for calculating the Hessian matrix data; a Hessian matrix storage unit, which includes a B matrix block storage area, a C matrix block storage area and an E matrix block storage area, for storing the calculated Hessian matrix data; and this system is used to execute the method for accelerating digital image processing of the bundle adjustment algorithm according to claim 1.

Images