Fusion type accelerator for sensing algorithm and execution method of fusion type accelerator

Abstract

The invention provides a fusion type accelerator for a perception algorithm and an execution method thereof, and relates to the technical field of computers, the fusion type accelerator comprises a top layer controller used for obtaining an instruction sent by a central processing unit; the on-chip storage unit is used for storing to-be-processed data of the fusion accelerator; the reconfigurable computing array is used for respectively acquiring an instruction of the top-layer controller and to-be-processed data of the on-chip storage unit so as to compute the to-be-processed data according to the instruction to obtain a multiply-accumulate result; the configurable post-processing unit is used for acquiring an instruction of the top layer controller so as to perform post-processing on a multiply-accumulate result obtained by the reconfigurable computing array according to the instruction; the on-chip interconnection unified system is used for being respectively connected with the top layer controller, the on-chip storage unit, the reconfigurable computing array and the configurable post-processing unit and providing a programmable data path; the programmable data path is used for executing different sensing algorithms through a unified execution stage, and the utilization rate of system computing resources can be increased.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a fusion accelerator for perception algorithms and its execution method. Background Technology

[0002] With the continuous development of capabilities such as autonomous navigation and environmental interaction, micro-robots are increasingly being used in fields such as warehousing and logistics, disaster relief, and industrial inspection. These applications place extremely high demands on the robots' real-time perception and decision-making capabilities, which rely heavily on the efficient processing of sensor information such as visual data.

[0003] Currently, in order to support the perception and computing tasks of microrobots, dedicated accelerators for different operators are generally integrated into a single system on a chip (SoC). However, when this discrete accelerator architecture performs perception and computing tasks for microrobots, the throughput between dedicated accelerator cores for different operators is difficult to match, and different operators are difficult to schedule flexibly, resulting in low utilization of system computing resources. Summary of the Invention

[0004] To address the problems existing in the prior art, this invention provides a fusion accelerator for perception algorithms and its execution method.

[0005] This invention provides a fusion accelerator for perception algorithms, comprising: The top-level controller is used to receive instructions sent by the central processing unit; On-chip storage unit for storing the data to be processed by the fused accelerator; A reconfigurable computing array is used to acquire instructions from the top-level controller and data to be processed from the on-chip storage unit, respectively, so as to calculate the data to be processed according to the instructions and obtain a multiply-accumulate result; A configurable post-processing unit is configured to acquire instructions from the top-level controller and perform post-processing on the multiply-accumulate results obtained from the reconfigurable computing array according to the instructions. An on-chip interconnect unified system is used to connect to the top-level controller, the on-chip storage unit, the reconfigurable computing array, and the configurable post-processing unit respectively, providing a programmable data path; the programmable data path is used to implement different perception algorithms through a unified execution phase.

[0006] According to the present invention, a fusion accelerator for a perception algorithm is provided, wherein the reconfigurable computing array comprises: At least two multipliers are used to calculate partial products of the data to be processed, and the partial products are transmitted to the selector. In the case of constructing a solver processing unit, the selector is used to receive the partial product transmitted by the corresponding multiplier and transmit the partial product to the adder according to the instruction. An adder is configured to receive a partial product transmitted by the selector of the first multiplier among the at least two multipliers, obtain a cached result from the register corresponding to the first multiplier, and perform an in-situ update based on the partial product of the first multiplier and the cached result; receive a partial product transmitted by the selector of the second multiplier among the at least two multipliers, obtain a cached result from the register corresponding to the second multiplier, and perform an in-situ update based on the partial product of the second multiplier and the cached result; the first multiplier and the second multiplier are adjacent.

[0007] According to the present invention, a fusion accelerator for a perception algorithm is provided, wherein the reconfigurable computing array comprises: At least two multipliers are used to calculate partial products of the data to be processed, and the partial products are transmitted to the selector. In the case of constructing a multiply-accumulate array, the selector is used to receive the partial product transmitted by the corresponding multiplier and transmit the partial product to the corresponding register according to the instruction. An adder is configured to obtain a first partial product from the register corresponding to the first multiplier of the at least two multipliers, and a second partial product from the register corresponding to the second multiplier of the at least two multipliers, to output a partial sum.

[0008] According to a fusion accelerator for a perception algorithm provided by the present invention, the initial cached result in the register corresponding to the multiplier is obtained by the configurable post-processing unit from the data to be processed in the on-chip storage unit.

[0009] According to the present invention, a fusion accelerator for a perception algorithm is provided, wherein the data broadcasting mechanism of the solver processing unit is a two-dimensional data broadcasting mechanism of rows and columns.

[0010] According to the present invention, a fusion accelerator for a perception algorithm is provided, wherein the multiplier includes a multiplication unit, a 4-bit addition unit, and an 8-bit addition unit; wherein the multiplication unit and the 4-bit addition unit cooperate to support floating-point 4-bit multiplication calculations, and the multiplication unit and the 8-bit addition unit cooperate to support floating-point 16-bit multiplication calculations.

[0011] According to the present invention, a fusion accelerator for a perception algorithm is provided, wherein the adder includes an adder unit, a comparator, and a local normalization unit; wherein the comparator controls the adder unit and the local normalization unit to support 4-bit accumulation calculation based on the partial product of floating-point 4-bit multiplication, and the comparator controls the adder unit and the local normalization unit to support 8-bit accumulation calculation based on the partial product of floating-point 8-bit multiplication.

[0012] According to the present invention, a fusion accelerator for a perception algorithm is provided, wherein the instructions include fusion instructions and basic instructions; wherein the fusion instructions are obtained by decomposing the perception algorithm; and the basic instructions are obtained by decomposing the fusion instructions.

[0013] The present invention also provides a method for executing a perception algorithm, comprising: based on a fusion accelerator for a perception algorithm as described in any of the preceding claims, including: The top-level controller receives instructions sent by the central processing unit and transmits these instructions to the reconfigurable computing array, the configurable post-processing unit, and the on-chip unified interconnect system, respectively. The data to be processed from the fused accelerator stored in the on-chip storage unit is moved to the reconfigurable computing array through the on-chip unified interconnect system, so as to control the reconfigurable computing array to calculate the data to be processed according to the instructions and obtain the multiply-accumulate result; The configurable post-processing unit is controlled to perform post-processing on the multiplication-accumulation result obtained from the reconfigurable computing array according to the instructions, so as to obtain the post-processing result; The post-processing result is written back to obtain the execution result of the perception algorithm.

[0014] The present invention also provides a computing device including a fusion accelerator for a perception algorithm as described in any of the preceding claims.

[0015] The fusion accelerator and its execution method for perception algorithms provided by this invention can enable a reconfigurable computing array to compute the data to be processed in different modes on the same hardware architecture by acquiring instructions sent by the central processing unit, and obtain the multiplication and accumulation results by computing the data to be processed in different modes. The multiplication and accumulation results are then processed by a configurable post-processing unit. This enables different perception algorithms to be executed through a unified execution stage, avoiding the integration problem of multiple dedicated accelerators and improving the utilization rate of system computing resources. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0017] Figure 1 This is one of the structural schematic diagrams of the fusion accelerator for perception algorithms provided by the present invention.

[0018] Figure 2 This is the second schematic diagram of the fusion accelerator for perception algorithms provided by the present invention.

[0019] Figure 3 This is the third schematic diagram of the fusion accelerator for perception algorithms provided by the present invention.

[0020] Figure 4 This is the fourth schematic diagram of the fusion accelerator for perception algorithms provided by the present invention.

[0021] Figure 5 This is a schematic diagram of a prototype chip system for a fusion accelerator for sensing algorithms provided by the present invention.

[0022] Figure 6 This is a flowchart illustrating the execution method of the perception algorithm provided by the present invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0024] Current accelerators generally only support acceleration of a single operator. However, in real-world scenarios, perception computing for micro-drones typically involves multiple operators, such as convolutional neural network computation tasks and Cholesky decomposition. These tasks introduce significant latency during computation, requiring accelerators to optimize them. This necessitates the integration of multiple dedicated accelerators into a discrete architecture, leading to difficulties in matching throughput between dedicated accelerator cores for different operators when performing perception computing tasks for micro-robots. This results in inflexible scheduling of different operators and low utilization of system computing resources.

[0025] Based on this, the technical solutions of the embodiments of the present invention will be described below with reference to the accompanying drawings.

[0026] Figure 1 This is one of the structural schematic diagrams of the fusion accelerator for perception algorithms provided by the present invention, such as... Figure 1 As shown, the accelerator includes: The top-level controller is used to receive instructions sent by the central processing unit; On-chip storage unit for storing the data to be processed by the fused accelerator; A reconfigurable computing array is used to acquire instructions from the top-level controller and data to be processed from the on-chip storage unit, respectively, so as to calculate the data to be processed according to the instructions and obtain a multiply-accumulate result; A configurable post-processing unit is configured to acquire instructions from the top-level controller and perform post-processing on the multiply-accumulate results obtained from the reconfigurable computing array according to the instructions. An on-chip interconnect unified system is used to connect to the top-level controller, the on-chip storage unit, the reconfigurable computing array, and the configurable post-processing unit respectively, providing a programmable data path; the programmable data path is used to implement different perception algorithms through a unified execution phase.

[0027] It should be noted that the fusion accelerator provided by this invention is used in general scenarios for microrobot perception, and can support typical perception applications such as neural network inference, pose estimation visual front-end, and pose estimation optimization back-end. Microrobot perception refers to the robot acquiring and processing environmental information through sensors.

[0028] Among them, the fusion accelerator is a general-purpose processor architecture that supports perception computation tasks with multiple different operators in the field of micro-robot perception. The programmable data path is the hardware foundation for the fusion accelerator to execute different perception algorithms through a unified execution phase.

[0029] For example, a reconfigurable computing array can be used as the main computing unit, combined with a configurable post-processing unit and an on-chip unified interconnect system to form a programmable data path.

[0030] Reconfigurable computing arrays are hardware computing units based on addition tree structures. They can dynamically reconfigure the computing structure according to different algorithm requirements, enabling general computing that integrates multiple operators.

[0031] A configurable post-processing unit is a processing unit that can dynamically configure the post-processing algorithm to be executed as the perception algorithm changes. Post-processing operations may include ReLU activation function, covariance calculation, and column decomposition calculation, etc.

[0032] It should be noted that by executing different perception algorithms through a unified execution phase, the execution of the fusion adder can be uniformly divided into five phases: fetch and translate, data transfer, matrix operation, post-processing, and write-back, when performing different perception operations.

[0033] For example, the top-level controller may include a 4kB instruction storage and fetch circuit and an instruction decoder circuit. The fetch circuit is used to read the original instruction from the central processing unit, and the instruction decoder circuit decodes the original instruction into instructions that can be understood by the on-chip reconfigurable computing array, configurable post-processing unit, and on-chip interconnect unified system.

[0034] Understandably, the top-level controller can achieve the interaction between data from the off-chip central processing unit and data on the chip by obtaining instructions sent by the central processing unit.

[0035] It should be noted that the on-chip storage unit, also known as unified data storage, can be connected to a configurable post-processing unit via a 32-bit data channel, to a reconfigurable computing array via a 32 / 128-bit data channel, and to receive external data via a 32-bit APB type interface.

[0036] Understandably, converged accelerators employ unified memory management, which can reduce redundant storage, improve resource utilization, and further optimize area efficiency.

[0037] For example, the size of an on-chip storage unit can be 512kB.

[0038] The fusion accelerator provided in this embodiment of the invention can obtain instructions sent by the central processing unit, enabling the reconfigurable computing array to compute the data to be processed in different modes on the same hardware architecture to obtain multiply-accumulate results. The configurable post-processing unit performs corresponding post-processing on the multiply-accumulate results, realizing the execution of different perception algorithms through a unified execution stage. This avoids the integration problem of multiple dedicated accelerators and improves the utilization rate of system computing resources.

[0039] Based on the above embodiments, the reconfigurable computing array includes: At least two multipliers are used to calculate partial products of the data to be processed, and the partial products are transmitted to the selector. In the case of constructing a solver processing unit, the selector is used to receive the partial product transmitted by the corresponding multiplier and transmit the partial product to the adder according to the instruction. An adder is configured to receive a partial product transmitted by the selector of the first multiplier among the at least two multipliers, obtain a cached result from the register corresponding to the first multiplier, and perform an in-situ update based on the partial product of the first multiplier and the cached result; receive a partial product transmitted by the selector of the second multiplier among the at least two multipliers, obtain a cached result from the register corresponding to the second multiplier, and perform an in-situ update based on the partial product of the second multiplier and the cached result; the first multiplier and the second multiplier are adjacent.

[0040] For example, the selector can be a MUX multiplexer.

[0041] It should be noted that when processing linear optimization solutions in robot localization algorithms, every two adjacent multipliers in the addition tree, the corresponding multiplication stage register, and the corresponding adder in the first level of the addition tree can be reconstructed into a solver processing unit (SPE).

[0042] like Figure 2 As shown, the reconfigurable computing array may include 32 RVEs from RVE#0 to RVE#31. Each RVE may include a register, selector and adder corresponding to at least two multiplier products, resulting in an 8×32 SPE array.

[0043] The row direction is [W] 1,1 … W 1,16 The column direction is [x1 x2 … x]. 16 After the matrix input is reconfigurable computation array, under the action of the selector, the first multiplier can calculate the partial product c*r0 by calculating c and r0. Based on the partial product c*r0 and the cached result a1 in the register corresponding to the first multiplier, a1-c*r0 is obtained and stored in the register corresponding to the first multiplier for in-place update. Similarly, the second multiplier can calculate the partial product c*r1 by calculating c and r1. Based on the partial product c*r1 and the cached result a2 in the register corresponding to the second multiplier, a2-c*r1 is obtained and stored in the register corresponding to the second multiplier for in-place update.

[0044] The data path between the adder, multiplier, and corresponding register can be 32 bits.

[0045] Based on any of the above embodiments, the reconfigurable computing array includes: At least two multipliers are used to calculate partial products of the data to be processed, and the partial products are transmitted to the selector. In the case of constructing a multiply-accumulate array, the selector is used to receive the partial product transmitted by the corresponding multiplier and transmit the partial product to the corresponding register according to the instruction. An adder is configured to obtain a first partial product from the register corresponding to the first multiplier of the at least two multipliers, and a second partial product from the register corresponding to the second multiplier of the at least two multipliers, to output a partial sum.

[0046] It should be noted that when processing matrix-vector multiplication, every two adjacent multipliers in the addition tree, the corresponding multiplication stage register, and the corresponding adder in the first level of the addition tree can be reconstructed into a 16x32 multiply-accumulate array with a computational precision of BF16.

[0047] Under the action of the selector, the first multiplier can cache the calculation result in the corresponding register, and the adjacent second multiplier can cache the calculation result in the corresponding register. The adder retrieves the cached calculation result from the two registers and performs the accumulation operation.

[0048] It is understandable that by controlling the selector through instructions, the computation array can be reconstructed into the corresponding multiply-accumulate array or solver processing unit, thereby achieving efficient support for at least two very different operators, neural network inference and optimization solution, on the same computation array.

[0049] Based on any of the above embodiments, the initial cached result in the register corresponding to the multiplier is obtained by the configurable post-processing unit from the data to be processed in the on-chip storage unit.

[0050] Taking Cholesky decomposition as an example, the initial cached result can be the result of parallel computation of the first column of the matrix obtained by the configurable post-processing unit from the on-chip storage unit. The matrix is ​​one of the smaller 32*16 matrices that the target matrix is ​​split into and transposed and stored in the on-chip storage unit.

[0051] like Figure 3 As shown, taking the Cholesky decomposition operation process as an example, the following steps are executed at least once: first, the first column of the uncomputed part of the matrix is ​​taken out and its result is calculated in parallel in the configurable post-processing unit, and the result of the first column is written back to the on-chip storage unit to broadcast the result of this column to the reconfigurable computing array composed of solver processing units, and the remaining part of the matrix is ​​updated in place, and the loop continues until the entire matrix is ​​updated.

[0052] It is understandable that the Cholesky decomposition algorithm requires processing the columns of the matrix step by step, and the calculation of the first column is the basis for subsequent steps. In this embodiment, the calculation is performed by a configurable post-processing unit and the result is transmitted to the register corresponding to the multiplier, which can facilitate maximizing the utilization of the computing array.

[0053] Based on any of the above embodiments, the data broadcasting mechanism of the solver processing unit is a two-dimensional data broadcasting mechanism of rows and columns.

[0054] The row-column two-dimensional data broadcasting mechanism refers to the two-dimensional broadcasting of data along the row and column directions in a reconfigurable computing array.

[0055] For example, column-direction interconnects for fast data transfer can be added to a reconfigurable computing array through hardware design. It is understood that, based on this, and combined with registers in the solver processing units used to store in-situ update results, the results of the first column can be broadcast to the reconfigurable computing array composed of solver processing units, thereby more efficiently supporting linear algebra computations requiring complex data exchange, such as Cholesky decomposition.

[0056] For example, when the data is broadcast in two dimensions, the bit width on the broadcast path is 16 bits.

[0057] like Figure 2 As shown, a 32B solver buffer H for row-direction broadcasting and a 64B solver buffer V for column-direction broadcasting can be obtained based on a 96B solver buffer. Here, #0 indicates that column-direction propagation is on RVE#0, and #0&1 indicates that row-direction propagation is from RVE#0 to RVE#1.

[0058] Understandably, different operators can also lead to differences in computational precision. For example, neural network inference can use precision such as FP4 or INT8 after quantization, while Cholesky decomposition still requires BF16 for computation after quantization optimization.

[0059] To enable reconfigurable computing arrays to meet multi-precision requirements, such as Figure 4 As shown, based on any of the above embodiments, the multiplier includes a multiplication unit, a 4-bit addition unit, and an 8-bit addition unit; wherein, the multiplication unit and the 4-bit addition unit cooperate to support floating-point 4-bit multiplication calculations, and the multiplication unit and the 8-bit addition unit cooperate to support floating-point 16-bit multiplication calculations.

[0060] It should be noted that the multiplication unit and the 8-bit addition unit can form an exponential-multiplication computation path (EDP) to support 16-bit floating-point multiplication computations, such as BF16 multiplication computations; the multiplication unit and the 4-bit addition unit can form a mantissa-multiplication computation path (MDP) to support 4-bit floating-point multiplication computations, such as FP4 multiplication computations.

[0061] For example, in the exponentiation-multiplication calculation path, INT8 and FP8 man can be input into an 8-bit multiplication unit, and FP8exp and BF16exp can be input into an 8-bit addition unit. In the mantissa-multiplication calculation path, INT8, FP8 man, and BF16 man can be input into an 8-bit multiplication unit, and FP8exp and FP8exp can be input into a 4-bit addition unit.

[0062] Understandably, when processing low-precision data such as FP4, the multiplier can utilize the newly added 4-bit addition unit to perform the low-bit-width calculations required for the multiplication operation; when processing higher-precision data such as BF16, the multiplier can utilize the newly added 8-bit addition unit to handle a wider data path; for natively supported 8-bit precision such as INT8 and FP8, the multiplier can directly use its basic 8-bit multiply-accumulate logic.

[0063] For example, the calculation results of EDP and MDP can be transmitted to the adder through the BF16 multiplication path, the adders are cascaded for BF16 addition, and finally output through concat processing.

[0064] Based on any of the above embodiments, the adder includes an adder unit, a comparator, and a local normalization unit; wherein, the comparator controls the adder unit and the local normalization unit to support 4-bit accumulation calculation based on the partial product of floating-point 4-bit multiplication calculation, and the comparator controls the adder unit and the local normalization unit to support 8-bit accumulation calculation based on the partial product of floating-point 8-bit multiplication calculation.

[0065] For example, the comparator can be used to analyze the precision characteristics of the partial product and generate control signals to control the addition unit to perform the core addition operation to accumulate the partial product and obtain the accumulated result, and control the local normalization unit to normalize the accumulated result to conform to the floating-point format.

[0066] It is understandable that instruction control can enable the computing array to select different computing units for computation to achieve logic reuse, thereby supporting different precisions through the same computing array and further reducing the area occupied by the computing array.

[0067] Among them, the reconfigurable computing array, also known as the fusion computing array, supports multi-operator and multi-precision fusion.

[0068] Compared to using three independent precisions, this embodiment achieves a 46.7% reduction in computing array area through logic multiplexing.

[0069] In some embodiments, the adder tree of the reconfigurable computation array supports early termination to reconfigure it into two independent adder trees.

[0070] Understandably, in fine-grained parallel multiply-accumulate operations, the utilization of the computing array can be improved by designing the addition tree of the reconfigurable computing array to terminate early and reconfigure into two independent addition trees.

[0071] For example, fine-grained parallel multiply-accumulate operations can be used in image filtering operators in pose estimation.

[0072] Based on any of the above embodiments, the instruction includes a fusion instruction and a basic instruction; wherein the fusion instruction is obtained by decomposition according to a perception algorithm; and the basic instruction is obtained by decomposition according to the fusion instruction.

[0073] For example, the perception algorithm may include a neural network inference, a pose estimation visual front-end, and a pose estimation optimization back-end. Key fusion operators such as convolution operations, Transformer matrix multiplication, neural network inference, CONV-BN layers, and other complex operations supported by arithmetic units in configurable post-processing units can be obtained by decomposing the neural network inference; key fusion operators such as image filtering, feature point detection, feature point tracking, Rodrigues transform, and other complex operations supported by arithmetic units in configurable post-processing units can be obtained by decomposing the pose estimation visual front-end; and key fusion operators such as Cholesky decomposition, Rodrigues transform, and other complex operations supported by arithmetic units in configurable post-processing units can be obtained by further decomposing the pose estimation optimization back-end.

[0074] The matrix operation portion corresponding to the reconfigurable computation array can be decomposed into highly parallel matrix-vector multiplication based on convolution operations and Transformer matrix multiplication, into low-parallel matrix-vector multiplication based on image filtering, and into two-dimensional broadcasting and in-situ updates based on Cholesky decomposition. The post-processing operation portion corresponding to the configurable post-processing unit can be decomposed into ReLU based on neural network inference and CONV-BN layers, into eigenvalue solving based on feature point detection and feature point tracking, and into square root division based on Cholesky decomposition.

[0075] Image filtering can be achieved using methods such as Sobel or Gaussian.

[0076] Please refer to the table below for details: Understandably, when programming, one can either directly call the fusion instructions using the already implemented templates, or implement more computational processes by controlling the processor with fine granularity.

[0077] In some embodiments, the configurable computation array further includes an input buffer, a solver buffer, and an accumulation buffer. The input buffer can be 32 bytes, the solver buffer can be 96 bytes, and the accumulation buffer can be 16 kB.

[0078] For example, during the weight loading phase of neural network inference, the weight data to be used in the on-chip storage unit can be moved into the registers of the reconfigurable computing array until the array is full.

[0079] In the multiply-accumulate calculation stage of neural network inference, a row of input activation values, whose size is adapted to the array height, is first read from the on-chip memory each cycle. This input activation value vector is then broadcast to 32 addition trees as the same input, and each tree performs parallel multiply-accumulate operations with the weights stored in the array, generating 32 corresponding accumulated results. These 32 accumulated results are then stored as partial sums of a vector-matrix multiplication result in an accumulation buffer, and the above steps are repeated until the accumulation buffer is full of partial sums of vector-matrix multiplications from different rows.

[0080] Then, returning to the beginning of the accumulator, the system continues to read the corresponding data from the unified data cache into the computation array. After the computation is completed in the array, the result is added to the result in the accumulator, and then written back to the accumulator. This process is repeated until the matrix multiplication in this part is finished.

[0081] In the post-processing stage of neural network inference, data in the accumulator buffer can be fed line by line into a highly flexible post-processing unit, and the post-processing computation process can be configured according to the application. For example, in convolutional neural networks, post-processing can be configured as normalization and ReLU modules. In this stage, the results of matrix multiplication are processed in parallel in the post-processing unit, then output line by line, and the results output by the post-processing unit are written back to the on-chip memory.

[0082] In some embodiments, each module's instructions have their own corresponding configuration words and operation words, resulting in 32bVLIW type instructions to support fine-grained control of the various modules within the accelerator and the data flow between them during programming.

[0083] The instruction decomposition for the 32bVLIW type is shown in the table below: For example, the computational process for optimizing feature point detection based on the Sobel operator for classical images can be implemented through the following steps.

[0084] Step 1: Load the Sobel image convolutional filter kernel into the input activation value of the input buffer of the reconfigurable computation array in one cycle and broadcast it to the processing unit of the reconfigurable computation array.

[0085] Step 2: Continuously read a portion of the input image from the on-chip storage unit, vectorize it, and then feed it as weights into the reconfigurable computing array. In this step, one row of the input image is read per cycle, and register buffering allows a set of vectors to be output to the reconfigurable computing array as weights each cycle after a cold start.

[0086] For example, the number of vectors in a set can be 32. Cold start is 3 cycles for filtering a 3x3 image and 4 cycles for filtering a 5x5 image; Step 3: Perform multiplication and accumulation operations in the reconfigurable array to obtain gradient maps in part of the X and Y directions.

[0087] Step 4: The vectorized gradient maps in the X and Y directions are used as inputs and weights and fed back into the reconstructable array to obtain the covariance map.

[0088] Step 5: Post-process the covariance map in the configurable post-processing unit to obtain the feature value corresponding to each pixel value.

[0089] Step 6: Write the feature value results back to the on-chip memory.

[0090] Step 7: Return to the previous step until the entire image has been processed and the image's feature values ​​are obtained.

[0091] The cold start in step two will not be affected by step four.

[0092] For example, in supporting Cholesky decomposition, at the start of the computation, the target matrix can be divided into small 32*16 matrices and transposed and stored in on-chip storage. Then, the top-left small matrix can be read and stored as a weight in a reconfigurable computation array.

[0093] The first submatrix can be updated at the following intervals: Cycle 1: Read the first column of data from the first submatrix and send it to the configurable post-processing unit; Period 2: Calculate the square root of the first data in the first column using a configurable post-processing unit; Period 3: Divide the square root of the first data obtained from Period 2 by the remaining data in the first column to get the result of the first column; for example... Figure 3 As shown, post-processing yields... l 11 , ..., l n1 ; Cycle 4: Write the results of the first column of Cycle 3 back to the on-chip memory and the solver cache next to the array; Period 5: Broadcast the results of Period 4 in two dimensions in the array and perform two in-situ updates in each solver unit; Then, cycle 1 through cycle 5, a total of 15 times, is performed to update the subarray and simultaneously write the results back to the on-chip memory. Figure 3 As shown, the result obtained from the post-processing of two-dimensional broadcasting ( l 11 , ..., l n1 And update the rest of the matrix; When updating subsequent submatrices, the results of the corresponding columns need to be read from the on-chip storage unit first, and then broadcast two-dimensionally to the array for updating according to the aforementioned period 5; such as Figure 3 As shown, we return to the first step starting from the remaining matrix. Taking the submatrix with index {3,2} as an example, it needs to read the columns from the submatrix results with indices {2,1} and {3,1} in sequence to complete the preliminary update, and then complete the update of the submatrix itself.

[0094] After all submatrices have been updated, the Cholesky decomposition of the target matrix is ​​completed, and the result is written to the on-chip storage unit.

[0095] The fusion accelerator provided by this invention was designed, manufactured, and tested using TSMC's 28nm 1p10m7x2r HPC Plus process. The final prototype chip system is shown below. Figure 5 As shown, the chip can achieve 90.9 FPS / mm² at the 28nm process node. 2 The visual navigation perception throughput density and perception energy efficiency of 103 uJ / frame are both higher than those of systems composed of existing dedicated accelerators, by 4.1 and 1.5 times, respectively. The fusion computing array occupies 51% of the core area. When the microrobot's task scenario changes, it can achieve a maximum computational density improvement of 8.9 times. Its performance is summarized in the table below: The prototype chip system includes control flow running on a zero-riscy CPU [PDSchiavone, PATOMS'17], which was implemented and tested on another 28nm chip in the same group.

[0096] The dimensions of the fusion accelerator are 1.98mm × 1.73mm.

[0097] The execution method of the perception algorithm provided by the present invention is described below. The execution method of the perception algorithm described below can be referred to in correspondence with the fusion accelerator described above.

[0098] Figure 6This is a flowchart illustrating the execution method of the perception algorithm provided by the present invention, as shown below. Figure 1 As shown, the method includes the following steps: Step 601: Obtain instructions sent by the central processing unit through the top-level controller, and transmit the instructions to the reconfigurable computing array, the configurable post-processing unit, and the on-chip unified interconnect system respectively; Step 602: The data to be processed from the fused accelerator stored in the on-chip storage unit is moved to the reconfigurable computing array through the on-chip unified interconnect system, so as to control the reconfigurable computing array to calculate the data to be processed according to the instructions and obtain the multiplication-accumulation result; Step 603: Control the configurable post-processing unit to perform post-processing on the multiplication and accumulation results obtained by the reconfigurable computing array according to the instructions, and obtain the post-processing result; Step 604: Write the post-processing result back to obtain the execution result of the perception algorithm.

[0099] The present invention also provides a computing device including a fusion accelerator for a perception algorithm as described in any of the preceding claims.

[0100] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A fused accelerator for perception algorithms, characterized in that, The application relates to a top-level controller for obtaining instructions sent by a central processing unit; an on-chip storage unit for storing data to be processed of the fusion accelerator; a reconfigurable computing array for respectively obtaining instructions of the top-level controller and data to be processed of the on-chip storage unit to calculate the data to be processed according to the instructions to obtain a multiply-accumulate result; a configurable post-processing unit for obtaining instructions of the top-level controller to post-process the multiply-accumulate result obtained by the reconfigurable computing array according to the instructions; and an on-chip interconnection unified system for being connected with the top-level controller, the on-chip storage unit, the reconfigurable computing array and the configurable post-processing unit respectively to provide a programmable data path; the programmable data path is used for implementing different perception algorithms through a unified execution stage. The reconfigurable computing array comprises at least two multipliers for calculating partial products of the data to be processed and transmitting the partial products to a selector; the selector is used for receiving the partial products transmitted by the corresponding multipliers and transmitting the partial products to an adder according to the instructions when the instructions are for constructing a solver processing unit; and the adder is used for receiving the partial products transmitted by the selector of a first multiplier of the at least two multipliers and obtaining a cache result from a register corresponding to the first multiplier to perform in-place updating according to the partial product and the cache result of the first multiplier, receiving the partial products transmitted by the selector of a second multiplier of the at least two multipliers and obtaining a cache result from a register corresponding to the second multiplier to perform in-place updating according to the partial product and the cache result of the second multiplier, and the first multiplier is adjacent to the second multiplier. The reconfigurable computing array comprises at least two multipliers for calculating partial products of the data to be processed and transmitting the partial products to a selector; the selector is used for receiving the partial products transmitted by the corresponding multipliers and transmitting the partial products to a corresponding register according to the instructions when the instructions are for constructing a multiply-accumulate array; and the adder is used for obtaining a first partial product from a register corresponding to a first multiplier of the at least two multipliers and obtaining a second partial product from a register corresponding to a second multiplier of the at least two multipliers to output a partial sum. The initial cache result in the register corresponding to the multiplier is calculated by the configurable post-processing unit through the data to be processed of the on-chip storage unit. The data broadcast mechanism of the solver processing unit is a row-column two-dimensional data broadcast mechanism. The multiplier comprises a multiplication unit, a 4-bit adder unit and an 8-bit adder unit; the multiplication unit and the 4-bit adder unit are matched to support floating-point 4-bit multiplication calculation, and the multiplication unit and the 8-bit adder unit are matched to support floating-point 16-bit multiplication calculation.

2. The fused accelerator for perception algorithms of claim 1, wherein, ​ ​ ​ ​ 3. The fused accelerator for perception algorithms of claim 1, wherein, ​ ​ ​ ​ 4. The fused accelerator for perception algorithms of claim 2, wherein, ​ 5. The fusion accelerator of claim 2, wherein, ​ 6. The fused accelerator for perception algorithms of claim 2 or 3, wherein, ​ 7. The fused accelerator for perception algorithms of claim 6, wherein, The adder comprises an adding unit, a comparator and a local normalization unit; wherein the comparator controls the adding unit and the local normalization unit to support 4-bit accumulation calculation according to partial products of floating-point 4-bit multiplication calculation, and controls the adding unit and the local normalization unit to support 8-bit accumulation calculation according to partial products of floating-point 8-bit multiplication calculation.

8. The fused accelerator for perception algorithms of claim 1, wherein, The instructions comprise fusion instructions and basic instructions; wherein the fusion instructions are obtained by decomposing a perception algorithm; and the basic instructions are obtained by decomposing the fusion instructions.

9. A method of executing a perception algorithm, the method comprising: The fusion accelerator for the perception algorithm according to any one of claims 1 to 7 comprises: An upper-layer controller acquires instructions sent by a central processing unit and transmits the instructions to a reconfigurable computing array, a configurable post-processing unit and an on-chip uniform interconnection system respectively; The reconfigurable computing array is controlled to calculate to-be-processed data of the fusion accelerator stored in an on-chip storage unit according to the instructions through the on-chip uniform interconnection system, so as to obtain multiplication-accumulation results; The configurable post-processing unit is controlled to post-process the multiplication-accumulation results obtained by the reconfigurable computing array according to the instructions, so as to obtain post-processing results; The post-processing results are written back to obtain execution results of the perception algorithm.

10. A computing device, comprising: The fusion accelerator for the perception algorithm according to any one of claims 1 to 8.

Images