Reconfigurable tensor calculation unit

Abstract

The invention discloses a reconfigurable tensor calculation unit, which comprises a pipeline controller, a processor and a processor, wherein the pipeline controller is responsible for analyzing an instruction, configuring a calculation mode and scheduling a task; the matrix calculation module is composed of reconfigurable subarrays and is used for executing multi-precision matrix multiplication and addition operation; the matrix cache module comprises four types of caches; and the AXI bus access module is used for realizing efficient data interaction with an off-chip memory through a DMA (Direct Memory Access) controller. Firstly, a matrix base address, calculation precision, a mixed mode mark and a target dimension are configured through a register; and then the DMA carries the matrix data into a corresponding cache through an AXI bus. The cache carries out dynamic recombination on the data according to the current mode and then sends the data to the reconfigurable array for calculation. After calculation is completed, a result is recombined through the D cache and written back to the memory. By using the tensor accelerator, the defects of an existing tensor accelerator in the aspects of flexibility, energy efficiency and hardware resource utilization rate can be overcome. The method can be widely applied to the field of hardware accelerators.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of hardware accelerators, in particular to a reconfigurable tensor computing unit. BACKGROUND

[0002] With the rapid development of artificial intelligence and deep learning technologies, tensor computing has become the core computing load of various intelligent applications (such as computer vision, natural language processing, and autonomous driving, etc.). Among them, large-scale matrix multiply-accumulate (MAC) operations are the key operations in the process of deep neural network inference and training, which puts high requirements on the computing efficiency and energy efficiency of the underlying hardware. Although traditional general-purpose processors (such as CPUs and GPUs) have strong versatility, they have significant energy efficiency bottlenecks when processing high-density tensor operations, making it difficult to meet the needs of edge or data center for low-power, high-performance accelerators.

[0003] To address this challenge, special-purpose tensor processing units (TPUs) have emerged. Existing accelerators represented by Google TPU usually adopt a large-scale systolic array architecture, which realizes efficient matrix operations through highly parallel processing elements (PEs). However, this type of architecture is mostly based on homogeneous PEs, and its fixed array structure often leads to a large amount of idle computing resources and low hardware utilization when facing tensor computing tasks of different dimensions or precisions. In addition, current mainstream AI models widely use multiple data precisions (such as INT4, INT8, FP16, BF16, FP32) during training and inference stages, and even introduce mixed precision strategies to balance precision and efficiency. However, most existing TPU designs only support a single or limited number of precision formats, lacking the flexibility to adapt to multiple precision and mixed precision modes, limiting their applicability in diverse AI scenarios. SUMMARY

[0004] Therefore, in order to solve the deficiencies of existing tensor computing units in flexibility, energy efficiency, and hardware resource utilization, the present application proposes a reconfigurable computing unit, which includes: a pipeline controller responsible for parsing instructions, configuring computing modes (precision types, matrix dimensions), and scheduling tasks; a matrix computing module composed of four 8x8 reconfigurable sub-arrays, which performs multi-precision matrix multiply-accumulate operations; a matrix cache module including four types of caches: first, second, third, and fourth cache areas, which are used to temporarily store input matrices A / B, accumulated initial values C, and output results D, and have mode-aware data packing and mapping capabilities; AXI bus access module: through the DMA controller, efficient data interaction with the off-chip memory is realized.

[0005] When the system starts, first, the register is configured with the matrix base address, the calculation precision (such as FP16 or INT8), the mixed mode flag and the target dimension (such as m16n16k16). Then, the DMA moves the A, B and C matrix data into the corresponding cache through the AXI bus from the external DDR. After the cache dynamically reorganizes the data according to the current mode, the data is sent to the reconfigurable array for calculation. After the calculation is completed, the result is reorganized by the D cache and written back to the memory.

[0006] Based on the above scheme, the application provides a reconfigurable tensor calculation unit, which is based on a reconfigurable architecture, and combines multiple precision support, mixed precision calculation, dynamic array reconfiguration, unified floating point processing path and mode-aware intelligent cache and other key technologies, realizes the comprehensive compatibility of multiple mainstream AI data formats (including INT4, INT8, FP16, BF16, FP32) and two mixed precision modes (FP16+FP32, BF16+FP32), and can efficiently execute the matrix multiplication and addition operation under three typical matrix dimension configurations (m16n16k16, m32n8k16, m8n32k16). BRIEF DESCRIPTION OF DRAWINGS

[0007] Figure 1 is the overall block diagram of the reconfigurable tensor calculation unit of the application; Figure 2 is the structural schematic diagram of the specific embodiment array of the application; Figure 3 is the m16n16k16 mode schematic diagram of the specific embodiment of the application Figure 4 is the m32n8k16 mode schematic diagram of the specific embodiment of the application; Figure 5 is the m8n32k16 mode schematic diagram of the specific embodiment of the application; Figure 6 is the structural schematic diagram of the PE of the specific embodiment of the application; Figure 7 is the normalization unit diagram of the specific embodiment of the application; Figure 8 is the cache A schematic diagram of the specific embodiment of the application; Figure 9 is the cache D structural schematic diagram of the specific embodiment of the application; Figure 10 is the cache D mode output schematic diagram of the specific embodiment of the application; Figure 11 is the resource consumption diagram of the specific embodiment of the application; Figure 12is a power consumption graph of a specific embodiment of the present invention. DETAILED DESCRIPTION

[0008] In addition to the problems mentioned in the background, the prior art also has: a mixed reconstruction-based multi-precision convolutional neural network accelerator, which dynamically configures computing resources to adapt to different precision requirements; and: respectively from the aspects of floating point processing unit reuse, three-dimensional systolic array expansion, and multi-precision fused multiply-add (FMA) structure are optimized. However, these schemes either rely on special DSP resources, or fail to achieve fine-grained reconstruction at the subarray level, or still have problems such as rigid data flow scheduling and low cache efficiency when supporting multi-dimensional matrix operations. Especially in terms of output data reorganization, most designs still entrust the result rearrangement task to software or DMA controllers, causing waste of I / O bandwidth and increase of system delay.

[0009] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0010] It should be noted that, for the convenience of description, only the parts related to the present application are shown in the drawings. The embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0011] It should be understood that the "system", "device", "unit" and / or "module" used in the present application is a method for distinguishing different components, elements, parts, portions or assemblies at different levels. However, if other words can achieve the same purpose, the words can be replaced by other expressions.

[0012] As shown in the present application and claims, unless the context clearly indicates otherwise, "one", "a", "an", and / or "the" do not refer to the singular, but also include the plural. Generally speaking, the terms "comprise" and "include" only indicate the inclusion of the steps and elements explicitly identified, and these steps and elements do not constitute an exclusive list, and the method or device can also include other steps or elements. The element defined by the statement "comprising a" does not exclude the presence of another identical element in the process, method, product or device comprising the element.

[0013] In the description of the embodiments of the present application, "multiple" refers to two or more than two. The following terms "first", "second" are only for the purpose of description, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include one or more features.

[0014] In addition, flowcharts are used in the present application to illustrate the operations performed by the system according to the embodiments of the present application. It should be understood that the preceding or subsequent operations are not necessarily performed in sequence. On the contrary, each step can be processed in reverse order or simultaneously. Meanwhile, other operations can be added to these processes, or one or more steps of operations can be removed from these processes.

[0015] Reference Figure 1 The architecture diagram of an optional example of the reconfigurable tensor computing unit proposed in the present application. The tensor computing unit proposed in the present embodiment can include but is not limited to the following parts: AXI_Lite register, used for data exchange with the outside; Pipeline controller, responsible for parsing instructions, configuring computing modes (precision type, matrix dimension), scheduling tasks, specifically including decoding, pulling data, setting data, obtaining results and pushing results; Computing core, i.e. matrix computing module, composed of 4 8x8 reconfigurable subarrays, used for performing multi-precision matrix multiplication and addition operation; Matrix buffer module, including A buffer, B buffer, C buffer and D buffer, used for temporarily storing input matrix A / B, accumulated initial value C and output result D, and having mode-aware data packing and mapping capability; AXI bus access module: through the DMA controller, efficient data interaction with the off-chip memory is realized.

[0016] In some feasible embodiments, the computing core is composed of 4 independent 8x8 PE subarrays (see Figure 2 ). Each subarray is equipped with a row-column broadcast bus, supporting outer product computing mode. Through the centralized interconnection switch after the buffer interface, the subarray connection mode is dynamically reconfigured according to the matrix dimension signal: m16n16k16 mode: subarrays 0 / 1 are connected in horizontal parallel, 2 / 3 are connected in horizontal parallel, and then are vertically spliced into a 16x16 array (see Figure 3 ); m32n8k16 mode: four subarrays process the same 8-column B data in parallel, A matrix is input in blocks, forming a 32x8 effective computing area (see Figure 4 ); m8n32k16 mode: four subarrays receive the same A row data in parallel, and the B matrix is input in column block, and the output is spliced into 8x32 results (see Figure 5 ).

[0017] Here, the order describes the relative position of the array, which is necessary to ensure that the input can be correctly mapped to the position of the subarray.

[0018] This reconstruction mechanism only requires a small number of multiplexers, avoiding the waste of resources in traditional large arrays when calculating non-square matrices.

[0019] Each PE uses a five-stage pipeline, which includes a unified data path (see Figure 6 ): Input preprocessing: according to the global mode signal, INT4 / INT8 is expanded to 13-bit complement; FP16 / BF16 is split into sign, exponent, and mantissa, and the mantissa is expanded and spliced with the sign to form a 13-bit operand; FP32 splits its mantissa into high / low 12 bits, and the complete product is achieved by splicing twice.

[0020] Shared computing core: all precisions share a 13-bit Booth multiplier array; floating-point exponent operations (addition, alignment, comparison) are completed by a shared exponent adder and comparator; Accumulation and normalization: intermediate results are accumulated in the accumulator, and finally sent to the normalization unit (see Figure 7 ). Normalization uses a delay strategy and is only executed once in the final result stage, including leading 1 detection, exponent adjustment, mantissa truncation, and overflow judgment; Special value processing: 4-bit special value flags (Sign / NaN / Inf / Zero) are generated in the input stage, and Boolean logic judgment and propagation are performed according to IEEE 754 rules at key nodes.

[0021] A cache: uses a 4x8x16 register array (32-bit wide). In non-FP32 mode, the elements of the adjacent two columns are packed into the same unit; when reading, the column priority order is broadcast to the corresponding subarray through the output switch (see Figure 8 ); B cache: similar structure, but in non-FP32 mode, the adjacent two rows are packed; C cache: directly mapped to the accumulator initial value port of the corresponding PE; D cache: composed of 4 distributed FIFOs (corresponding to 4 subarrays) and a configurable data stream selector (see Figure 9). According to the mode dynamic reorganization output sequence: m16n16k16 under the interlaced output [FIFO0, FIFO1] and [FIFO2, FIFO3]; m32n8k16 under the polling output of each FIFO continuous block; m8n32k16 under the whole FIFO is read in turn to splice 8x32 matrix (see Figure 10 ). The design ensures that the output data is continuously stored in the memory, maximizing the DMA burst transmission efficiency.

[0022] At the architecture level, the present application decomposes the traditional large-scale homogeneous systolic array into four 8x8 sub-arrays, dynamically reconfigures the data flow path between the sub-arrays through the centralized interconnection switch, thereby flexibly adapting to different shape tensor operation tasks, and significantly improving the hardware utilization. Each sub-array internally uses shared data processing modules and row-column broadcast mechanism to greatly reduce redundant logic; the normalization unit uses time division multiplexing strategy, and only one row of normalization circuit is configured to poll all 8 rows of results, saving 87.5% of hardware overhead.

[0023] At the computing unit level, the present application designs a highly multiplexed unified data path: all precision type mantissa or integer operands are mapped to 13-bit complement format, sharing the same set of Booth multipliers; floating point exponent operations (addition, alignment, comparison) are completed by shared exponent processing unit; FP32 high precision operation is implemented by splicing partial products through low precision multiplier, avoiding the use of dedicated high bandwidth multiplier. In addition, the delay normalization strategy is introduced, and only one normalization operation is performed after accumulation, effectively suppressing the accumulation of intermediate rounding errors and reducing the risk of overflow misjudgment. At the same time, the special value (NaN, Inf, Zero) detection and propagation logic compatible with IEEE 754 standard is integrated, ensuring the reliability of floating point operation.

[0024] In terms of storage system, the present application innovatively proposes a mode-aware intelligent cache design. A / B / C cache can dynamically package input data according to the current computing mode and map it to the corresponding sub-array port; D cache (output cache) uses a distributed FIFO network and a configurable data flow selector to directly complete the continuous reorganization of non-square matrix results at the hardware level, so that DMA always transmits data with the maximum burst length, greatly improving I / O efficiency and AXI bus utilization.

[0025] Based on the above method overall process, the present application also gives the comparison data: Existing method 1: An fpga-based reconfigurable cnn accelerator for yolo; Prior art 2: Hardware resource and computational density efficient cnn accelerator design based on fpga Prior art 3: Design and Implementation of an Easy-to-Deploy Energy-Efficient Inference Acceleration System for Multi-Precision Neural Networks Prior art 4: A configurable floating-point multiple-precision processing element for HPC and AI converged computing Prior art 5: A reconfigurable processing element for multiple-precision floating / fixed-point HPC Prior art 6: A 3-D Multi-Precision Scalable Systolic FMA Architecture.

[0026] The resource consumption and power consumption of the present application refer to Figure 11 and Figure 12 , the working frequency of the present application after synthesis on the Xilinx VCU118 FPGA platform is 251.13 MHz, the peak computing power reaches 257.16 GOPS / GFLOPS (INT4 / INT8 / FP16 / BF16) and 64.29 GFLOPS (FP32). The system-level power consumption is only 6.056 W, and the energy efficiency ratio is as high as 42.46 GOPS / W, which is significantly better than the existing similar designs: compared with the prior art, the energy efficiency ratio is increased by 391%, 217% and 76% respectively. It is worth noting that the present design does not use any DSP hard core, and the high-performance computing is completely based on LUT logic, which shows excellent hardware resource efficiency (only 22.90% LUT and 8.85% FF are occupied). The power consumption of a single PE unit is as low as 0.125 W, and the balanced performance of 2.009 GFLOPS is realized under FP16 / BF16, and the FP32 performance is increased by 11.6% compared with the prior art.

[0027] Table 1 Performance comparison of the PE proposed in the present application and the prior art Table 2 Performance comparison of proposed TPU with existing methods The above is a specific description of the preferred embodiments of the present application, but the present application is not limited to the described embodiments, and those skilled in the art can make various equivalent modifications or replacements without departing from the spirit of the present application, and these equivalent modifications or replacements are all included in the scope defined by the claims of the present application.

Claims

1. A reconfigurable tensor computation unit, characterized in that, include: Pipeline controllers are used to parse instructions, configure computing modes, and schedule tasks; The matrix calculation module is used to perform multi-precision matrix multiplication and addition operations; The matrix caching module is used to cache data; The AXI bus access module is used for data interaction with off-chip memory.

2. The reconfigurable tensor computation unit according to claim 1, characterized in that, The matrix calculation module includes four 8×8 reconfigurable subarrays, and dynamically reconfigures the subarray connection method according to the matrix dimension signal, including: First mode: Connect subarray 0 and subarray 1 horizontally in parallel, connect subarray 2 and subarray 3 horizontally in parallel, and then splice them vertically to form a 16×16 array; Second mode: Four subarrays process the same 8 columns of second input matrix data in parallel. The first matrix data is divided into blocks by row and input to the four subarrays to form a 32×8 effective computing area. The third mode: Four subarrays receive the same 8 rows of first input matrix data in parallel, and input the second matrix data into the four subarrays in columns, and the output is spliced ​​into an 8×32 result.

3. The reconfigurable tensor computation unit according to claim 2, characterized in that, The matrix caching module includes: The first buffer is used to buffer the first input matrix; The second buffer is used to cache the second input matrix; The third buffer is used to cache the initial accumulated value; The fourth buffer is used to cache output results.

4. The reconfigurable tensor computation unit according to claim 3, characterized in that, Its working process is as follows: Configure the matrix base address, calculation precision, mixed mode flag, and target dimension through registers; The first input matrix data, the second input matrix data, and the initial accumulated value are moved into the corresponding cache from the external DDR via the AXI bus. The matrix caching module dynamically reorganizes the data according to the current mode and then sends it to the reconfigurable array for computation. After the calculation is completed, the matrix caching module reassembles the output results and writes them back to memory.

5. The reconfigurable tensor computation unit according to claim 4, characterized in that, Also includes: After the matrix cache module loads the data from the AXI bus access module, the pipeline master controller extracts the data from the matrix cache module and sends it to the tensor calculation unit. Based on the global mode signal, the tensor computation unit expands INT4 / INT8 into 13-bit two's complement; FP16 / BF16 is split into sign, exponent, and mantissa, and the mantissa is expanded and concatenated with the sign to form a 13-bit operand; FP32 splits its mantissa into high / low 12 bits and achieves the complete product by concatenating two multiplications. All precisions share a 13-bit Booth multiplier array; floating-point exponentiation is performed by a shared exponent adder and comparator. Intermediate results are accumulated in the accumulator and finally sent to the normalization unit; After the tensor calculation unit completes its calculation, the pipeline controller extracts the result from the tensor calculation unit and puts it into the matrix cache module, and finally sends it through the AXI bus access module.

6. The reconfigurable tensor computation unit according to claim 5, characterized in that, Also includes: The normalization unit adopts a delayed strategy and is executed only once in the final result stage, including leading 1 detection, exponent adjustment, mantissa truncation and overflow judgment.

7. The reconfigurable tensor computation unit according to claim 6, characterized in that, Also includes: Special value flags are generated during the input phase, and Boolean logic judgments and propagation are performed at critical nodes in accordance with the IEEE 754 rules.

Images