Multi-scale visual Transform accelerator architecture, edge chip and device

Abstract

The invention discloses a multi-scale visual Transform accelerator architecture, an edge chip and equipment, and the accelerator architecture comprises an instruction queue and an instruction scheduler which are used for receiving an instruction of a CPU (Central Processing Unit) and distributing the instruction to an IMC (International Mobile Communication) array or a DMA (Direct Memory Access) engine; the unified cache region is used for storing input features and output features of the convolution and a Query matrix and an output matrix of the MHA; the IMC array is used for storing the weight of the convolution or the Key and Value matrix of the MHA, and executing the convolution and MHA calculation; the DMA engine is used for internal storage of the accelerator and data transmission between the internal storage of the accelerator and an off-chip DRAM of the accelerator; the pre-processing module is used for reading input features of convolution or a Query matrix of MHA, aligning data and then sending the aligned data to the IMC array; and the post-processing module is used for collecting an output result of the IMC array and executing a corresponding activation function and quantization operation. The parallelism degree can be dynamically adjusted according to different layer sizes so as to improve the space utilization rate.

Description

Technical Field

[0001] The present invention relates to the field of integrated circuit technology, and in particular to a multi-scale visual Transformer accelerator architecture, edge chip and device based on in-memory computing. Background Art

[0002] The Transformer model is a type of neural network model, which is mainly composed of multiple Transformer Encoder blocks stacked together. Each block contains a Multi-head Attention (MHA) layer and a Feed-forward Network (FFN) layer. It has been proven that it can be used for computer vision tasks. Compared with CNN, the standard Vision Transformer (ViT) model has almost no image-related inductive bias, which often requires more parameters and calculations to learn the corresponding visual representation, making it difficult to deploy on the resource-constrained edge. Convolutional Neural Netowrk (CNN) has long dominated the field of computer vision and is relatively mature.

[0003] In order to take advantage of the advantages of both the Transformer model and CNN, a multi-scale ViT model has emerged, which adopts a hybrid network structure and has variable layer sizes. The hybrid structure contains more convolution calculations, and the variable layer size adopts a multi-stage design similar to modern CNN. The sequence length and number of heads change with different stages. In order to improve the space rate, the existing CNN accelerators can usually support dynamic data flow and dynamically adjust the parallelism allocation of the computing array during the calculation process to efficiently support convolution operations of different sizes; however, this type of accelerator is not optimized for the MHA layer in the Transformer, and cannot eliminate the additional intermediate data handling during the MHA matrix multiplication operation. Therefore, this accelerator cannot be used for the multi-scale ViT model; and the existing Transformer accelerator is not designed to be compatible with convolution calculations, does not support variable convolution kernel shapes, and because the softmax function in the MHA in the Transformer has the global reduction-broadcast property, it is usually difficult to dynamically allocate the parallelism of the computing array. It is usually only applicable to the standard Transformer with fixed layer size. When facing the multi-scale ViT with variable layer size, there is a risk of reduced resource utilization and may increase the calculation delay. Therefore, existing accelerators are not suitable for multi-scale ViT, and new accelerators suitable for multi-scale ViT need to be designed. Summary of the invention

[0004] To solve the above technical problems, the present invention provides a multi-scale visual Transformer accelerator architecture, edge chip and device based on in-memory computing, which can be applicable to the dynamic data flow of both convolution and MHA, can dynamically adjust the parallelism according to different layer sizes to improve space utilization, and is suitable for the deployment of multi-scale ViT.

[0005] The present invention provides a multi-scale visual Transformer accelerator architecture, comprising:

[0006] Instruction queue and instruction scheduler, which are used to receive instructions from the central processing unit and distribute them to the IMC array or direct memory access engine to control calculation or data transmission;

[0007] Unified buffer area, used to store the input features and output features of convolution, as well as the query matrix and output matrix of MHA;

[0008] IMC array, used to store convolution weights or MHA Key and Value matrices, and to perform convolution and MHA calculations;

[0009] Direct memory access engine, used for internal storage of the accelerator and data transfer with the accelerator's off-chip DRAM;

[0010] The pre-processing module is used to read the input features of the convolution or the query matrix of the MHA from the unified buffer, align the data and send it to the IMC array;

[0011] The post-processing module is used to collect the output results of the IMC array, accumulate them with the partial sums in the unified buffer, and perform corresponding activation functions and quantization operations.

[0012] The present invention also provides an edge chip, comprising the above-mentioned multi-scale visual Transformer accelerator architecture.

[0013] The present invention also provides an edge device, comprising the above-mentioned edge chip.

[0014] In the embodiment of the present invention, by introducing pipeline rearrangement to MHA and designing a three-stage softmax function, the communication overhead of multiple global reductions-broadcasts between computing blocks in the existing computing scheme is avoided, and the flexibility of mapping is improved; based on pipeline rearrangement, a reconfigurable IMC engine is designed, which efficiently supports convolution and MHA calculations through different modes, eliminates the intermediate data handling overhead for MHA calculations, and is friendly to the hybrid network structure of multi-scale ViT; based on pipeline rearrangement, a reconfigurable distribution network is designed, which efficiently supports the dynamic data flow of convolution and MHA, supports flexible mapping of convolution and MHA of different layer sizes, and effectively improves the space utilization of deploying multi-scale ViT. Compared with the prior art, the present invention realizes the dynamic data flow of both convolution and MHA, can dynamically adjust the parallelism according to different layer sizes to improve the space utilization, and is more suitable for the deployment of multi-scale ViT. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0016] Figure 1 A schematic diagram of a multi-scale visual Transformer accelerator architecture provided by an embodiment of the present invention;

[0017] Figure 2 Schematic diagram of the calculation operation of MHA;

[0018] Figure 3 for Figure 1 There are two working modes executed by each IMC engine in;

[0019] Figure 4 A schematic diagram of four working modes of a reconfigurable distribution network provided in an embodiment of the present invention;

[0020] Figure 5 A schematic diagram comparing the delay time and power consumption of the accelerator architecture provided by an embodiment of the present invention and the existing architecture. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution in the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] As mentioned above, the existing deep learning dedicated accelerators cannot support multi-scale ViT well. The main reason is that the existing accelerator architecture is usually only optimized for CNN or Transformer, and cannot efficiently process the hybrid network structure of multi-scale ViT; and the accelerator architecture for Transformer usually has a fixed degree of parallelism and does not support dynamic data flow, so there is a problem of reduced utilization when processing multi-scale ViT with variable layer sizes. To this end, in order to be suitable for multi-scale ViT, the present invention provides a multi-scale visual Transformer accelerator architecture based on in-memory computing, which optimizes the mapping flexibility of MHA through a pipeline rearrangement solution; based on pipeline rearrangement, a reconfigurable IMC computing engine is designed to efficiently support convolution computing and MHA computing in different modes, respectively, and reduce redundant intermediate data handling; at the same time, based on pipeline rearrangement, a reconfigurable distribution network is designed to support dynamic data flow of convolution computing and MHA computing, and improve array space utilization.

[0023] To make the technical solution of the present invention clearer, embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0024] Figure 1 A schematic diagram of a multi-scale visual Transformer accelerator architecture provided by an embodiment of the present invention is shown in FIG. Figure 1As shown, the accelerator architecture in this embodiment includes: an instruction queue and an instruction scheduler, which are used to receive instructions from a central processing unit (CPU) and distribute them to an IMC array or a direct memory access engine to control calculation or data transmission; a unified cache area, which is used to store input features and output features of convolution, as well as a Query matrix and an output matrix of MHA; an IMC array, which is used to store convolution weights or a Key, Value matrix of MHA, and to perform convolution and MHA calculations; a direct memory access (DMA) engine, which is used for internal storage of the accelerator and data transmission between the accelerator and an off-chip DRAM; a pre-processing module, which is used to read the input features of the convolution or the Query matrix of MHA from the unified buffer, and send the data to the IMC array after aligning the data; a post-processing module, which is used to collect the output results of the IMC array, accumulate them with the partial sums in the unified buffer, and perform corresponding activation functions and quantization operations.

[0025] In this embodiment, the instruction queue and instruction scheduler are responsible for receiving commands from the host CPU and distributing them to the DMA engine or computing array to control them to perform data transmission or calculation; the unified buffer is implemented by SRAM and consists of multiple SRAM banks, which are used to store the input and output features of the convolution, as well as the Query matrix and output matrix of the MHA; the IMC array is composed of multiple IMC engines and a distribution network, which stores the weights of the convolution or the Key and Value matrices of the MHA, and supports flexible convolution and MHA calculations; the DMA engine is responsible for the internal storage of the accelerator, including the unified buffer and the IMC array, and the data transmission between the off-chip DRAM; the pre-processing module is responsible for reading the input features of the convolution or the Query matrix of the MHA from the unified buffer, aligning the data, and sending it to the IMC array to carry out the corresponding calculation; the post-processing module is responsible for collecting the output results of the IMC array, accumulating them with the partial sums in the unified buffer, and executing the corresponding activation function and quantization operation.

[0026] In the above embodiments, in order to reduce the data handling in the calculation process, the present invention adjusts the single-stage softmax operation in the prior art to a three-stage or two-stage softmax operation. Figure 2 The schematic diagram of the calculation operation of MHA, where (a) is a schematic diagram of the existing single-stage softmax operation, the input Query vector passes through QK T Matrix multiplication (QK T-MatMul), then reduced and maximized, then broadcasted, subtraction and fetching calculations are performed, then reduced summation and broadcasting are performed, then division is performed, S'V matrix multiplication (S'V-MatMul), and finally addition is performed to obtain the output vector; (b) is a schematic diagram of a two-stage softmax operation formed by combining scaled addition and division. In practical applications, scaled addition (Scaled Add) and division (Div) can be used as the second-stage softmax operation and the third-stage softmax operation, respectively. Among them, the first stage is performed inside each calculation block to complete the maximum value (Max), subtraction (Sub), exponential (Exp) and summation (Sum) operations. In the second stage, the calculation results from different calculation blocks are scaled and added (Scaled Add), and division (Div) is performed in the third stage. Compared with the single-stage softmax operation, the three-stage or two-stage maximum value and summation operations used in the embodiment of the present invention do not require global communication, but only require local communication inside each calculation block, thereby reducing communication overhead and improving mapping flexibility.

[0027] In the above embodiment, the IMC array includes multiple IMC engines and a distribution network, each IMC engine includes two SRAM-based digital IMC macros (IMC Macro) (IMC0 and IMC1), and the IMC engine can operate in two working modes: a parallel mode for convolution calculation and a pipeline mode for attention calculation. Figure 3 for Figure 1 Each IMC engine in the CNN performs two working modes: (a) is the parallel mode for convolution calculation, and (b) is the pipeline mode for attention calculation. When working in parallel mode, the two IMC macros (IMC0 and IMC1) in the same IMC engine synchronously receive the bit string input and perform calculations. The calculation results are added and then output, reducing the partial sum of the convolution calculation; when working in pipeline mode, the two IMC macros (IMC0 and IMC1) in the same IMC engine form a two-stage pipeline to calculate the two matrix multiplications in MHA respectively, eliminating the intermediate data transfer. The first IMC macro (IMC0) receives the input data and performs the first matrix multiplication (QK T -MatMul) operation, the calculation result is processed by the first stage softmax (softmax-stage1) as the input of the second IMC macro (IMC1), and the second matrix multiplication (S'V-MatMul) operation is performed, and the output of the second IMC macro is used as the output of the IMC engine. Compared with the prior art, the embodiment of the present invention can efficiently support convolution calculation and MHA calculation at the same time.

[0028] In the IMC array in the above embodiment, the distribution network is a reconfigurable distribution network, and a tree-shaped distribution network is used to connect the IMC engines in the IMC array. In actual applications, according to the operator and parallelism allocation requirements, each node of the distribution network can have four working modes: a split-merge mode for convolution, a copy-splice mode for convolution, a split-merge mode for MHA, and a copy-splice mode for MHA. Figure 4 Schematic diagram of four working modes of the reconfigurable distribution network provided in the embodiment of the present invention, (a) Split-Merge mode for convolution: the input is split into two parts and then distributed to the child nodes; the output results of the child nodes are summed and sent to the parent node; (b) Copy-Splice mode for convolution: the input is copied and then distributed to the child nodes; the output results of the child nodes are spliced ​​and sent to the parent node; (c) Split-Merge mode for MHA: the input is split into two parts and then distributed to the child nodes; for the output results of the child nodes, the second-stage softmax operation needs to be performed, the maximum value and exponential calculation are recalculated, and the original output results are scaled and added, and then sent to the parent node; (d) Copy-Splice mode for MHA: the input is copied and then distributed to the child nodes; for the output results of the child nodes, the third-stage softmax operation needs to be performed, that is, the original output results are divided, and then sent to the parent node. By configuring each node of the distribution network to different modes, the convolution or MHA calculation parallelism can be flexibly adjusted to achieve dynamic data flow.

[0029] The embodiment of the present invention introduces pipeline rearrangement to MHA and designs a three-stage softmax function, thereby avoiding the communication overhead of multiple global reductions and broadcasts between computing blocks in the existing computing scheme, and improving the flexibility of mapping; based on pipeline rearrangement, a reconfigurable IMC engine is designed, which efficiently supports convolution and MHA calculations through different modes, eliminates the intermediate data handling overhead for MHA calculations, and is friendly to the hybrid network structure of multi-scale ViT; based on pipeline rearrangement, a reconfigurable distribution network is designed, which efficiently supports the dynamic data flow of convolution and MHA, supports flexible mapping of convolution and MHA of different layer sizes, and effectively improves the space utilization of deploying multi-scale ViT. Compared with the prior art, the present invention realizes the dynamic data flow of both convolution and MHA, can dynamically adjust the parallelism according to different layer sizes to improve the space utilization, and is more suitable for the deployment of multi-scale ViT.

[0030] The present invention also provides an edge chip, including the multi-scale visual Transformer accelerator architecture in the above embodiment. The edge chip can be configured in an edge device with limited resources.

[0031] In the technical solution provided by the present invention, the IMC macro is designed using Cadence Virtuoso, and transistor-level simulation is performed to evaluate power consumption and layout placement to evaluate area; Memory Compiler is used to generate SRAM; CACTI7 is used to simulate the dynamic power consumption of off-chip DDR3-1600 DRAM; the rest of the accelerator is synthesized and evaluated using Synopsys Design Compiler under TSMC 22nm process; and a performance model developed in Python is used to evaluate the overall system delay and energy consumption. The comprehensive results show that the total area of ​​the proposed in-memory computing accelerator is 2.18mm 2 , where the in-memory computation macro and SRAM buffer occupy 57.7% and 27.8% respectively, and the distribution network only occupies 9.3% of the area, which brings less burden to the system.

[0032] Figure 5 A schematic diagram comparing the delay time (NormalizedDelay) and power consumption (NormalizedEnergy) obtained by the accelerator architecture provided by the embodiment of the present invention and the existing architecture is shown in FIG. Figure 5 The acceleration and energy saving effects of the MHA layer for different sequence lengths can be seen in Figure 1. Compared with the accelerator that does not use the proposed optimization method, the present invention achieves up to 2.52 times the speed increase and 3.97 times the energy saving, and the longer the sequence length, the more obvious the advantage.

[0033] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-scale visual Transformer accelerator architecture, characterized in that: include: Instruction queue and instruction scheduler, which are used to receive instructions from the central processing unit and distribute them to the IMC array or direct memory access engine to control calculation or data transmission; Unified buffer area, used to store the input features and output features of convolution, as well as the query matrix and output matrix of MHA; IMC array, used to store convolution weights or MHA Key and Value matrices, and to perform convolution and MHA calculations; Direct memory access engine, used for internal storage of the accelerator and data transfer with the accelerator's off-chip DRAM; The pre-processing module is used to read the input features of the convolution or the query matrix of the MHA from the unified buffer, align the data and send it to the IMC array; The post-processing module is used to collect the output results of the IMC array, accumulate them with the partial sums in the unified buffer, and perform corresponding activation functions and quantization operations.

2. The accelerator architecture according to claim 1, characterized in that: The MHA calculation adopts three stages to calculate the softmax function. The first stage is performed inside each calculation block to complete the maximum value, difference, exponential and sum operations. The second stage scales and adds the calculation results from different calculation blocks, and the third stage performs division.

3. The accelerator architecture according to claim 2, characterized in that: The IMC array includes multiple IMC engines and a distribution network, and each IMC engine includes: two SRAM-based digital IMC macros.

4. The accelerator architecture according to claim 3, characterized in that: The IMC engine has two working modes: parallel mode for convolution calculation and pipeline mode for attention calculation. When working in parallel mode, two IMC macros in the same IMC engine receive input and perform calculation synchronously, and the calculation results are added and output. When working in pipeline mode, the two IMC macros in the same IMC engine form a two-stage pipeline to calculate the two matrix multiplications in MHA respectively. The first IMC macro receives input data and performs the first matrix multiplication operation. The calculation result is processed by the first stage softmax and used as the input of the second IMC macro to perform the second matrix multiplication operation. The output of the second IMC macro is used as the output of the IMC engine.

5. The accelerator architecture according to claim 3, characterized in that: The distribution network is a reconfigurable distribution network, and uses a tree-shaped distribution network to connect various IMC engines in the IMC array.

6. The accelerator architecture according to claim 5, characterized in that: According to the needs of operator and parallelism allocation, each node of the distribution network has four working modes: split-merge mode for convolution, copy-splice mode for convolution, split-merge mode for MHA, and copy-splice mode for MHA.

7. An edge chip, characterized in that: Comprising a multi-scale visual Transformer accelerator architecture as described in any one of claims 1-6.

8. An edge device, characterized in that: Comprising the edge chip as claimed in claim 7.

Images