Accelerator architecture for near IO pipeline computing, and ai acceleration system

Abstract

The present application relates to the field of accelerators, and discloses an accelerator architecture for near IO pipeline computing, and an AI acceleration system. A multi-channel direct access module comprises N DRAM controllers, and the N DRAM controllers are connected to DRAMs in a one-to-one correspondence; each DRAM controller is at least connected to k DMA controllers, and is connected to a corresponding core group cluster by means of the DMA controllers; a pipeline synchronization ring is connected to N core group clusters and comprises M cascaded forward transmission blocks and M cascaded backward transmission blocks, and the head-end forward transmission block and the tail-end forward transmission block are respectively connected to a data receiving module and a data transmitting module; the output of the tail-end forward transmission block is cascaded to the first backward transmission block; the ith forward transmission block and the (M-i)th backward transmission block correspond to each other in a front-rear direction, and are jointly connected to at least one computing core group. The accelerator of the architecture replaces a traditional multi-level cache structure, reduces the time delay caused multi-layer search, and accelerates the computing rate between the interior of the accelerator and the accelerator.

Description

Accelerator architecture and AI acceleration system for near-IO pipelined computing TECHNICAL FIELD

[0001] Embodiments of the present application relate to the field of accelerators, in particular to an accelerator architecture and AI acceleration system for near-IO pipelined computing. BACKGROUND

[0002] In current AI acceleration GPU, SOC, ASIC designs, the relationship between the computing core and the storage is implemented around the typical architecture shown in FIG. 1. In specific designs, there will be some fine-tuning, and the number of cache layers may be different. For example, in the Ampere architecture of the A100 high-end graphics card, the original L0 layer is merged into the L1 layer. The L1 cache in this architecture is KB level, and its cache capacity can be up to 100 KB, such as 192 KB for the A100 graphics card, 256 KB for the H100 graphics card, etc. The L2 cache is MB level, and its cache capacity can be up to several tens of MB, such as 50 MB for the A100 graphics card and the H100 graphics card, etc. For DRAM, its cache capacity is usually GB level. Due to the speed requirement, the HBM and GDDR solutions are more commonly used, and the capacity varies from several tens of GB to several hundreds of GB, for example, the B200 recently launched by NVDIA has a capacity of 192 GB (HBM). TECHNICAL PROBLEM

[0003] In related technologies, the parameters weights of AI computing are usually initialized in DRAM in the computer, and need to be imported to the computing core for operation during calculation. As can be seen from the architecture disclosed in FIG. 1, the computing core imports parameters through multiple layers of cache, and the access sequence is to query the nearest level cache first. If it is not hit (miss), the next level cache is queried, until the data is hit and returned. When the data is returned, some designs can be directly returned to the computing core, and some need to be synchronized to the next level cache first, and then the computing core can be obtained after synchronization to L0 / L1. Therefore, if the L0 / L1 cache is not hit, the parameter reading delay will be very long. The above is a problem caused by one-time parameter reading, which will be executed repeatedly in large model AI calculation, resulting in cumulative delay. The insufficient cache capacity causes the cache to be unable to hold, and further causes the repeated reading of the upper layer cache and even the DRAM, which will amplify the storage access bottleneck problem. TECHNICAL SOLUTION

[0004] Embodiments of the present application provide an accelerator architecture and AI acceleration system for near-IO pipelined computing, which solves the problem of delay caused by the computing core group in the accelerator and the acceleration system based on multi-level cache reading data.

[0005] The application discloses an accelerator architecture of near-IO stream computing, and the accelerator comprises a multi-channel direct access module, M computing core groups and a stream synchronization ring.

[0006] The multi-channel direct access module comprises N DRAM controllers, and the N DRAM controllers are connected with external DRAM in one-to-one correspondence; each DRAM controller is connected with at least K direct access storage DMA controllers, and is connected to a corresponding core group through the DMA controller; data in the i th DRAM is directly read to the i th computing core group, and result data output by the i th computing core group is directly cached to the i th DRAM.

[0007] The stream synchronization ring is connected with the N core groups, and comprises M cascaded forward transmission blocks and M cascaded backward transmission blocks; the output of the tail end forward transmission block is cascaded with the first backward transmission block.

[0008] The i th forward transmission block and the M-i+1 th backward transmission block correspond to each other in front and back, and are connected with at least one computing core group; data is sent to a target computing core group in a stream form for calculation, and data output by the computing core group is sent to the stream synchronization ring; wherein M, N, K, i and j are positive integers.

[0009] Specifically, the multi-channel direct access module further comprises a Switch arbitrator, and the N DRAM controllers and the N core groups are connected with ports of the Switch arbitrator through ports; data in the i th DRAM is cross-read to the j th core group, and result data output by the j th core group is cross-cached to the i th DRAM; wherein i and j are different positive integers.

[0010] Specifically, the K DMA controllers and the computing cores in the computing core group are in one-to-one connection, one-to-many connection or many-to-one connection.

[0011] Specifically, the stream synchronization ring is further connected with a data sending module and a data receiving module; the data sending module receives data output from the stream synchronization ring and sends the data out of the accelerator, and the data receiving module receives data sent from outside, sends the data into the stream synchronization ring and pushes the data to a target computing core in the core group.

[0012] Specifically, the backward transmission block comprises a backward register and P backward write units cascaded in front and back; P is a positive integer.

[0013] The backward register is connected with the output end of the previous stage of the transmission block, receives and registers the intermediate data; one output of the backward register is connected with the first backward write unit, and the data is pushed to the target backward write unit in a pipeline manner through the cascade structure; and the other output is directly connected with the next stage of the transmission block, and the intermediate data is pushed to the target transmission block in a pipeline manner.

[0014] Specifically, the first write register in each backward write unit in the same backward transmission block is connected in cascade before and after the write register, the data is input through the backward register, and the data is pushed to different forward write units in a pipeline manner.

[0015] The backward write unit comprises at least Q write registers connected in cascade before and after each other, and each write register is connected with a calculation core group, and the data is pushed to the target calculation core group in the forward write unit in a pipeline manner.

[0016] Specifically, the forward transmission block comprises a forward FIFO buffer, a forward register, P forward receiving units and forward write units; P is a positive integer.

[0017] The P forward receiving units are connected in cascade before and after each other, the P forward write units are connected in cascade before and after each other, and the forward write units and the forward receiving units in the same group are connected with the same calculation core group.

[0018] One input end of the forward FIFO buffer is connected with the output of the previous stage of the transmission block, receives and registers the intermediate data, and the other input end is connected with the output of the tail end of the forward receiving unit, receives the data output from the tail end of the forward receiving unit.

[0019] The output of the forward FIFO buffer is connected with the forward register, and the output of the forward register is connected with the first forward write unit, and the pipeline data is pushed to the target forward write unit through the cascade structure.

[0020] Specifically, the forward write unit comprises at least Q write registers connected in cascade before and after each other, and each write register is connected with a calculation core group, and the data is pushed to the target calculation core group in the forward write unit in a pipeline manner.

[0021] The forward receiving unit comprises at least Q receiving FIFO buffers connected in cascade before and after each other, and each receiving FIFO buffer is connected with a calculation core group, buffers the intermediate data input by the calculation core group, and sends the intermediate data to the pipeline synchronization ring in a pipeline manner.

[0022] Specifically, the first write register in each forward write unit in the same forward transmission block is connected in cascade before and after the write register, the data is input through the forward register, and the data is pushed to different forward write units in a pipeline manner.

[0023] The tail-end receiving FIFO buffer in each forward receiving unit in the same forward transmission block is cascaded before and after, and the intermediate data input by different calculation core groups are sent into the forward FIFO buffer in a pipelined manner.

[0024] Specifically, in the i-th forward transmission block and the M-i+1-th backward transmission block, the structure of the forward writing unit and the backward writing unit in the same group is the same.

[0025] In the same group of forward receiving units and forward writing units in the same forward transmission block, the number of registers and FIFO buffers is the same.

[0026] Specifically, the data pushed into the pipelined synchronization ring is provided with a bit identifier, and the forward transmission block and the backward transmission block determine the state and flow direction of the data based on the corresponding flag bit of the bit identifier.

[0027] When the flag bit of the bit identifier indicates that the data is exhausted, the data is directly discarded.

[0028] When the flag bit of the bit identifier indicates that the data is not exhausted, the data is sent into the writing register layer by layer until the target calculation core group is pushed.

[0029] On the other hand, the application provides an AI acceleration system, which comprises at least two accelerators and a data exchange circuit, the at least two accelerators are respectively connected with a plurality of DRAMs, and the accelerators are internally provided with a data sending module and a data receiving module connected with the pipelined synchronization ring, and the data sending module and the data receiving module are respectively connected with the data exchange circuit.

[0030] The calculation core group in the accelerator is used to perform at least one of matrix operation and vector operation, the sending-end accelerator performs a data acceleration task and sends matrix data and / or vector data to the data exchange circuit through the data sending module, and the receiving-end accelerator receives the matrix data and / or the vector data through the data receiving module and performs the data acceleration task. Advantages

[0031] The technical scheme provided by the embodiments of the present application has at least the following beneficial effects: a pipeline delivery structure is formed by cascading a forward transmission block and a backward transmission block, the forward transmission block is divided into a cascaded forward receiving unit and a forward writing unit according to input and output; each forward receiving unit is connected to a computing core group by cascading a plurality of FIFOs, and each forward writing unit is connected to a computing core group by cascading a plurality of registers, so as to realize high-concurrency data input and output; the first register of adjacent writing units in each transmission block is cascaded, and the tail-end FIFO of adjacent receiving units is cascaded, so as to realize pipeline pushing of input and output data between the writing units and the receiving units, and pipeline jumping pushing in the writing units and the receiving units, and realize fast data transmission.

[0032] Inside the accelerator, the computing core group forms a core group cluster, which can directly access the external storage medium through multi-channel DMA control, accelerate data extraction and transmission efficiency, read data in the DRAM to the computing core group, or receive data output by the computing core group and write back to the DRAM; for data interaction across the core group cluster and the DRAM, a Switch arbitrator is set to realize global DRAM and core group cluster data cross-connection. The accelerator discards the multi-layer cache design mode, reduces the overall cost, and transmits data to the specified computing core with the smallest delay, thereby accelerating the data processing speed of the accelerator. BRIEF DESCRIPTION OF DRAWINGS

[0033] FIG. 1 is a multi-level cache computing core group storage acceleration architecture disclosed in the related art;

[0034] FIG. 2 is a hardware deployment schematic diagram based on a flash attention scheme in the related art;

[0035] FIG. 3 shows a synchronous data flow schematic diagram based on a current AI accelerator chip;

[0036] FIG. 4 is a structure schematic diagram of an accelerator architecture for near-IO pipeline computing provided by the embodiments of the present application;

[0037] FIG. 5 is a structure schematic diagram of a pipeline synchronization ring;

[0038] FIG. 6 is a structure schematic diagram of a backward transmission block in the pipeline synchronization ring;

[0039] FIG. 7 is a structure schematic diagram of a forward transmission block in the pipeline synchronization ring;

[0040] FIG. 8 is a structure schematic diagram of an AI acceleration system provided by the present application;

[0041] FIG. 9 is a structure schematic diagram of an AI acceleration system for long-sequence linear calculation;

[0042] FIG. 10 is an AI accelerator designed based on the flash attention calculation principle. Embodiments of the present application

[0043] For the purpose, technical solutions and advantages of the present application to be clearer, the embodiments of the present application will be described in further detail below with reference to the drawings.

[0044] “Multiple” mentioned herein refers to two or more. “And / or” describes the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B can represent the following three cases: A exists alone, A and B exist together, and B exists alone. The character “ / ” generally represents that the associated objects before and after it are in an “or” relationship.

[0045] FIG. 1 is a multi-level cache computing core group storage acceleration architecture disclosed in the related art. For the computing core group part of the architecture, different product designs produced and designed by different companies are different, and usually contain one computing core or multiple parallel computing cores. The computing core usually supports pipelined computing, and when multiple computing cores are included, the computing functions of different computing cores can be different, such as including computing cores of different precisions at the same time, including matrix computing cores (often MAC arrays), vector computing cores, etc. Taking A100 and H100 architectures as examples, one L1 cache corresponds to one streaming multiprocessor SM, which includes four computing core groups, and each computing core group contains multiple streaming processors SP, which include different numbers of FP32 pipelined computing cores, INT32 pipelined computing cores, FP64 pipelined computing cores, and NVDIA's own Tensor core. The Tensor core in A100 is the third generation, and the Tensor core in H100 is the fourth generation. Taking the architecture disclosed in FIG. 1 as an example, in some designs, the L0 layer cache is merged into the core group as a shared memory, and the L1 and upper layers are located outside the core group. Several core groups share one L1 cache for parameter access, and multiple L1 caches and upper L2 caches perform parameter access. Multiple L2 caches are finally connected to the storage medium of the storage model and data parameters, which can be HBM, DDR, LPDDR or GDDR DRAM, etc.

[0046] From the above architecture, it can be seen that when designing current AI computing chips, the requirements of convolutional neural networks (CNN) are often considered. The parameters are composed of convolution kernels. After a convolution kernel is imported, it will slide on the data to be calculated and perform convolution operation. A convolution operation can be understood as a multiply-add operation (MAC operation) for a parameter. A convolution kernel in CNN will participate in multiple operations, and the calculation density of a unit parameter is high. For example, in resnet50, a convolution kernel in the first layer convolves the 224x224 input to 112x112 size. The convolution kernel is 7x7, the step is 2, and the convolution calculation times are 112x112=12544. It can be considered that each parameter reads the result after 12544 times (MAC calculation). After the CNN parameter is read, more calculations will occur. If the import of the next set of parameters is performed during the calculation delay, the delay caused by multiple cache layers can be hidden, and the next calculation can start as soon as possible.

[0047] However, when the success of GPT drives more and more applications of large models based on the Transformer architecture, the computing requirements have changed. For the Transformer, the parameters are mainly large matrices, and the most involved calculation can be abstracted as a vector multiplied by a matrix equal to a vector:

[0048]

[0049] where x is an m-dimensional vector, w is an m*n matrix, and y is an n-dimensional vector. For the elements of y, (k∈[1,n]) has:

[0050]

[0051] As can be seen from the formula, each parameter data only participates in one multiply-add operation, and the calculation density is low. The computing core can easily meet this performance requirement. At the same time, the model parameters under the Transformer architecture are large. For example, the smallest model of LLaMa2 at the present stage is 7B, i.e. the parameter quantity is about 7 billion, and the storage capacity of FP32 parameters is about 28 GB. Even if it is quantized to int8, it still needs 7 GB. The above factors cause the running performance of the Transformer architecture model on the AI accelerator to be directly related to the DRAM capacity and the speed (throughput) and latency of the direct transfer of parameters from DRAM to the computing core. The cache of the current architecture AI accelerator cannot provide sufficient capacity, and the computing core needs to directly read DRAM to obtain parameters. At this time, the delay caused by the multi-layer cache architecture seriously affects the performance of the computing core. Although multi-batch computing (multi-task parallelism) brings certain improvement in the utilization of computing resources, the performance loss caused by the insufficient utilization of physical bandwidth has not been alleviated.

[0052] Cache capacity shortage also brings more problems. For example, the optimization of softmax in GPU, the pseudo code written to prevent overflow function at least contains 3 times loop, the use of loop needs to occupy the cache of the intermediate results, especially in the long dimension model such as Transformer (softmax may reach ten K level according to the support token length, and the minimum is K level). At the same time, for the multi-core parallel operation of GPU, the data synchronization between cores also increases the delay overhead outside the algorithm. In view of the above defects, many optimized codes in the CNN era use online softmax algorithm to optimize the implementation. Combined with the calculation formula of attention layer, the exp can also be split into flash attention scheme to further reduce the loop.

[0053] Figure 2 is a schematic diagram of hardware deployment based on the flash attention scheme in the related art. The shaded part in figure 2 is the data that needs to be stored in the L0 / L1 cache closest to the calculation core group. The additional boxed part of the shaded part identifies the valid data (for example, in A, the maximum cache amount of the intermediate result is the shaded part, and the final accumulated result is the boxed part in it). D is the dimension of the head (usually small, such as 128 for LLaMa2), and B corresponds to the parallelism b, which can be manually set. L is the sequence length, which may be very long (such as 16K), but according to the current calculation method, the entire L sequence length matrix does not need to be placed in the L0 / L1 cache, only the DxB size needs to be considered. Each tile is a pipeline, and the cache is small, which greatly solves the problem of parameter amplification. This calculation method has certain limitations:

[0054] First, the current assumption is that D is small. This method is to better run the original attention calculation with L equal to D, which is divided into the current D (D=L / n_heads, n_heads is the number of heads in multi-head attention). Moreover, with the evolution of the model, it cannot be confirmed whether D will continue to grow, which may become a limitation to the evolution direction of the algorithm. The control factor of parallelism B is the size of the cache, but usually this parameter should consider how many parallelism the calculation core has. For example, given a hardware, the cache size is fixed, and the BxD value is fixed. Different models have different D, and different B can be set, but the parallelism of the calculation core of the given hardware is fixed, which may lead to mismatch of parallelism and serious reduction of computing resource utilization.

[0055] In the generation phase of Inference, single batch calculation is essentially vector multiplication matrix, and Q is a vector. In the design of AI computing core, systolic array is mostly used for matrix multiplication matrix. In order to improve resource utilization, multi-batch operation is mostly carried out. When KT and V slide with the sequence, Q needs to be stored completely, and in multi-batch, batch Q needs to be stored. At this time, due to the limitation of cache size, the number of batches is also limited, which further limits the performance of the computing resource utilization. The above two problems have caused the performance of the computing core with high computing density to be severely limited. At the same time, flash attention only solves the problem of amplification of multi-layer cache access delay, and does not improve the problem itself. The above KT and V still need to access DRAM through multi-layer cache mechanism.

[0056] Based on the above architecture, the AI acceleration device device also has device delay between multiple devices. Because before the Transformer, most AI calculations are only used for multi-device joint calculation during training, and inference is mostly run on a single device. In the Transformer model, multiple devices are often used for joint inference calculation. For example, llama2-70B model needs two or four A100 / H100 to get better performance.

[0057] Figure 3 shows a synchronous data flow diagram based on the current AI accelerator chip. The computing core group in the two devices synchronizes data, and the data in device 1 needs to be transmitted to DRAM through a multi-layer cache mechanism (two layers are taken as an example in Figure 3), and then transmitted to the target device (device 2) by the synchronization subsystem. The data in the target device DRAM also needs to go through a multi-layer cache mechanism to reach the target computing core group. In this way, without considering the inter-chip transmission and the delay of DRAM itself, the multi-layer delay between the computing core group and the DRAM is usually 100 clock cycles or more, which has a great impact on efficiency. At the same time, if global data exchange is performed within the device, similar delay will also affect the computing efficiency.

[0058] Figure 4 is a structure diagram of an accelerator architecture for near-IO pipelined computing according to an embodiment of the present application. The AI accelerator (AI acceleration Device) in the architecture is internally provided with a multi-channel direct access module, M computing core groups and a pipelined synchronization ring. The M computing core groups are divided into N core groups containing the same or different number of core groups. The design of the core group is mainly to concentrate several computing core groups, which can realize larger integration and higher scale of acceleration tasks.

[0059] The multi-channel direct access module includes N DRAM controllers, which are connected one-to-one with external DRAMs. The N DRAM controllers are set according to actual needs and are connected with a certain number of storage devices, so as to realize data reading and writing operations of the DRAM. In order to accelerate data transmission efficiency and reduce the control dependence on the computing core group, at least K direct access storage DMA controllers can be connected to each DRAM controller, and the DMA controller is connected to the corresponding core group. Thus, data in the ith DRAM can be directly read to the ith computing core group, and the result data output by the ith computing core group can be directly cached to the ith DRAM.

[0060] Each DRAM controller corresponds to multiple DMA access channels and is directly connected to the corresponding core group, so that fast direct access can be realized, and the delay caused by the multi-layer cache mechanism can be eliminated. In addition, the delay of switch arbitration and bus forwarding can be eliminated, and the physical performance of the DRAM can be maximized. Because the calling of the ordinary access port needs to continuously call the resources of the computing core group for control.

[0061] It should be noted that the controller type in the multi-channel direct access module can be determined according to the design of the AI acceleration chip. Currently, the mainstream preferred HBM cannot be excluded due to process, procurement line, historical technology, and other reasons. GDDR, DDR, LPDDR, and other controllers can be used. In addition, in the present application, the whole core group is directly connected with the DMA channel, and the number and scale of the computing core groups in the core group corresponding to different DRAMs can be designed according to the design ideas and target scenarios of different designers. The relationship between the DMA channel and these computing core groups can be defined as one-to-one, one-to-many, many-to-one, etc. according to the needs. In principle, many-to-many is not encouraged and supported, which will reduce the access efficiency.

[0062] In a possible implementation, a one-to-one design can be used, that is, the number of DMAs and the number of computing core groups in the core group are consistent, and the same number of ordinary access ports need to be set for the DMA controller to connect with all the computing core groups, which is mainly for data exchange and execution of acceleration tasks across the core group and across the DRAM device. Therefore, the present application also sets a switch arbitrator in the multi-channel direct access module.

[0063] Switch arbitrator is connected with N DRAM controllers and N core groups respectively through common access ports, the purpose is to cross read data in the i th DRAM to the j th core group, and cross cache result data output by the j th core group to the i th DRAM. Where i and j are different positive integers. Each DRAM controller corresponds to a common access channel, and the common channel between controllers is realized by a switch to realize unified access, and each computing core group is provided with an access port. This design provides a global access channel for computing core, and the speed of this channel is relatively slow compared with DMA channel, but the delay of global data exchange is still better than that through multi-layer cache mechanism.

[0064] In addition, in some embodiments, the accelerator can also set a data sending module and a data receiving module for effective data exchange with other peripheral devices. The data sending module and the data receiving module are connected to the pipeline synchronization ring respectively. After the data calculated by the accelerator is pushed through the pipeline synchronization ring, the result can be sent from the data sending module to the outside of the accelerator. Similarly, external data can be received by the data receiving module and sent to the pipeline synchronization ring, and then pushed to the target computing core group in the core group.

[0065] Figure 5 is a structural diagram of a pipeline synchronization ring. The pipeline synchronization ring is connected with N core groups, and internally contains M cascaded forward transmission blocks Forward step and M cascaded backward transmission blocks Backward step. The forward transmission block is used for forward pushing data transmission, and the backward transmission block is used for backward pushing data transmission. In particular, the head-end forward transmission block and the tail-end forward transmission block are connected with the data receiving module Receiver and the data sending module Sender respectively. Of course, the Receiver and the Sender can be exchanged, or the data sending and receiving modules can be set at the head-end and tail-end of the cascaded backward transmission blocks, which does not affect the overall structure. The forward and backward transmission blocks are in a cascaded structure, that is, the output of the tail-end forward transmission block is cascaded with the first backward transmission block. In Figure 5, Forward step 1 is connected with Receiver, and is connected with Forward step M in turn in the forward direction, and Forward step M is connected with Sender and then outputs. Meanwhile, Forward step M is further cascaded with Backward step 1, and is connected with Backward step M in turn in the backward direction. Note that the forward and backward transmission blocks do not form a complete ring; M, N, K, i, j are positive integers.

[0066] The ith forward transmission block and the M-i+1 backward transmission block of the pipeline synchronization ring correspond to each other and are connected to at least one computing core group. That is, Forward step 1 and Backward step M correspond to each other in FIG. 5, Forward step M and Backward step 1 correspond to each other, and so on. Each Forward step i and Backward step M-i+1 is a group, which should be connected to the same computing core group. The number of computing core groups connected in each group can be freely set. For example, if N computing core groups are evenly divided into M groups, then each group has N / M computing core groups.

[0067] The design structure aims to send data to the target computing core group in a pipelined manner for calculation, and send the data output by the computing core group to the pipeline synchronization ring. Under the action of the forward and backward transmission blocks, the data sent from any computing core group can be pushed to any other computing core group in a pipelined manner through the transmission blocks, and the special structure of the forward and backward cascading folding reciprocation can realize that the computing core group 1 is pushed to the computing core group N step by step, and then the data is pushed back to the computing core group 1 from the computing core group N in reverse. Similarly, the Receiver and the Sender can realize the exchange of data between accelerators, receive external data for push calculation, and send the result data or intermediate data of the pipelined calculation, which is also the core embodiment of the "synchronization ring" of the present application.

[0068] FIG. 6 is a structural schematic diagram of a backward transmission block in the pipeline synchronization ring. The Backward step includes a backward register Backward register and P backward write units write step cascaded in front and back; wherein P is a positive integer. Three backward write units are shown in FIG. 6 for convenience of description.

[0069] The Backward register is connected to the output end of the previous cascaded transmission block to receive and store intermediate data. One output of the Backward register is connected to the first backward write unit write step 1, and the data is pushed to the target backward write unit in a pipelined manner through the cascading structure. The other output is directly connected to the next cascaded transmission block to push the intermediate data to the target transmission block in a pipelined manner. Assuming that the intermediate data needs to be sent to the computing core group 4 in the write step 2, it needs to be sent to the write step 1 first, and then transmitted to the write step 2.

[0070] The first write register in each Backward write in the same Backward step is cascaded before and after, inputs data through the Backward register, and pushes the data in a pipelined manner to different Backward write steps. The Backward write step includes at least Q cascaded write registers, each write register is connected to a computing core group, and data is pushed to the target computing core group in the forward write unit in a pipelined manner. The speciality of this structure is that the continuous storage of the register will occupy the clock period and cause delay, so the first write register 1 in each Backward write step is cascaded, assuming that the intermediate data is to be pushed to the computing core group 8 in the write step 3, the data is pushed in a pipelined manner, that is, Backward register→(Backward write step 1)write register 1→(Backward write step 2)write register 1→(Backward write step 3)write register 1, and then write register 1→write register 2→computing core group 8 in the layer. It is particularly pointed out that the number of cascaded registers (computing core groups) in each write step can be the same or different, which is specifically set according to actual needs.

[0071] Figure 7 is a schematic diagram of the structure of a forward transmission block in a pipelined synchronization ring, the Forward step includes a forward FIFO cache Forward fifo, a forward register Forward register, P groups of forward receiving units Forward receive step and forward write units Forward write step; P is a positive integer.

[0072] P Forward receive steps are cascaded in front and back, P Forward write steps are cascaded in front and back, and the Forward write steps and the Forward receive steps in the same group are connected to the same group of computing cores. Because one is for sending and the other is for data receiving, the data flow is opposite, and for the convenience of description, the numbering is described in the pipeline timing, that is, the Forward write step i and the Forward receive step P-i+1 are a group, corresponding to the Forward write step 1 and the Forward receive step 3 in FIG. 7, and the others are the same.

[0073] One input of the Forward fifo is connected to the output of the front cascaded transmission block to receive and store intermediate data, and the other input is connected to the output of the tail end Forward receive step P to receive data output from the Pth Forward receive step. The output of the Forward fifo is connected to the Forward register, and the output of the Forward register is connected to the first Forward write step 1, and the data in the pipeline is pushed to the target Forward write step through the cascaded structure.

[0074] The Forward write step includes at least Q write registers cascaded in front and back, and each write register is connected to a group of computing cores, and the purpose is to push the (intermediate) data to the target group of computing cores in the Forward write step in the form of a pipeline.

[0075] The Forward receive step includes at least Q Switch fifos cascaded in front and back, and each Switch fifo is connected to a group of computing cores, buffers the intermediate data input by the group of computing cores, and sends the intermediate data to the pipeline synchronization ring layer by layer in the form of a pipeline.

[0076] In some embodiments, the Forward step and Backward step can contain multiple layers of Backward write step and Forward receive step. For example, in FIG. 7, the write register 1 in the Forward write step 1 is not directly connected to the compute core group 1, but is connected to the Forward write step 11, and the Forward write step 11 further contains Q write registers, each of which is connected to a compute core group, thus forming a multi-layer nested structure, and the last layer is connected to the compute core group, and the other parts are the same.

[0077] In the pipeline synchronization ring provided in the present application, the first write register write register 1 in each Forward write step in the same Forward step is cascaded before and after, inputs data through the Forward register, and pushes the data to different forward write units in a pipeline manner. This way can realize the forward advancement of the way of pipeline jump according to the final destination, reduce the data transmission process, and quickly push the (intermediate) data to the target forward write unit, and then sequentially transmit the input target compute core group in the unit.

[0078] Similarly, for the structure of the input data, the tail end Forward receive step Q in each Forward receive step in the same Forward step is cascaded before and after, and the intermediate data input by different compute core groups is sent to the Forward fifo in a pipeline manner. This way can realize the forward advancement of the way of pipeline jump, and reduce the FIFO cache transmission process.

[0079] In particular, because the Backward step i and the Forward step M-i+1 are the same group of transport blocks, they maintain the same compute core group, so in the ith forward transport block and the M-i+1 backward transport block, the structure of the forward write unit and the backward write unit in the same group is the same. That is, it contains the same number and structure of write registers, and the number of registers and FIFO buffers in the same group of forward receive units and forward write units in the same forward transport block is also the same.

[0080] The above transmission block structure, the forward and backward writing units only store one data, so it is full pipelining, and the register can be used to complete the layer-by-layer transmission. The forward receiving unit has Qos control, and may buffer an uncertain number of multiple data. Therefore, when transmitting write data at each clock cycle, a register + judgment logic is used to achieve it. Each cycle adopts a fifo structure buffer when receiving the calculation core group data to the transmission main pipeline, mainly to solve the conflict of two data sources. In the specific design, attention should be paid that the closer to the main pipeline, the higher the upper limit of the data bandwidth should be designed. At the same time, for the data sent by the calculation core group, multiple receive paths in the pipeline synchronization ring may arrive at the same time, so fifo is used for buffering. In some applications, such as data expected data flow conflict is more serious, QoS restriction can be added where necessary.

[0081] Assuming that the pipeline synchronization ring is considered in layers, the uppermost layer is the structure of Forward step 1- Forward step M, and Backward step 1- Backward step M, when Forward step and Backward step are only single-layer register structure (and FIFO buffer) inside, the second layer is a single Forward step and Backward step structure, and the third layer is a calculation core group. Figures 5-7 are described by taking three layers as an example, when Forward step and Backward step contain multiple layers of register structure, the pipeline level of the whole synchronization ring increases accordingly, assuming that the number of pipeline layers is sync_layers, and the number of pipeline steps in each layer is sync_steps (that is, the Q value in the foregoing embodiment, the number in different units can be different, figures 6-7 describe sync_layers=3, sync_steps=3) the maximum number of calculation core groups supported is:

[0082]

[0083] And the minimum delay and the maximum delay in the device are:

[0084]

[0085]

[0086] Taking A100 graphics card as an example, the calculation core group SM has 108, at this time, 3 layers and 5 levels of pipelining are adopted, and at most 125 calculation core groups can be supported, the minimum synchronization delay is 3 cycles, and the maximum is 15 cycles. Compared with the minimum synchronization cycle of 100 of the multi-level cache mechanism, it is close to the hit of L1 cache of about 10 cycles, or even faster.

[0087] For the above pipelined data pushing rule of the pipelined synchronization ring, the bit identifier is used to determine whether to send to the next accelerator device and to determine which computing core group to receive in the data pushing process. Specifically, the bit identifier is set for the data pushed into the pipelined synchronization ring, and the forward transmission block and the backward transmission block determine the state and flow direction of the data based on the corresponding flag bit of the bit identifier. Before pushing forward and backward one step each time, the direction of the data and whether the data is exhausted are judged according to the flag bit of the bit identifier. If the data is exhausted, the data is discarded; if the data is not exhausted, the transmission direction is judged and the data is pushed. Each step is divided into multiple layers of pipelining, which is to cope with the case that the computing core group is large, to push the data with the shortest delay, and to shorten the length of the bit identifier of the computing core group.

[0088] In summary, the application uses the cascaded forward transmission block and the cascaded backward transmission block to form a pipelined transmission structure. The forward transmission block is divided into cascaded forward receiving units and forward writing units according to the input and output. Each forward receiving unit uses a multi-stage FIFO cascade and accesses the computing core group, and each forward writing unit uses a multi-stage register cascade and accesses the computing core group, realizing high-concurrency data input and output. The first register of adjacent writing units in each transmission block is cascaded, and the tail-end FIFO of adjacent receiving units is cascaded, so that the pipelined pushing of input and output data between writing units and receiving units, and the pipelined jumping pushing inside the writing units and the receiving units can be realized, realizing fast data transmission.

[0089] Inside the accelerator, the computing core group can directly access the external storage medium through multi-channel DMA control, accelerating data extraction and transmission efficiency, reading the data in the DRAM to the computing core group, or receiving the data output by the computing core group and writing back to the DRAM. For data interaction across the core group and across the DRAM, a switch arbitrator is set to realize the data cross-connection between the global DRAM and the core group. The accelerator discards the multi-layer cache design mode, reduces the overall cost, and transmits data to the specified computing core with the smallest delay, accelerating the data processing speed of the accelerator.

[0090] Figure 8 is a structural schematic diagram of an AI acceleration system provided by the application, which includes at least two accelerators disclosed above and a data exchange circuit. The at least two accelerators are respectively connected with a plurality of DRAMs, and the data sending module and the data receiving module of the accelerator are respectively connected with the data exchange circuit.

[0091] The computing core group in the accelerator is used to perform at least one of a matrix operation and a vector operation, the sending end accelerator performs a data acceleration task, and sends matrix data and / or vector data to a data exchange circuit through a data sending module, and the receiving end accelerator receives the matrix data and / or vector data through a data receiving module, and performs the data acceleration task.

[0092] Taking a Transformer (GPT, LLaMa, Gemini, etc. are transformer architecture models) as an example to illustrate the working principle of the architecture and system, the main calculation of the Transformer is concentrated in attention calculation (attention) and linear layer calculation (linear).

[0093] 1. For long sequence linear calculation

[0094] For a sequence calculation O(k) = A(n)W(n, k), there are different parallel computing methods according to the external topology, the number of AI acceleration devices (hereinafter referred to as devices) in the system, and the different sizes of vectors and matrices.

[0095] For the scene of complete parallel division of A, assuming that the system has 4 devices, each device has a computing core group, then (linear parallel acceleration 1) is:

[0096]

[0097]

[0098]

[0099]

[0100]

[0101] In this design, the matrix multiplication is generally completed by the MAC array, that is, the computing core group in the core group is a MAC array that completes the matrix operation, and each of the 4 devices extracts data through a DMA channel, and performs data transmission after flow synchronization ring execution flow operation. The 4 devices are connected to an external exchange circuit or switch at the same time, and the purpose is to package and send data. This system is mainly used for multiple devices to cooperatively execute model acceleration tasks.

[0102] Of course, for the scene of flow topology or A sequence division, assuming that the system has 2 devices, each device has two computing core groups, then (linear parallel acceleration 2) is:

[0103]

[0104]

[0105]

[0106]

[0107]

[0108] The system architecture corresponding to this scheme is shown in Figure 9. Since the vector addition operation is completed by the vector computing core (computation core group), and since there are two devices, the switch or switching circuit can be directly omitted. The sender of one device can be directly connected to the receiver of another device to realize fast data interaction between devices.

[0109] 2. Regarding attention calculation

[0110] for The algorithm can adopt the computational principle of flash attention. When deploying based on this architecture, further optimizations can be made to the data flow. Figure 10 shows an AI accelerator designed based on the computational principle of flash attention, which divides the core group into two computational core groups. Each computational core group contains both matrix operations and vector operations. One of the computational core groups contains a MAC array for execution. The Vector computation core performs the calculations of m and d, while another set of computation cores, containing a MAC array, performs the matrix multiplication V, and the Vector computation core performs the calculation of o. Because the execution... The algorithm requires only two computational kernels. To further simplify the system structure, other structures are omitted; only the inputs and outputs of the two computational kernels are simultaneously connected to a pipelined synchronization loop to achieve attention computation. The main characteristics of this structure are as follows:

[0111] The parameters are calculated directly from the MAC array without going through a multi-layer caching mechanism;

[0112] Data between computing cores is transmitted through a pipeline synchronization system (consisting of a multi-layer pipeline synchronization ring, sender, receiver, etc., for single or multiple devices), enabling cache-free pipeline transmission.

[0113] It can support larger computing cores (higher computing density), and the overall number of computing cores that need to be synchronously interacted is reduced;

[0114] The parallelism setting is based on the computing core resources, without needing to consider cache size, and can achieve optimal performance.

[0115] In summary, the system architecture applied in the AI model calculation scene has the following beneficial effects:

[0116] 1) In the AI calculation scene with large model parameters, especially in the multi-device collaborative operation, the data synchronization delay can be effectively reduced when the global data exchange within the device, thereby improving the overall performance;

[0117] 2) The cache requirement of the architecture is small, which can effectively reduce the device area on the basis of maintaining the computing power and DRAM storage capacity, thereby improving the yield or increasing the computing power density, and achieving the goal of saving cost;

[0118] 3) The architecture is friendly to scale expansion, and the performance is nearly linearly expanded with the increase of the number of devices inside the system. The cost performance ratio of cluster deployment is strong, especially when the demand for large models is becoming more and more widespread, which can reduce the TCO of data center;

[0119] 4) The architecture has strong adaptability, and has less restrictions on the design of computing core group and external connection method, which can better play the existing technical advantages of different designers.

[0120] The preferred embodiments of the present application are described above; it should be understood that the present application is not limited to the above specific embodiments, and the devices and structures not fully described should be understood as being implemented in the ordinary way in the art; any person skilled in the art can make many possible changes and modifications, or modify equivalent embodiments without departing from the technical scheme of the present application, which does not affect the essential content of the present application; therefore, any simple modification, equivalent change and modification made to the above embodiments according to the technical essence of the present application, which does not depart from the technical scheme of the present application, is still within the scope of protection of the present application.

Claims

1. An accelerator architecture for near-IO streaming computation, the accelerator architecture comprising: The accelerator comprises a multi-channel direct access module, M groups of computing cores and a pipeline synchronization ring; the M groups of computing cores are divided into N groups of core groups containing the same or different number of cores; The multi-channel direct access module comprises N DRAM controllers, the N DRAM controllers are connected to the external DRAM one by one, each DRAM controller is connected to at least K direct access storage DMA controllers, and the DMA controller is connected to the corresponding core group to directly read the data in the i-th DRAM to the i-th computing core group and directly cache the result data output by the i-th computing core group to the i-th DRAM; The pipeline synchronization ring is connected to the N core groups and comprises M cascaded forward transmission blocks and M cascaded backward transmission blocks, the output of the tail end forward transmission block is cascaded with the first backward transmission block; The i-th forward transmission block and the M-i+1-th backward transmission block correspond to each other and are connected to at least one computing core, data is sent to the target computing core in a pipeline form for calculation, and the data output by the computing core is sent to the pipeline synchronization ring; wherein M, N, K, i and j are positive integers.

2. The accelerator architecture for near-IO streaming computation of claim 1, wherein, The multi-channel direct access module further comprises a Switch arbitrator, the N DRAM controllers and the N core groups are connected to the ports of the Switch arbitrator through the ports, the data in the i-th DRAM is cross-read to the j-th core group, and the result data output by the j-th core group is cross-cached to the i-th DRAM; wherein i and j are different positive integers.

3. The accelerator architecture for near-IO streaming computation of claim 2, wherein, The K DMA controllers are connected to the computing cores in the computing core group in a one-to-one connection, one-to-many connection or many-to-one connection relationship.

4. The accelerator architecture for near-IO streaming computation of claim 3, wherein, The data sending module and the data receiving module connected to the pipeline synchronization ring are further provided; the data sending module receives the data output from the pipeline synchronization ring and sends the data out of the accelerator, and the data receiving module receives the data sent from the outside, sends the data into the pipeline synchronization ring and pushes the data to the target computing core in the core group.

5. The accelerator architecture for near-IO streaming computation of claim 1, wherein, The backward transmission block comprises a backward register and P backward write units cascaded in front and back; P is a positive integer; The backward register is connected to the output end of the previous cascaded transmission block to receive and store intermediate data; One output of the backward register is connected to the first backward write unit to push the data to the target backward write unit in a pipeline form through the cascaded structure, and the other output is directly connected to the next cascaded transmission block to push the intermediate data to the target transmission block in a pipeline form.

6. The accelerator architecture for near-IO streaming computation of claim 5, wherein, The first write register in each backward write unit in the same backward transmission block is cascaded in front and back, receives data through the backward register and pushes the data to different forward write units in a pipeline form; The backward write unit comprises at least Q write registers cascaded in front and back, each write register is connected to a computing core to push the data to the target computing core in the forward write unit in a pipeline form.

7. The accelerator architecture for near-IO streaming computation of claim 1, wherein, The forward transmission block comprises a forward FIFO cache, a forward register, P groups of forward receiving units and forward write units; P is a positive integer; P forward receiving units are cascaded in front and back, P forward writing units are cascaded in front and back, and the forward writing units and the forward receiving units in the same group are connected to the same calculation core group; One input of the forward FIFO cache is connected to the output of the previous cascaded transmission block to receive and store intermediate data, and the other input is connected to the output of the tail forward receiving unit to receive data output from the tail forward receiving unit; The output of the forward FIFO cache is connected to the forward register, and the output of the forward register is connected to the first forward writing unit to push the flow data to the target calculation core group in the forward writing unit through the cascaded structure.

8. The accelerator architecture for near-IO streaming computation of claim 7, wherein, The forward writing unit contains at least Q writing registers cascaded in front and back, and each writing register is connected to a calculation core group to push data to the target calculation core group in the forward writing unit in a flow manner. The forward receiving unit contains at least Q receiving FIFO caches cascaded in front and back, and each receiving FIFO cache is connected to a calculation core group to buffer the intermediate data input by the calculation core group and send the intermediate data to the flow synchronization ring in a flow manner.

9. The accelerator architecture for near-IO streaming computation of claim 8, wherein, The first writing register in each forward writing unit in the same forward transmission block is cascaded in front and back, inputs data through the forward register, and pushes the data to different forward writing units in a flow manner. The tail receiving FIFO cache in each forward receiving unit in the same forward transmission block is cascaded in front and back, and the intermediate data input by different calculation core groups are sent to the forward FIFO cache in a flow manner.

10. The accelerator architecture for near-IO streaming computation of claim 8, wherein, In the ith forward transmission block and the M-i+1 backward transmission block, the forward writing unit and the backward writing unit in the same group have the same structure. In the same group of forward receiving units and forward writing units in the same forward transmission block, the number of registers and FIFO caches is the same.

11. The accelerator architecture for near-IO streaming computation of claim 8, wherein, The data pushed into the flow synchronization ring is provided with a bit identifier, and the forward transmission block and the backward transmission block determine the state and flow direction of the data based on the corresponding flag bit of the bit identifier. When the flag bit of the bit identifier indicates that the data is exhausted, the data is directly discarded. When the flag bit of the bit identifier indicates that the data is not exhausted, the data is sent to the writing register layer by layer until it is pushed into the target calculation core group.

12. An AI acceleration system, comprising: The system contains at least two accelerators and data exchange circuits as claimed in claim 1, at least two accelerators are respectively connected to a plurality of DRAMs, and the accelerators are internally provided with data sending modules and data receiving modules connected to the flow synchronization ring, and the data sending modules and the data receiving modules are respectively connected to the data exchange circuit; The calculation core group in the accelerator is used to perform at least one of matrix operation and vector operation, the sending end accelerator performs data acceleration task and sends matrix data and / or vector data to the data exchange circuit through the data sending module, and the receiving end accelerator receives matrix data and / or vector data through the data receiving module to perform data acceleration task.

Images