3D integrated GPU-FPGA heterogeneous system and computing acceleration method

Abstract

The present invention belongs to the field of deep learning technology, and discloses a 3D integrated GPU-FPGA heterogeneous system and a computing acceleration method, wherein the heterogeneous system includes a GPU core, an FPGA core and a packaging substrate, wherein the GPU core is stacked on the FPGA core, and the GPU core and the FPGA core are connected together through silicon vias using copper hybrid bonding, wherein the silicon vias pass through the GPU core and the FPGA core to be vertically electrically connected and connected together in a back-to-back manner, and the FPGA core is connected to the packaging substrate through micro-bumps, and the computing acceleration method is as follows: the FPGA core is configured as a batch normalization computing unit, and data is processed before the data is sent from the GPU core to the main memory controller and loaded from the main memory controller to the GPU core, so as to reduce the number of times the GPU core reads and writes the memory. The present invention can improve the ability of the terminal GPU to process deep learning and scientific computing.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning technology, and specifically relates to a 3D integrated GPU-FPGA heterogeneous system and a computing acceleration method. Background Art

[0002] In recent years, artificial intelligence (AI), autonomous driving, and augmented / virtual reality have continued to develop. The widespread application of these cutting-edge technologies requires chip manufacturers to develop high-bandwidth, high-performance, and low-power GPU devices without significantly increasing production costs. However, due to the slowdown of Moore's Law, traditional two-dimensional silicon wafer expansion has reached its cost limit. In order to solve these problems, academia and industry are also constantly promoting architecture and manufacturing innovations, and core stacking and heterogeneous technologies are two of the most attractive directions of these innovations.

[0003] In a chiplet-based system, the traditional monolithic system-on-chip (SoC) is broken down into multiple smaller chiplets. Each chip contains only part of the functions of the monolithic system-level chiplet, so the development cost is lower, it is easier to reuse in multiple products, and it can be implemented with the most appropriate process technology. At the same time, to meet the growing computing needs, another attractive solution is heterogeneity. Due to the different types of computing and workloads, a single powerful processor can no longer efficiently provide the required computing power for the application.

[0004] Therefore, it is a theoretically feasible solution to integrate GPU and FPGA using chip stacking technology and form a fusion architecture by co-designing two chiplets and their interconnection network, so as to provide high-bandwidth, high-performance, low-cost and low-power GPU devices for terminal devices. However, this solution still faces many problems. On the one hand, the architecture based on chip stacking requires through silicon via (TSV) technology or other related technologies to connect the integrated chiplets, so it is necessary to adopt the chiplet interconnection method and the communication cost of more complex on-chip networks; on the other hand, in the face of a variety of complex applications, how to configure FPGA to help GPU perform data processing is also a challenge. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides a 3D integrated GPU-FPGA heterogeneous system and a method for accelerating data computing based on the system, which can greatly improve the terminal GPU's ability to process deep learning and scientific computing.

[0006] In order to achieve the above object, the present invention is achieved through the following technical solutions:

[0007] The present invention discloses a 3D integrated GPU-FPGA heterogeneous system, which comprises a GPU core, an FPGA core and a packaging substrate, wherein the GPU core is stacked on the FPGA core, the GPU core and the FPGA core are connected together through silicon vias using copper hybrid bonding, the silicon vias pass through the GPU core and the FPGA core to vertically connect the GPU core and the FPGA core, and the GPU core and the FPGA core are connected together in a back-to-back manner, and the FPGA core is connected to the packaging substrate through micro-bumps.

[0008] A further improvement of the present invention is that: the GPU core comprises a plurality of streaming multiprocessors (SMs), each of the streaming multiprocessors (SMs) is divided into four processing blocks, each of the processing blocks comprises a Warp scheduler and dispatch unit, a group of processing elements (PEs) and a register file, all processing blocks in the streaming multiprocessor share a unified data cache (L1D) and a shared memory (SHM), an L1 instruction cache (L1i) and a texture cache (TeX), the unified data cache (L1D) and the shared memory (SHM) enable an adaptive cache mechanism, that is, the device driver configures the shared memory capacity and the L1D capacity on a per-core basis, each main memory controller partition comprises an L2 cache library shared by all streaming multiprocessors, and each L2 cache library is associated with a main memory controller channel.

[0009] A further improvement of the present invention is that the FPGA core is integrated between the L2 cache and the main memory controller, the FPGA core includes a logic unit, an IO unit, an interconnection unit, an on-chip RAM and hundreds of thousands of configurable logic modules (CLBs), the configurable logic module (CLB) includes a small 6-input lookup table (LUT), a D flip-flop and a 2:1 multiplexer, and two adjacent configurable logic modules (CLBs) are interconnected by switch boxes (SBs) and wires placed around the configurable logic modules. By connecting hundreds of thousands of configurable logic modules (CLBs) together, the FPGA core realizes complex logic functions.

[0010] A further improvement of the present invention is that: an on-chip network NoC is designed on the FPGA core particle to connect the GPU core particle, the FPGA core particle and the main memory controller MC, the on-chip network NoC provides a high-bandwidth connection between the L2 cache and the main memory controller MC belonging to the same partition, and the on-chip network NoC includes an outer ring and an inner ring, the outer ring is responsible for transmitting data between the main memory controller MC and the IO unit, and the inner ring is responsible for transmitting data between the FPGA core particle and the GPU core particle.

[0011] The 3D integrated GPU-FPGA heterogeneous system based on the present invention performs computational acceleration processing on data, and the processing method is: the FPGA core is configured as a batch normalization computing unit, and the data is processed before the data is sent from the GPU core to the main memory controller MC and before the data is loaded from the main memory controller MC to the GPU core, thereby reducing the number of times the GPU core reads and writes the memory.

[0012] In the computing acceleration method, two groups of cross switch arrays and arbiters and processing units corresponding to the memory partitions of each main memory controller are configured in the FPGA core, and each processing unit includes a group of G2M and M2G units, which are used for batch normalization calculations in model training during data transmission between the GPU core and the main memory controller.

[0013] When the GPU core writes data back to the main memory controller, the arbitrator arbitrates the data packet. If the current data does not require batch normalization calculation processing, it is directly passed and sent to the main memory controller. Otherwise, according to the instructions in the data packet, the data is transferred to the G2M unit of the corresponding channel. At this stage, the G2M unit calculates the mean of the data. and the inverse of the standard deviation :

[0014]

[0015] In calculating the mean When the G2M unit first traverses all the data, After the accumulation, the result is multiplied by Get the mean , is the amount of data, and the average It is not output directly, but saved in the latch and used when waiting to calculate the standard deviation and wait for the reciprocal of the standard deviation. After the calculation is completed, it is synchronously written into the main memory controller;

[0016] Calculate the inverse of the variance When traversing all the data , calculate each data The square of and then accumulate, and then multiply the result by get , and at the same time the average value in the latch Square it and subtract the accumulated result from get , and finally Calculate the square root and take the reciprocal .

[0017] When the GPU core reads data from the main memory controller, the arbitrator will arbitrate the data packet. If the data needs to be batch normalized, it will enter the M2G unit of the corresponding channel. Otherwise, it will be directly returned to the GPU core. The M2G unit uses the calculated mean for all incoming data. and the inverse of the standard deviation and preconfigured learning variables and Processing is performed to obtain batch normalized data and send it to the GPU core for calculation. During this process, the M2G unit traverses all the data and subtracts the mean in turn. , multiplied by the inverse of the variance , multiplied by and minus ,The M2G unit chooses to perform ReLU or directly output the calculation result according to the ,packet configuration information.

[0018] The beneficial effects of the present invention are:

[0019] The present invention stacks a GPU core on top of an FPGA core, and the two cores are connected together through through-silicon vias (TSVs) using copper hybrid bonding. This tight integration achieves low-latency and high-bandwidth communication, as well as fine-grained resource sharing and interaction between the GPU core and the FPGA core.

[0020] The FPGA of the present invention is integrated between the L2 cache and the main memory controller, which can ensure that the granularity of the sharing of FPGA resources and the memory partition is consistent.

[0021] The on-chip network designed by the present invention provides a dedicated high-bandwidth connection between the L2 cache and the memory controller belonging to the same partition.

[0022] The present invention configures the FPGA as a batch normalization computing unit, which is used to process data before sending data from the GPU to the main memory / loading data from the main memory to the GPU, so as to reduce the number of times the GPU reads / writes the memory. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a structural diagram of the GPU-FPGA heterogeneous system of the present invention.

[0024] Figure 2 It is a structural diagram of the GPU core particle of the present invention.

[0025] Figure 3 It is a structural diagram of the FPGA core particle of the present invention.

[0026] Figure 4 It is a structural diagram of the on-chip network of the present invention.

[0027] Figure 5 It is a schematic diagram of the computing acceleration method of the GPU-FPGA heterogeneous system of the present invention.

[0028] Figure 6 It is a schematic diagram of batch normalization calculation of the present invention.

[0029] Figure 7 It is a schematic diagram of the cache enhancement mode of the present invention.

[0030] Figure 8 It is a schematic diagram of the memory partition processing unit of the present invention. DETAILED DESCRIPTION

[0031] The following will disclose the embodiments of the present invention with drawings. For the purpose of clear description, many practical details will be described together in the following description. However, it should be understood that these practical details should not be used to limit the present invention. That is, in some embodiments of the present invention, these practical details are not necessary. In addition, for the purpose of simplifying the drawings, some conventional structures and components will be depicted in a simple schematic manner in the drawings.

[0032] like Figure 1 As shown, the present invention is a 3D integrated GPU-FPGA heterogeneous system, which includes a GPU core, an FPGA core and a packaging substrate, wherein the GPU core is stacked on the FPGA core, the GPU core and the FPGA core are connected together through silicon vias using copper hybrid bonding, the silicon vias pass through the GPU core and the FPGA core to vertically connect and the GPU core and the FPGA core are connected together in a back-to-back manner, and the FPGA core is connected to the packaging substrate through micro-bumps.

[0033] like Figure 2 As shown, the GPU core comprises a plurality of streaming multiprocessors (SMs), each of which is divided into four processing blocks, each of which comprises a Warp scheduler and dispatch unit, a group of processing elements (PEs) and a register file, all processing blocks in the streaming multiprocessor share a unified data cache (L1D) and a shared memory (SHM), an L1 instruction cache (L1i) and a texture cache (TeX), the unified data cache (L1D) and the shared memory (SHM) enable an adaptive cache mechanism, that is, the device driver configures the shared memory capacity and the L1D capacity on a per-core basis, each main memory controller partition comprises an L2 cache library shared by all streaming multiprocessors, and each L2 cache library is associated with a main memory controller channel.

[0034] The FPGA core is integrated between the L2 cache and the main memory controller, which ensures that the granularity of FPGA resource sharing and memory partitioning is consistent. This is because the vertical interconnection bandwidth provided by 3D stacking technology is much smaller than that of two-dimensional circuits. Integrating the FPGA between the L2 cache and the main memory controller can reduce the requirements for vertical bandwidth to a certain extent. In addition, the operating frequency of FPGAs is usually between 100-200MHz, which is much slower than that of GPUs. Therefore, using FPGAs to perform calculations or access data stored on FPGAs may have a longer latency than performing the same operations on GPUs. In order to reduce the impact of frequency and latency on GPUs, FPGAs are integrated between the L2 cache and the main memory controller, which are farther away from the GPU, rather than between the L1 cache and the L2 cache.

[0035] like Figure 3 As shown, the FPGA core includes logic units, IO units, interconnection units, on-chip RAM and hundreds of thousands of configurable logic modules (CLBs). The configurable logic module (CLB) contains a small 6-input lookup table (LUT), a D flip-flop and a 2:1 multiplexer. Two adjacent configurable logic modules (CLBs) are interconnected by switch boxes (SBs) and wires placed around the configurable logic modules. By connecting hundreds of thousands of configurable logic modules (CLBs) together, the FPGA core realizes complex logic functions.

[0036] like Figure 4 As shown, an on-chip network NoC connecting the GPU core, the FPGA core and the main memory controller MC is designed on the FPGA core. Since the GPU divides the memory into multiple independent storage groups, the on-chip network NoC provides a high-bandwidth connection between the L2 cache and the main memory controller MC belonging to the same partition. The on-chip network NoC includes an outer ring and an inner ring. The outer ring is responsible for transmitting data between the main memory controller MC and the IO unit, and the inner ring is responsible for transmitting data between the FPGA core and the GPU core.

[0037] Based on the 3D integrated GPU-FPGA heterogeneous system provided by the present invention, the FPGA integrated in the L2 cache and the main memory controller can be configured as a cache to increase the cache capacity of the GPU.

[0038] like Figure 7As shown, when performing calculations or DNN / Transformer reasoning, the present invention configures the heterogeneous system into a cache enhancement mode, that is, configuring the FPGA as an L3 cache to increase the GPU cache capacity, reduce cache misses, reduce the number of GPU accesses to the main memory, and improve operating efficiency. In this mode, the L3 cache assumes the function of a victim cache similar to the L2 cache, and accelerates the L2 cache to read data from the main memory controller MC in a bypass working mode. For the L3 cache, it only retains the cache data evicted from the L2 cache, and does not pre-fetch data or read data due to read misses (Read Miss) and write misses (Write Miss).

[0039] Specifically, when a write miss occurs in the L2 cache, it does not send the write request directly to the main memory controller MC, but sends it to the L3 cache and the main memory controller MC at the same time. If the request hits in the L3 cache, the corresponding data in the L3 cache is modified, otherwise it is ignored directly. When the L2 cache evicts data, the data is also sent to the L3 cache and MC at the same time. Since all L2 write operations are simultaneous operations on the MC and L3 cache, the consistency of the data in the L3 cache can be ensured, so that when the L3 data is overwritten, it does not need to be written back to the main memory. When the L2 read miss occurs, the request is also sent to the L3 cache and MC at the same time. At this time, the load request will first be registered in the LRT (Load Register Table). If the L3 read hits, the data will be returned, otherwise the L3 cache will directly ignore the request. This may result in two situations: the L3 cache returns the data first and the main memory returns the data first. When the LRT receives the data write back, it will check whether there is a corresponding registered load request in the table. If so, it will be passed to L2 and the current table entry will be cancelled, otherwise the data will be ignored and returned directly. The significance of this operation is that if L3 hits, the data can be returned to the L2 cache more quickly, improving data transmission efficiency. At the same time, since the request is transmitted to the MC and L3 cache in parallel in a Bypass manner, even if L3 misses, it only needs to wait for the data from the main memory to be returned, which will not cause excessive additional delay.

[0040] like Figure 5 and Figure 8 As shown, when training a DNN or Transformer model, the heterogeneous system of the present invention is relied upon to process data, and the processing method is as follows: the FPGA core is configured as a batch normalization computing unit, and the data is processed before the data is sent from the GPU core to the main memory controller MC and before the data is loaded from the main memory controller MC to the GPU core, so as to reduce the number of times the GPU core reads and writes the memory.

[0041] Batch normalization is a technique widely used to train deep learning models. Batch normalization standardizes the input data of each batch, which helps stabilize the model training process and reduce the time required for convergence. Figure 6 As shown in the figure, batch normalization has four main steps: 1) For each channel, calculate the mean of all elements; 2) For each channel, calculate the variance of all elements; 3) Use the mean and variance to normalize the feature map; 4) Scale and offset the feature map. Among them, although the mean and variance can be calculated simultaneously, it is still inevitable to traverse the data in sequence, so there must be a data read and write; then use the mean and variance to normalize the input and use two trainable parameters and Scaling and offsetting each input element can be combined, but it is also necessary to move data between memory and device again; after batch normalization, there is usually activation for each output element generated by the batch normalization layer, which will lead to another round of data movement. Therefore, in a pure GPU training environment, the calculation of batch normalization and activation layers will lead to frequent data movement between GPU and main memory, including 3 times from main memory to GPU and 3 times from GPU to main memory.

[0042] In the GPU-FPGA heterogeneous system of the present invention, the present invention can use FPGA to effectively reduce the data movement in this process. In this computing acceleration method, the present invention configures two groups of crossbar switch arrays and arbiters in the FPGA, as well as processing units corresponding to each memory partition, each processing unit includes a group of G2M and M2G units, which are used for batch normalization calculations in model training during the data transmission process between GPU and main memory.

[0043] When the GPU core writes data back to the main memory controller, the arbitrator arbitrates the data packet. If the current data does not require batch normalization calculation processing, it is directly passed and sent to the main memory controller. Otherwise, according to the instructions in the data packet, the data is passed to the G2M unit of the corresponding channel. Each input channel has its own mean , , learning variables and , attached Figure 6 H represents the feature map height, W represents the feature map width. At this stage, the G2M unit calculates the mean of the data. and the inverse of the standard deviation :

[0044]

[0045] In calculating the mean When the G2M unit first traverses all the data, After the accumulation, the result is multiplied by Get the mean , is the amount of data, and the average It is not output directly, but saved in the latch and used when waiting to calculate the standard deviation and wait for the reciprocal of the standard deviation. After the calculation is completed, it is synchronously written into the main memory controller; It is the final result of standardized output.

[0046] Calculate the inverse of the variance When , we also traverse all the data and calculate each data The square of and then accumulate, and then multiply the result by get , and at the same time the average value in the latch Square it and subtract the accumulated result from get , and finally Calculate the square root and take the reciprocal .

[0047] When the GPU core reads data from the main memory controller, the arbitrator will arbitrate the data packet. If the data needs to be batch normalized, it will enter the M2G unit of the corresponding channel. Otherwise, it will be directly returned to the GPU core. The M2G unit uses the calculated mean for all incoming data. and the inverse of the standard deviation and preconfigured learning variables and Processing is performed to obtain batch normalized data and send it to the GPU core for calculation. During this process, the M2G unit traverses all the data and subtracts the mean in turn. , multiplied by the inverse of the variance , multiplied by and minus ,The M2G unit chooses to perform ReLU or directly output the calculation result according to the ,packet configuration information.

[0048] In order to verify the effectiveness of the GPU-FPGA heterogeneous system of the present invention, the present invention uses a modified GPGPU-SIM to simulate and verify the system. GPGPU-SIM is a cycle-level GPU simulator that can run CUDA or OpenCL code. The FPGA part of the heterogeneous system is based on the XCVU125-2FLVB1760E Xilinx FPGA. The present invention first uses Verilog HDL to implement the proposed L3 cache and batch normalization computing unit in the XCVU125-2FLVB1760E Xilinx FPGA, obtains relevant hardware parameters, and imports them into GPGPU-SIM for system simulation.

[0049] In cache enhancement mode, the present invention uses DNNMark_conv, Polybench_fdtd2d, Lonestar_mst and Lonestar_sssp as test cases. The present invention provides a GPU-FPGA heterogeneous system test configuration based on the NVIDIA Jetson AGX ORIN 64GB hardware parameters. Using the number of cycles of the original GPU running the test case as a benchmark, the system based on NVIDIA Jetson AGX ORIN 64 GB improves by 3.71% to 13.91% compared to the original GPU. The comparison of GPU and GPU-FPGA execution cycles in memory enhancement mode is shown in Table 1.

[0050] Table 1

[0051]

[0052] In the mode of processing data under the heterogeneous system, the present invention uses two different sizes of input data as test cases based on DNNMark. The present invention provides two GPU-FPGA architecture test configurations based on the hardware parameters of NVIDIA Jetson AGX ORIN 64 GB and NVIDIAGeForce RTX 3070. Here, the number of cycles of the original GPU running the test case is also used as a benchmark. The system based on NVIDIA Jetson AGX ORIN 64 GB is 27.02% higher than the original GPU in the size of 224×224×3, and 69.96% higher than the original GPU in the size of 28×28×256. The system based on NVIDIAGeForce RTX 3070 is 25.73% higher than the original GPU in the size of 224×224×3, and 25.51% higher than the original GPU in the size of 28×28×256. The comparison of the execution cycle of GPU and GPU-FPGA in the computing enhancement mode is shown in Table 2.

[0053] Table 2

[0054]

[0055] In summary, the present invention can improve the ability of terminal GPU to process deep learning and scientific computing.

[0056] The above description is only an embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A 3D integrated GPU-FPGA heterogeneous system, the GPU-FPGA heterogeneous system comprising a GPU core, an FPGA core and a packaging substrate, characterized in that: The GPU core is stacked on the FPGA core, and the GPU core and the FPGA core are connected together through silicon vias using copper hybrid bonding. The silicon vias pass through the GPU core and the FPGA core to vertically connect and the GPU core and the FPGA core are connected together in a back-to-back manner. The FPGA core is connected to the packaging substrate through micro-bumps. The GPU core includes multiple streaming multiprocessors, each of which is divided into four processing blocks, each of which includes a Warp scheduler and dispatch unit, a group of processing elements and a register file. All processing blocks in the streaming multiprocessor share a unified data cache and shared memory, an L1 instruction cache and a texture cache. The unified data cache and the shared memory enable an adaptive cache mechanism. Each main memory controller partition includes an L2 cache library shared by all streaming multiprocessors, and each L2 cache library is associated with a main memory controller channel. The FPGA core is integrated between the L2 cache and the main memory controller. The FPGA core includes a logic unit, an IO unit, an interconnect unit, an on-chip RAM and hundreds of thousands of configurable logic modules. The configurable logic module includes a small 6-input lookup table, a D flip-flop and a 2:1 multiplexer. Two adjacent configurable logic modules are interconnected by switch boxes and wires placed around the configurable logic modules. By connecting hundreds of thousands of configurable logic modules together, the FPGA core implements complex logic functions. An on-chip network NoC is designed on the FPGA core to connect the GPU core, the FPGA core and the main memory controller MC. The on-chip network NoC provides high-bandwidth connection between the L2 cache and the main memory controller MC belonging to the same partition. The on-chip network NoC includes an outer ring and an inner ring. The outer ring is responsible for transmitting data between the main memory controller MC and the IO unit, and the inner ring is responsible for transmitting data between the FPGA core and the GPU core.

2. A computing acceleration method using the 3D integrated GPU-FPGA heterogeneous system of claim 1, characterized in that: The computing acceleration method is as follows: the FPGA core is configured as a batch normalization computing unit, and data is processed before the data is sent from the GPU core to the main memory controller MC and before the data is loaded from the main memory controller MC to the GPU core, so as to reduce the number of times the GPU core reads and writes the memory.

3. The computing acceleration method according to claim 2, characterized in that: In the computing acceleration method, two groups of cross switch arrays and arbiters and processing units corresponding to the memory partitions of each main memory controller are configured in the FPGA core, and each processing unit includes a group of G2M and M2G units, which are used for batch normalization calculations in model training during data transmission between the GPU core and the main memory controller.

4. The computing acceleration method according to claim 3, characterized in that: When the GPU core writes data back to the main memory controller, the arbitrator arbitrates the data packet. If the current data does not require batch normalization calculation processing, it is directly passed and sent to the main memory controller. Otherwise, according to the instructions in the data packet, the data is passed to the G2M unit of the corresponding channel. At this stage, the G2M unit calculates the mean μ and the inverse of the standard deviation of the data. When calculating the mean μ, the G2M unit first traverses all the data and calculates the mean μ for all the data x. i Accumulate, multiply the result after accumulation by 1 / M to get the mean μ, where M is the amount of data. The mean μ is not output directly, but saved in the latch and used when waiting to calculate the standard deviation, and wait for the reciprocal of the standard deviation After the calculation is completed, it is synchronously written into the main memory controller; Calculate the inverse of the variance When , we also traverse all the data x i , calculate each data x i The square of the sum is added, and the result after the addition is multiplied by 1 / M to get At the same time, the mean value μ in the latch is squared and μ is subtracted from the accumulated result. 2 Get σ 2 , and finally 2 Calculate the square root and take the reciprocal 5. The computing acceleration method according to claim 3, characterized in that: When the GPU core reads data from the main memory controller, the arbitrator will arbitrate the data packet. If the data needs to be batch normalized, it will enter the M2G unit of the corresponding channel. Otherwise, it will be directly returned to the GPU core. The M2G unit uses the calculated mean μ and the inverse of the standard deviation for all incoming data. And the pre-configured learning variables γ and β are processed to obtain batch normalized data and sent to the GPU core for calculation. In this process, the M2G unit traverses all the data and subtracts the mean μ in turn, multiplies it by the inverse of the variance After multiplying by γ and subtracting β, the M2G unit chooses to perform ReLU on the calculation result or output it directly according to the data packet configuration information.

Images