A Dynamically Reconfigurable Pulsating Array Accelerator for Multi-DNN

Abstract

This invention discloses a dynamically reconfigurable systolic array accelerator for Multi-DNN, comprising: a CPU, a DDR control module, a DDR module, a cross-switch matrix module, and a computing system module. The computing system module includes several cores and several routers; each core independently runs a DNN load; each core corresponds to one router; cores dynamically combine to form an on-chip network; routers are electrically connected to form a toroidal topology; the CPU communicates with the computing system module via an AXI bus to control the dynamic combination of cores within the computing system module; the DDR module is used for data storage; the DDR control module receives signals from the CPU, initiates read / write commands to the DDR module, and sends data from the DDR module to the cross-switch matrix module; the cross-switch matrix module is used for data communication between the DDR control module and the cores in the computing system module. This invention improves the acceleration performance of the accelerator under multiple different DNN loads.

Description

Technical Field

[0001] This invention relates to the field of deep neural network hardware accelerator technology, and in particular to a dynamically reconfigurable systolic array accelerator for Multi-DNN. Background Technology

[0002] Over the past, deep neural networks have significantly improved the performance of various artificial intelligence systems, such as image processing, speech recognition, and autonomous driving. Deep Neural Networks (DNNs) are therefore considered a key technology for new applications in embedded systems, mobile devices, and the cloud. And regardless of whether they are embedded systems, mobile devices, or the cloud, the systems they deploy are increasingly integrating multiple DNNs.

[0003] However, current mainstream designs primarily focus on the design of a single DNN hardware accelerator. For applications with multiple DNNs, acceleration can only be achieved by time-sharing a single DNN hardware accelerator. But different DNN workloads often exhibit extremely high heterogeneity. This extreme heterogeneity makes traditional DNN hardware accelerators perform poorly and difficult to run multiple DNN workloads simultaneously. Summary of the Invention

[0004] To address the issue of poor acceleration performance in existing hardware accelerators when running multiple DNN loads, a dynamically reconfigurable systolic array accelerator for Multi-DNN is proposed, which improves the acceleration performance under multiple different DNN loads.

[0005] The technical solution of the present invention is as follows:

[0006] A dynamically reconfigurable pulsating array accelerator for Multi-DNN includes: a CPU, a DDR control module, a DDR module, a cross-switch matrix module, and a computing system module.

[0007] The computing system module includes several cores and several routers; each core independently runs a DNN load; each core corresponds to one router; the cores are dynamically combined to form an on-chip network; the routers are connected to form a toroidal topology.

[0008] The CPU communicates with the computing system module via the AXI bus, controlling the dynamic combination of cores within the computing system module.

[0009] The DDR module is used for data storage.

[0010] The DDR control module is used to receive signals from the CPU, initiate read and write commands to the DDR module, and send the data in the DDR module to the cross switch matrix module.

[0011] The cross switch matrix module is used for data communication between the DDR control module and the core in the computing system module.

[0012] Preferably, the toroidal topology includes:

[0013] The routers are electrically connected via a bidirectional bus, and each router is electrically connected to several adjacent routers in different directions. Routers located at the boundary are connected to adjacent router modules located at the boundary in a ring configuration. Routers in the same row at the boundary are connected, and routers in the same column at the boundary are also connected.

[0014] Preferably, each core includes a systolic array, an input feature cache block, a preload cache block, a partial cache block, a control unit, a first selector, and a second selector.

[0015] The first selector is used to receive the feature data output by the router corresponding to the core, and the feature data of the core output from the cross switch matrix module to the computing system module.

[0016] The input feature buffer block is used to receive feature data output by the first selector.

[0017] The aforementioned portion and buffer block are used to receive portions and data output by the pulsating array.

[0018] The preloaded cache block is used to receive weight data or partial sum data of the core output from the cross switch matrix module to the computing system module.

[0019] The second selector is used to receive partial data output from the router corresponding to the core, weight data or partial data output from the preloaded cache block.

[0020] The pulse array is used to receive feature data output from the input feature buffer block, weight data or partial sum data output from the preloaded buffer block, and partial sum data output from the second selector, perform data processing, and output the data to the partial sum buffer block.

[0021] The control unit is used to receive information transmitted from the CPU to the computing system module, control the read and write behavior of the input feature cache block, the preloaded cache block, the partial cache block, and the selection behavior of the first selector and the second selector.

[0022] Preferably, each core of the computing system module runs a DNN load independently.

[0023] Furthermore, the crossbar switch matrix module enables one-to-many master-slave communication for the cores in the computing system module.

[0024] Furthermore, the CPU configures the DDR control module via the AXI bus. After the DDR control module reads the data in the DDR module, it transmits it to the core of the computing system module through the crossbar switch matrix module.

[0025] Furthermore, the CPU configures the computing system module via the AXI bus, and configures several routers according to the characteristics of the DNN load, so that the cores can be combined to form an on-chip network adapted to the DNN load characteristics for data processing.

[0026] Preferably, the cores transmit parts and data from top to bottom through a router, thereby achieving vertical combination between cores.

[0027] Preferably, the feature data is transmitted from left to right between the cores via a router, thereby achieving horizontal combination between the cores.

[0028] Furthermore, the input feature cache block adopts a true dual-port RAM; both ports of the true dual-port RAM are used for read and write data operations.

[0029] The preloaded cache block, the portion, and the cache block all use pseudo-dual-port RAM; the first port of the pseudo-dual-port RAM is used only for write data operations, and the second port of the pseudo-dual-port RAM is used only for read data operations.

[0030] The beneficial effects of this invention are:

[0031] 1. By dynamically combining cores to form an on-chip network and using routers to form a toroidal topology, communication is achieved with lower latency, thus improving bandwidth utilization.

[0032] 2. Cores can be dynamically combined, supporting any number of cores. The maximum number depends on the total number of cores. When the number of running DNN loads is less than the number of cores, different cores can be combined into corresponding shapes for calculation based on the DNN load characteristics. Attached Figure Description

[0033] Figure 1 The overall framework diagram of a dynamically reconfigurable systolic array accelerator for Multi-DNN provided for an embodiment.

[0034] Figure 2 This is a diagram of the internal structure of a router provided for an embodiment.

[0035] Figure 3This is a dynamic combination diagram of the cores in different directions provided for an embodiment.

[0036] Figure 4 The diagram shows the dynamic combination of non-adjacent cores provided for the embodiment.

[0037] Figure 5 The internal structure diagram of the core provided for the embodiment.

[0038] Figure 6 This is a dynamic combination diagram of different cores performing tasks, provided in the embodiments.

[0039] Figure 7 This is a diagram of the internal structure of the pulsating array provided for an embodiment. Detailed Implementation

[0040] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0041] Example 1

[0042] In this embodiment, as Figure 1 As shown, a dynamically reconfigurable pulsating array accelerator for Multi-DNN includes: a CPU, a DDR control module, a DDR module, a cross-switch matrix module, and a computing system module.

[0043] The computing system module includes several cores and several routers; each core independently runs a DNN load; each core corresponds to one router; the cores are dynamically combined to form an on-chip network; the routers are connected to form a toroidal topology.

[0044] The CPU communicates with the computing system module via the AXI bus, controlling the dynamic combination of cores within the computing system module.

[0045] The DDR module is used for data storage.

[0046] The DDR control module is used to receive signals from the CPU, initiate read and write commands to the DDR module, and send the data in the DDR module to the cross switch matrix module.

[0047] The cross switch matrix module is used for data communication between the DDR control module and the core in the computing system module.

[0048] In this embodiment, the toroidal topology includes:

[0049] The routers are electrically connected via a bidirectional bus, and each router is electrically connected to several adjacent routers in different directions. Routers located at the boundary are connected to adjacent router modules located at the boundary in a ring configuration. Routers in the same row at the boundary are connected, and routers in the same column at the boundary are also connected.

[0050] The DDR control module uses a double-rate synchronous dynamic random access controller, and the DDR module uses a double-rate synchronous dynamic random access memory.

[0051] Each core includes a systolic array, an input feature cache block, a preload cache block, a partial cache block, a control unit, a first selector, and a second selector.

[0052] The first selector is used to receive the feature data output by the router corresponding to the core, and the feature data of the core output from the cross switch matrix module to the computing system module.

[0053] The input feature buffer block is used to receive feature data output by the first selector.

[0054] The aforementioned portion and buffer block are used to receive portions and data output by the pulsating array.

[0055] The preloaded cache block is used to receive weight data or partial sum data of the core output from the cross switch matrix module to the computing system module.

[0056] The second selector is used to receive partial data output from the router corresponding to the core, weight data or partial data output from the preloaded cache block.

[0057] The pulse array is used to receive feature data output from the input feature buffer block, weight data or partial sum data output from the preloaded buffer block, and partial sum data output from the second selector, perform data processing, and output the data to the partial sum buffer block.

[0058] The control unit is used to receive information transmitted from the CPU to the computing system module, control the read and write behavior of the input feature cache block, the preloaded cache block, the partial cache block, and the selection behavior of the first selector and the second selector.

[0059] Before calculation begins, the weight data needs to be preloaded into the control unit within the systolic array. Only after loading is complete is the partial data transmitted. The weight data refers to the weight values ​​in the neural network. These weight values ​​are stored in the DDR (Memory Memory) and need to be sent from the DDR to the preloaded cache block via the cross-switch matrix module, and then to the control unit within the systolic array.

[0060] In this embodiment, the cross switch matrix module enables one-to-many master-slave communication for the cores in the computing system module.

[0061] More specifically, the CPU configures the DDR control module via the AXI bus. After reading the data from the DDR module, the DDR control module transmits it to the core of the computing system module via a crossbar switch matrix module. The data for the core's calculations is in the DDR module; this data needs to be sent to the core before the core can begin calculations.

[0062] More specifically, the CPU configures the computing system module via the AXI bus, configures several routers according to the characteristics of the DNN load, and combines the cores to form an on-chip network adapted to the DNN load characteristics for data processing.

[0063] In this embodiment, the cores transmit parts and data from top to bottom through a router, thereby achieving vertical combination between cores.

[0064] In this embodiment, the feature data is transmitted from left to right between the cores via a router, thereby achieving horizontal combination between the cores.

[0065] More specifically, the aforementioned portions and data include:

[0066] The result data generated by a kernel during computation, if it needs to be added to other result data later, is called partial sum data. This partial sum data needs to be passed to other kernels, and added to the partial sums generated by those other kernels to obtain the final result.

[0067] More specifically, the feature data includes:

[0068] A portion of the input data required for the kernel to perform calculations.

[0069] The direction of data transmission depends on the specific data type being transmitted. When the transmitted data is a feature array, it is a horizontal splicing; when the transmitted data is a partial array, it is a vertical splicing. The specific type of data being transmitted can be configured by the CPU to control the router, thus determining which directional combination to use.

[0070] In this embodiment, the input feature cache block is a true dual-port RAM; both ports of the true dual-port RAM are used for read and write data operations.

[0071] The preloaded cache block, the portion, and the cache block all use pseudo-dual-port RAM; the first port of the pseudo-dual-port RAM is used only for write data operations, and the second port of the pseudo-dual-port RAM is used only for read data operations.

[0072] Example 2

[0073] A dynamically reconfigurable pulsating array accelerator for Multi-DNN includes: a CPU, a DDR control module, a DDR module, a cross-switch matrix module, and a computing system module.

[0074] The first port of the CPU is electrically connected to the first port of the DDR control module.

[0075] The second port of the CPU is electrically connected to the first port of the computing system module.

[0076] The second port of the DDR control module is electrically connected to the DDR module.

[0077] The third port of the DDR control module is electrically connected to the first port of the cross switch matrix module.

[0078] The second port of the cross switch matrix module is electrically connected to the second port of the computing system module.

[0079] The computing system module includes several cores and several routers; each core corresponds to at least one router.

[0080] The cores are dynamically combined to form an on-chip network; any one of the cores is electrically connected to the second port of the cross switch matrix module as the second port of the computing system module.

[0081] The electrical connections between the routers constitute a toroidal topology.

[0082] This accelerator system uses 16 cores to process DNN workloads, with each core capable of independently running a DNN workload. A cross-connect matrix module is used to implement one-to-many master-slave communication, meaning each core can communicate with the DDR module through the cross-connect matrix module. Figure 1 The number 3 in the middle corresponds to the data path marked by the line. This data path is bidirectional, meaning the crossbar switch matrix module can both receive data from and send data to the cores. The crossbar switch matrix module is 1×16 in size. DDR stands for Double Data Rate SDRAM, used for storing large amounts of data. The CPU module can configure the DDR control module via the AXI bus and initiate read / write operations on the DDR. Data is then transmitted to the corresponding core via the crossbar switch matrix module. Furthermore, the CPU communicates with the computing system module via the AXI bus, enabling the CPU to control the dynamic combination of cores. Each core is equipped with a corresponding router. A core can send data to the router, which then transmits it to other cores. It can also receive data from its corresponding router. Figure 1 The data path is identified by the number 2. Routers are connected via a bidirectional bus, and each router connects to routers in its four directions: above, below, left, and right. Figure 1 The data path is identified by the number 1 in the diagram; for routers at the boundaries, a ring connection is used to connect them to routers at another boundary, thus forming a ring topology for the on-chip network. For example... Figure 1 The router in the top left corner has its left input channel coming from the output of the rightmost router in that row, while its top input channel comes from the output of the bottom router in that column.

[0083] When the system processes multiple DNN workloads, the CPU first configures the DDR controller via the AXI bus, enabling the DDR controller to read data from the DDR memory and transfer it to the corresponding memory blocks in the cores via the crossbar switch matrix module. Simultaneously, the CPU configures multiple routers based on the characteristics of different DNN workloads, allowing cores to be combined to adapt to and process the specific DNN workload. After completing the corresponding DNN workload task, the CPU configures the accelerator via the AXI bus and retrieves the computation results from the corresponding cores via the crossbar switch matrix module, writing them back to the DDR memory, thus completing the processing of multiple DNN workloads.

[0084] Example 3

[0085] In this embodiment, as Figure 2 As shown, this router receives data from its corresponding core and from routers in its up, down, left, and right directions within the on-chip network. Depending on the CPU configuration, data from the corresponding input direction can be output on different output channels, thus enabling flexible and dynamic combinations between cores. Taking the characteristic data output by the router to the left router as an example, it is generated according to the CPU configuration. Figure 2 The selection signal of the selector, indicated by the dashed line, enables the selector to select the corresponding input data from the feature data output from the core and the feature data output from the routers above, below, and to the right, and then output the corresponding input data.

[0086] In this embodiment, as Figure 3 As shown, dynamic combination of cores can be achieved through routers and on-chip networks composed of routers. Figure 3 The system hides a router. Vertical input and output data paths represent the transmission of partial data, while horizontal input and output data paths represent the transmission of feature data. When partial data is transmitted between cores, it's a vertical combination, equivalent to increasing the core's height while keeping its width unchanged. When feature data is transmitted between cores, it's a horizontal combination, equivalent to increasing the core's width while keeping its height unchanged. Figure 3In this configuration, cores _0_0, _1_0, _2_0, and _3_0 transmit partial data from top to bottom via a router, thus achieving vertical combination between cores. Meanwhile, cores _2_2, _2_3, _3_2, and _3_3 transmit feature data from right to left via a router, achieving horizontal combination between cores. Through flexible on-chip network design and corresponding router design, the combination shape between cores is not limited by the original on-chip network topology. For example, in... Figure 3 In the CPU configuration, cores 2_1 and 3_1 are positioned vertically, but they can still be combined horizontally depending on the CPU configuration.

[0087] In this embodiment, as Figure 4 As shown, non-adjacent cores can also be combined. Core 2 _1 and Core 1 _2 are not adjacent, but data can be transmitted to Core 1 _2 via a router after passing through Core 2 _2, thus achieving a combination of Core 2 _1 and Core 1 _2. Figure 4 The data path is indicated by the straight line in the middle. It's important to note that this on-chip network provides a large number of cabling channels; therefore, theoretically, data transmission can also occur from other cabling channels as long as the corresponding cabling channel is not occupied. Figure 4 Core 2 can also transmit data from core 1 to core 2 after passing through core 1, that is... Figure 4 The data path is indicated by the dashed line.

[0088] In this embodiment, as Figure 5 As shown, the external feature data is the feature data received by the core from the cross-connect matrix module; the router's feature output is the feature data output to the core from the router corresponding to the core; the external weight / partial sum data is the weight or partial sum data received by the core from the cross-connect matrix module; the router's partial sum output is the partial sum data output to the core from the corresponding router; and the partial sum output is the partial sum data output by the core to its corresponding router. The core's internal structure consists of a control unit, an input feature cache block, a preload cache block, a partial sum cache, and two multiplexers. The control unit, based on the CPU configuration, can control the read / write behavior of the input feature cache block, the preload cache block, and the partial sum cache block, as well as the selection behavior of the two multiplexers. The input feature cache block is identified as Act RAM; the preload cache block is identified as Load RAM; and the partial sum cache block is identified as PartialSum RAM.

[0089] In this embodiment, as Figure 6As shown, to achieve flexible dynamic combination between cores, some data within the cores needs to be reused. This allows the dynamic combination of cores to support runtime combination. Therefore, a true dual-port RAM implementation is used for the input feature buffer block, meaning that its A port and B port can perform read and write operations on the input feature buffer block respectively, and the operations of the two ports are independent of each other. When core A is running task 1, after time t1, core B completes task 2. Through the true dual-port design of the input feature buffer block in the core, core B can read the data in the input feature block of core A without affecting the operation of core A itself, thus achieving the effect of core A and core B working together to execute task 1, thereby significantly shortening the processing time of task 1. For the preload buffer block and partial buffer block, a pseudo dual-port RAM implementation is used, meaning that its A port can only perform write data operations, and its B port can only perform read data operations, thereby reducing unnecessary resource consumption.

[0090] In this embodiment, as Figure 7 As shown, this systolic array inserts registers at the boundaries, thus simplifying the design of external memory. The registers are located in... Figure 7 The symbol D is used to identify the computational unit. Each computational unit consists of multiple Processing Arrays (PEs). PEs in the same row transmit input feature data (data paths indicated by dashed lines), while PEs in the same column transmit the computational data (data paths indicated by solid lines). Each PE consists of an adder, a multiplier, and corresponding registers. When a task needs to be performed, weight data must be pre-stored in the weight register inside the PE. After the weight data is loaded, features, computational data, and results are continuously input from outside the systolic array. The output of the systolic array flows out from the bottom PE, passes through registers, and is then output to the outside.

[0091] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the claims of the present invention.

Claims

1. A dynamically reconfigurable systolic array accelerator for Multi-DNN, characterized in that, include: CPU, DDR control module, DDR module, cross switch matrix module, computing system module; The computing system module includes several cores and several routers; each core independently runs a DNN load; each core corresponds to one router; the cores are dynamically combined to form an on-chip network; the routers are connected to form a toroidal topology. The CPU communicates with the computing system module via the AXI bus to control the dynamic combination of cores in the computing system module; The DDR module is used for data storage; The DDR control module is used to receive signals from the CPU, initiate read and write commands to the DDR module, and send the data in the DDR module to the cross switch matrix module. The cross switch matrix module is used for data communication between the DDR control module and the core in the computing system module.

2. The dynamically reconfigurable systolic array accelerator for Multi-DNN according to claim 1, characterized in that, The toroidal topology is as follows: The routers are electrically connected via a bidirectional bus, and each router is electrically connected to several adjacent routers in different directions. Routers located at the boundary are connected to adjacent router modules located at the boundary in a ring configuration. Routers in the same row at the boundary are connected, and routers in the same column at the boundary are also connected.

3. The dynamically reconfigurable systolic array accelerator for Multi-DNN according to claim 1, characterized in that, Each core includes a systolic array, an input feature buffer block, a preload buffer block, a partial buffer block, a control unit, a first selector, and a second selector; The first selector is used to receive the feature data output by the router corresponding to the core, and the feature data of the core output from the cross switch matrix module to the computing system module; The input feature buffer block is used to receive feature data output by the first selector; The aforementioned portion and buffer block are used to receive portions and data output by the pulsating array; The preloaded cache block is used to receive weight data or partial sum data of the core output from the cross switch matrix module to the computing system module; The second selector is used to receive partial and data output from the router corresponding to the core, weight data or partial and data output from the preloaded cache block; The pulse array is used to receive feature data output from the input feature buffer block, weight data or partial sum data output from the preloaded buffer block, and partial sum data output from the second selector, process the data, and output the data to the partial sum buffer block. The control unit is used to receive information transmitted from the CPU to the computing system module, control the read and write behavior of the input feature cache block, the preloaded cache block, the partial cache block, and the selection behavior of the first selector and the second selector.

4. The dynamically reconfigurable systolic array accelerator for Multi-DNN according to claim 1, characterized in that, The crossbar switch matrix module enables one-to-many master-slave communication for the cores in the computing system module.

5. A dynamically reconfigurable systolic array accelerator for Multi-DNN according to claim 4, characterized in that, The CPU configures the DDR control module via the AXI bus. After reading the data in the DDR module, the DDR control module transmits it to the core of the computing system module through the cross switch matrix module.

6. A dynamically reconfigurable systolic array accelerator for Multi-DNN according to claim 4, characterized in that, The CPU configures the computing system module via the AXI bus and configures several routers according to the characteristics of the DNN load, so that the cores can be combined to form an on-chip network adapted to the DNN load characteristics for data processing.

7. A dynamically reconfigurable systolic array accelerator for Multi-DNN according to claim 4, characterized in that, The cores are connected via a router, which transmits data and components from top to bottom to achieve vertical integration between cores. The cores are connected via a router, which transmits feature data from left to right to achieve horizontal combination between cores.

8. A dynamically reconfigurable systolic array accelerator for Multi-DNN according to claim 7, characterized in that, The aforementioned portions and data include: The result data generated during the calculation process, if this result data must be added to other result data later, is called partial sum data.

9. A dynamically reconfigurable systolic array accelerator for Multi-DNN according to claim 7, characterized in that, The feature data includes: A portion of the input data required for the kernel to perform calculations.

10. A dynamically reconfigurable systolic array accelerator for Multi-DNN according to claim 3, characterized in that, The input feature buffer block uses a true dual-port RAM; both ports of the true dual-port RAM are used for read and write data operations. The preloaded cache block, the portion, and the cache block all use pseudo-dual-port RAM; the first port of the pseudo-dual-port RAM is used only for write data operations, and the second port of the pseudo-dual-port RAM is used only for read data operations.

Images