A tool for deployment mapping of compute-in-memory fpga

Abstract

The application discloses a tool for deployment mapping of a storage-computing FPGA, which comprises input modules, an instruction generation module, an instruction compiler, a scheduling module and the like, takes a network input in the form of prototxt and a storage-computing FPGA architecture in the form of xml as input, transforms and compiles the network model input; according to the number of layers of the input network model and the number of CIM_pb on the storage-computing FPGA architecture, the network model is deployed; according to the deployment result, a class template with parameters is called and the parameters are instantiated to generate the RTL code of the network; then the RTL code and the FPGA architecture are sent into a logic synthesis and technology mapping module to generate a corresponding circuit netlist; then the netlist and the FPGA architecture are sent into a layout and routing module to package, layout and route the same on the storage-computing FPGA architecture, and finally the whole process of network deployment mapping is completed and an analysis report file of power consumption, area, timing and the like is generated.

Description

Technical Field

[0001] This invention belongs to the field of FPGA architecture modification and in-memory computing technology, specifically relating to a deployment mapping tool for in-memory FPGAs. Background Technology

[0002] In recent years, research on machine learning algorithms has increased significantly. Deep neural networks, especially convolutional neural networks (CNNs), are among the most representative algorithms in machine learning and are widely used in fields such as computer vision, pattern recognition, speech recognition, and natural language processing. While CNNs offer extremely high accuracy, they also require a substantial amount of computation. Furthermore, advanced hardware is limited by bandwidth and latency when interacting with off-chip memory, hindering overall performance improvement. Therefore, much research on non-von Neumann architectures, such as in-memory computing and near-memory computing, has been proposed.

[0003] Field-Programmable Gate Arrays (FPGAs) are frequently used as hardware platforms for deep learning accelerators due to their advantages in parallelism and reconfigurability. Most FPGAs currently contain programmable logic blocks (CLBs), computing units (DSPs), and distributed block random access memory (BRAMs), all interconnected via highly configurable interconnects. BRAMs, as crucial on-chip storage resources, enable high-speed data exchange with the computing units. However, since on-chip data is primarily stored in BRAMs, and computation requires data transfer to the computing units, frequent and large-scale data movements consume significant wiring and interconnect resources. Summary of the Invention

[0004] To address the problems in existing technologies, this invention proposes a deployment mapping tool for in-memory FPGAs. This invention aims to leverage the parallelism, reconfigurability, and low power consumption of FPGAs, along with in-memory computing, to design an in-memory FPGA tool that supports convolutional neural network mapping.

[0005] The technical solution of the present invention is as follows:

[0006] This invention first provides a deployment mapping tool for in-memory FPGAs, which includes:

[0007] The input module is used to acquire input, which includes the network model to be deployed and the FPGA architecture file;

[0008] The instruction generation module is used to convert the input network model into several lines of 54-bit instructions;

[0009] The instruction compiler compiles the 54-bit instructions generated by the instruction generation module.

[0010] The scheduling module deploys the network model based on the number of layers in the input network model and the number of CIM_pb on the in-memory FPGA architecture.

[0011] The RTL code generation module instantiates the parameterized parts of a single-layer or multi-layer network class template based on the deployment results and generates RTL code after instantiation and parameter passing.

[0012] The logic synthesis and technology mapping module is used to obtain the RTL code and FPGA architecture files generated by the deployment schedule, perform front-end logic synthesis and technology mapping, and convert the network to be deployed into the corresponding netlist file.

[0013] The placement and routing module, based on the netlist and FPGA architecture file generated by the logic synthesis and technical mapping module, performs packaging, placement, and routing on the in-memory FPGA architecture. After completion, it generates an analysis report on the timing, area, and power consumption of this deployment mapping.

[0014] Furthermore, the network model to be deployed is input in prototxt format. This input includes two parts: the first part is the name of the entire network and the dimension of the input feature map enclosed in the keyword input_shape; the second part consists of several keywords layer, which describe the information of each layer of the network model, including the network type of each layer, the data input source of each layer, the data output destination of each layer, and the network size of each layer.

[0015] Furthermore, the FPGA architecture file is in XML format; in addition to the logic unit CLB and the basic storage unit BRAM, the in-memory FPGA architecture also embeds several in-memory cores CIM_pb arranged in an array; before the in-memory core CIM_pb performs MAC operation, all weight information of the network model is stored in the SRAM array inside the allocated in-memory core CIM_pb; after the weights are pre-stored, the input feature map is sent to the corresponding in-memory core CIM_pb in a certain order, and multiply-add with the weights in the in-memory core to complete the MAC operation.

[0016] Furthermore, each line of instruction generated by the instruction generation module is divided into 8 parts; the number of bits and their meanings in each part of the instruction are as follows:

[0017] The first part, corresponding to the 1st to 3rd positions, uses a 3-bit number to represent the network type of this layer;

[0018] The second part, corresponding to bits 4-16, uses a 13-bit number to represent the number of output channels;

[0019] The third part, corresponding to bits 17-29, uses 13 bits to represent the number of input channels;

[0020] The fourth part, corresponding to bits 30-37, uses an 8-bit number to represent the height of the input feature map;

[0021] The fifth part, corresponding to bits 38-45, uses 8 bits to represent the width of the input feature map;

[0022] Part 6, corresponding to bits 46-49, uses 4 bits to represent the window size;

[0023] Part 7, corresponding to bits 50-52, uses 3 bits to represent the sliding window step size;

[0024] Part 8, corresponding to bits 53-54, uses 2 bits to represent the padding size.

[0025] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0026] (1) The present invention adopts in-memory computing technology. By using a large number of in-memory computing cores CIM_pb, data storage and computing are performed in the same place, which reduces the amount of data moving during network computing and thus achieves low power consumption.

[0027] (2) The present invention uses FPGA as the hardware platform for network deployment. By making full use of the characteristics of short development cycle, high flexibility and configurability of FPGA, the present invention can quickly deploy and map various different networks.

[0028] (3) This invention adopts a method combining FPGA and in-memory computing, utilizes a novel in-memory FPGA architecture, and proposes a tool adapted to this architecture to complete network deployment mapping, enabling this invention to complete hardware deployment of a specific network more quickly and conveniently, and to evaluate the timing, power consumption, area and other performance of the hardware circuit. Attached Figure Description

[0029] Figure 1 This is a flowchart illustrating the tool operation of the deployment mapping for in-memory FPGAs according to the present invention.

[0030] Figure 2 The input for the network model in prototxt format used in the example embodiments;

[0031] Figure 3 This is an example of a memory-based FPGA architecture in the embodiments;

[0032] Figure 4 This is a schematic diagram illustrating the port definition of CIM_pb in the example embodiment;

[0033] Figure 5 This is a schematic diagram showing the bit width of each part of the 54-bit instruction of the present invention;

[0034] Figure 6 This is a schematic diagram of a multilayer network structure;

[0035] Figure 7 The allocation of resources for a single deployment of an 8x8 CIM_pb;

[0036] Figure 8 The allocation of resources for multiple deployments of the 5x5 CIM_pb;

[0037] Figure 9 A schematic diagram illustrating the data flow during a single deployment on a storage-based FPGA;

[0038] Figure 10 Input the generated instruction file into the test case LeNet network model;

[0039] Figure 11 This is a schematic diagram of the layout of the LeNet network as a test example on a 42x42 in-memory FPGA architecture. Detailed Implementation

[0040] The present invention will be further described and illustrated below with reference to specific embodiments. The embodiments described are merely examples of the content of this disclosure and do not limit the scope of the invention. The technical features of each embodiment in the present invention can be combined accordingly, provided that there is no mutual conflict.

[0041] like Figure 1 As shown, this embodiment of the invention provides a deployment mapping tool for in-memory computing FPGAs, which includes:

[0042] The input module is used to acquire input, which includes the network model to be deployed and the FPGA architecture file;

[0043] The instruction generation module is used to convert the input network model into several lines of 54-bit instructions;

[0044] The instruction compiler compiles the 54-bit instructions generated by the instruction generation module.

[0045] The scheduling module deploys the network model based on the number of layers in the input network model and the number of CIM_pb on the in-memory FPGA architecture.

[0046] The RTL code generation module instantiates the parameterized parts of a single-layer or multi-layer network class template based on the deployment results and generates RTL code after instantiation and parameter passing.

[0047] The logic synthesis and technology mapping module is used to obtain the RTL code and FPGA architecture files generated by the deployment schedule, perform front-end logic synthesis and technology mapping, and convert the network to be deployed into the corresponding netlist file.

[0048] The placement and routing module, based on the netlist and FPGA architecture file generated in the previous module, performs packaging, placement, and routing on the in-memory FPGA architecture. After completion, it generates an analysis report on the timing, area, and power consumption of this deployment mapping. The overall flowchart of this invention is as follows: Figure 1 As shown. The entire tool's input consists of two parts. The first part is the network model to be deployed, input in prototxt format. The network model input format is as follows: Figure 2 As shown. Figure 2 The image shows a typical LeNet network model. Only the parameters of the first two layers are displayed: a convolutional layer and a pooling layer. The network input consists of two main parts: the first is the name of the entire network and the dimension of the input feature map enclosed in the keyword `input_shape`; the second part comprises several keywords `layer`, each describing information such as the network type, data input source, data output destination, and network size. The second input to the tool is an FPGA architecture file in XML format. The architecture file describes the types of resources, their placement, and the size of the FPGA to be deployed.

[0049] like Figure 3 The diagram illustrates a typical in-memory FPGA architecture deployed according to this invention. In addition to basic logic units (CLBs) and basic memory units (BRAMs), this in-memory FPGA architecture embeds a large number of newly defined CIM_pb resources. CIM_pb is the in-memory core, playing two main roles in the entire network mapping and deployment process. First, before computation begins, all network weight information is stored in the SRAM array within the CIM_pb. Second, after computation begins, the input feature map is allocated to different in-memory cores in a specific order, and then multiplied and added with the weights pre-stored in the in-memory cores to complete the basic MAC operation.

[0050] The port type and bit width of each CIM_pb are as follows: Figure 4As shown. For the CIM_pb with an internal SRAM array size of 64x256, it is encapsulated into 8 external ports: data, addr, baddr, clk, rstn, we, enable, and out. The first seven ports are of type input, and the last one, out, is of type output. data is the data input port with a bit width of 256, supporting 256-bit input per cycle. clk is a 1-bit clock input, rstn is a 1-bit active-low reset port, we is a 1-bit write SRAM enable port, and enable is a 1-bit enable port to start the memory-in-memory core's computation. addr is a 6-bit address selection port used to select whether to read or write a row of SRAM in the current cycle. baddr is a 3-bit bit select signal used to activate an 8-bit input. out is a 21-bit partial output port. The output of out can either be directly stored in BRAM as an intermediate value, or the out outputs of multiple CIM_pb need to be added together outside the core.

[0051] The CIM_pb core consists of an SRAM memory array, an address decoding circuit, a control circuit, several NOR gates, an addition tree, and a shift-add module. The SRAM memory array stores the network weights; the address decoding circuit decodes the address signal `addr` and selects a row from the SRAM memory array; the control circuit controls the external activation input and selects the active bit; the NOR gates implement the 1-bit product of the weights and the active bit; the addition tree performs parallel addition operations on multiple 8-bit numbers; and the shift-add module accumulates multiple partial sums. The combined components of the entire CIM_pb core perform the matrix-vector product operation.

[0052] The instruction generation module will convert the input prototxt network model into several lines of 54-bit instructions. Each line of instructions is divided into 8 parts, and the bit width of each part is as follows: Figure 5 As shown, the meaning of each part is as follows.

[0053] ① (1st-3rd bits): A 3-bit number represents the network type of this layer. 000 represents a convolutional layer, 001 represents a pooling layer, 010 represents a fully connected layer, 011 represents an activation function layer, etc.

[0054] ② (bits 4-16): The number of output channels (OC) is represented by 13 bits, ranging from 0 to 8191.

[0055] ③ (Bits 17-29): The number of input channels (IC) is represented by 13 bits, ranging from 0 to 8191.

[0056] ④ (30th-37th bits): The input feature map height (IH) is represented by an 8-bit number, ranging from 0 to 255.

[0057] ⑤ (bits 38-45): The input feature map width (IW) is represented by an 8-bit number, ranging from 0 to 255.

[0058] ⑥ (46th-49th bits): The window size (KW / KH) is represented by a 4-bit number, ranging from 0 to 15.

[0059] ⑦ (bits 50-52): The sliding window step size (KS) is represented by a 3-bit number, ranging from 0 to 7.

[0060] ⑧ (53rd-54th bits): The padding size is represented by 2 bits, ranging from 0 to 3.

[0061] The instruction compiler then compiles the generated 54-bit instructions, and accesses the text file instruction.txt that stores the instructions.

[0062] After compilation, the scheduling module analyzes the number of instruction lines to determine the number of layers in the input network model. The tool will perform scheduling for single-layer and multi-layer networks separately, inferring the number of network layers based on the number of stored instructions and deciding whether to execute single-layer or multi-layer network scheduling. During network scheduling, the tool also reads the in-memory FPGA architecture to determine the number of available CIM_pb. Specifically, since single-layer networks do not need to consider data transmission between layers or the impact of the output of the previous layer on the input of the next layer, when the network layer is small, or when the number of CIM_pb on the in-memory FPGA architecture to be deployed is sufficient, the allocated CIM_pb can start computing in parallel without needing to consider waiting for input. However, for multi-layer network scheduling, the dependencies between layers must be considered, as the computation times of all layers are not simultaneous. Therefore, the scheduling tools for single-layer and multi-layer networks have different parameterized class templates, determining whether to execute single-layer or multi-layer network scheduling by reading the instructions to determine the number of layers of the network input to be deployed. During scheduling, the tool further analyzes the instructions, segments the 54-bit instructions, obtains information such as the network type and size, and then allocates a certain number of CIM_pb for network calculations at that layer.

[0063] Of course, scheduling and allocation must balance the size of the network to be deployed with the amount of resources on the in-memory FPGA architecture to be deployed. When there are sufficient resources on the default FPGA, all layers of the multi-layer network will be deployed on the entire in-memory FPGA at once. After the FPGA architecture is input, the tool reads the number of CIM_pb values ​​on the architecture and analyzes the number of CIM_pb values ​​required for each layer's network deployment. When the total number of CIM_pb values ​​required by all layers is less than the number of CIM_pb values ​​on the in-memory FPGA architecture, the multi-layer network will be deployed once; when the total number of CIM_pb values ​​required by all layers is greater than the number of CIM_pb values ​​on the in-memory FPGA architecture, the multi-layer network will be deployed multiple times. The tool supports the following multi-layer network structures: Figure 6 As shown, it supports a structure that includes multiple convolutional layers connected to pooling layers, and a structure that includes multiple sequentially connected fully connected layers in a convolutional neural network.

[0064] like Figure 7 As shown, assume the size of the CIM_pb array on the in-memory FPGA architecture is 8x8, meaning there are 64 CIM_pbs. Also assume the input network model has four layers, and each layer requires 8, 16, 16, and 8 CIM_pbs respectively. Clearly, deploying the entire network requires 48 CIM_pbs, while the architecture has 64 CIM_pbs. Therefore, the tool will proceed as follows... Figure 7 The entire network mapping is completed in one deployment. If the size of the deployed in-memory computing FPGA architecture is changed, such as... Figure 8 As shown, the original 8x8 CIM_pb array is reduced to a 5x5 CIM_pb array. After the tool reads the in-memory FPGA architecture, it analyzes the relationship between available and required resources, ultimately achieving a more reasonable allocation. Figure 8 As shown, without changing the input network model, the tool will perform two deployments this time. The first deployment will first deploy the first and second layers of the network. After the calculation of the first two layers is completed, the weights stored in CIM_pb will be updated, and then the deployment of the third and fourth layers will be performed.

[0065] Of course, if the given in-memory FPGA architecture is large enough, it will be possible to deploy a multi-layer network in one go, such as... Figure 9As shown, the entire calculation process is as follows: the input feature map is input through the FPGA's peripheral I / O and stored in the on-chip BRAM in a certain order. Then, it is read from the BRAM in the required order, and the input activation is sent to the allocated CIM_pb. In CIM_pb, the input activation is processed by MAC operation with the pre-stored weights, and the partial sum is sent out from the CIM_pb port. Then, depending on the situation, it is determined whether the partial sums need to be added again. If so, the result is sent to the on-chip CLB resource; otherwise, the calculation result of this layer is sent back to the BRAM. When the next layer is ready to start calculation, it reads the calculation result of the previous layer from the BRAM and sends it to the allocated CIM_pb for that layer, and so on, until the calculation of the entire network is completed.

[0066] The tool then sends the RTL code generated by the deployment scheduler and the FPGA architecture file into the logic synthesis and technology mapping module. This module mainly performs front-end processing on the input RTL code, synthesizing it into existing resources on the given FPGA architecture and sending it out in the form of a netlist. Most of the modules are synthesized into the most basic lookup tables and register resources on the FPGA (these two resources are embedded in CLBs), some logic is synthesized into BRAM resources, and the core MAC calculation code is mapped to the in-memory computing core CIM_pb for completion.

[0067] Finally, the tool sends the generated netlist and FPGA architecture file to the place-and-route module. This module first packages the input netlist, such as packaging several lookup tables and registers into a CLB block; then it performs placement operations according to the placement positions of resources on the FPGA architecture; finally, it performs routing operations between the packaged and placed logic blocks according to a certain routing algorithm, and then performs timing, area, and power consumption evaluations and generates corresponding report files.

[0068] The following is a test case demonstrating the deployment and mapping of a LeNet network on a memory FPGA using this tool. The input to the network model is... Figure 2 The input shown is in prototxt format. After passing through the instruction generation module, it generates 6 lines of 54-bit instructions, as follows: Figure 10 As shown. The 6 lines of instructions correspond to the 6 layers of the LeNet network: convolutional layer, pooling layer, convolutional layer, pooling layer, fully connected layer, and fully connected layer. The meaning of each layer and the meaning of each part of the instruction are as follows. Figure 10 As marked.

[0069] The in-memory FPGA architecture input in the test case is 42x42 in size, with I / O ports on the periphery. The two leftmost columns of the architecture are multipliers, each 4x1 in size, totaling 16. Next to the multipliers are BRAMs, each 6x1 in size, totaling 10. The right side of the entire architecture is filled with our in-memory cores CIM_pb, each 3x1 in size, totaling 48, spaced 2 units vertically and 5 units horizontally. The remaining space is filled with CLBs, each 1x1 in size.

[0070] Then, based on the tool's multi-layer scheduling, it was analyzed that the LeNet network's 6 layers require a total of 40 CIM_pb cores for deployment. Clearly, the number of in-memory compute cores (CIM_pb) on the FPGA architecture exceeds the total number required for the entire network, so the tool performs one deployment. Afterwards, the RTL code generation module generates the corresponding RTL code and sends it to the next module for synthesis, mapping, placement, and routing.

[0071] Figure 11 The simulation results for the LeNet network in the test case after packaging and placement are shown in the figure. The information in the figure verifies the correctness of the deployment mapping; solid borders indicate resource blocks that have been used, while dashed borders indicate unused resource blocks. According to the power report file output by the tool, with a clock frequency of 100MHz, a temperature of 25 degrees Celsius, and a 45nm process, the overall power consumption was measured to be 69.95mW, and the calculated energy efficiency was 473.19 GOPS / W. This is several tens of times higher than other LeNet networks deployed on general FPGAs under the same environmental process and only achieving a dozen or so GOPS / W. The above test case demonstrates that our tool's network deployment mapping implementation on in-memory FPGAs can significantly improve energy efficiency and overall hardware computing performance.

[0072] The above-described embodiments are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. Those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.

Claims

1. A deployment mapping tool for in-memory FPGAs, characterized in that... include: The input module is used to acquire input, which includes the network model to be deployed and the FPGA architecture file; The network model is input in prototxt format; the FPGA architecture file is in XML format; in addition to logic units (CLBs) and basic memory units (BRAMs), the in-memory FPGA architecture also embeds several in-memory cores (CIM_pb) arranged in an array; before the in-memory cores (CIM_pb) perform MAC operations, all weight information of the network model is stored in the SRAM array inside the allocated in-memory core (CIM_pb); after the weights are pre-stored, the input feature map is sent to the corresponding in-memory core (CIM_pb) in a certain order, and multiply-add operations are performed with the weights in the in-memory core to complete the MAC operation; The instruction generation module is used to convert the input network model into several lines of 54-bit instructions; The instruction compiler compiles the 54-bit instructions generated by the instruction generation module. The scheduling module deploys the network model based on the number of layers in the input network model and the number of CIM_pb on the in-memory FPGA architecture. For a single-layer network, the allocated CIM_pb are computed in parallel. For a multi-layer network, after the FPGA architecture is input, the module reads the number of CIM_pb on the architecture and analyzes the number of CIM_pb required for each layer's deployment. When the total number of CIM_pb required by all layers is less than the number of CIM_pb on the in-memory FPGA architecture, the multi-layer network will be deployed once. When the total number of CIM_pb required by all layers is greater than the number of CIM_pb on the in-memory FPGA architecture, the multi-layer network will be deployed multiple times. The RTL code generation module instantiates the parameterized parts of a single-layer or multi-layer network class template based on the deployment results and generates RTL code after instantiation and parameter passing. The logic synthesis and technology mapping module is used to obtain the RTL code and FPGA architecture files generated by deployment scheduling, perform front-end logic synthesis and technology mapping, and convert the network model to be deployed into the corresponding netlist file. The placement and routing module, based on the netlist and FPGA architecture file generated by the logic synthesis and technical mapping module, performs packaging, placement, and routing on the in-memory FPGA architecture. After completion, it generates an analysis report on the timing, area, and power consumption of this deployment mapping.

2. The deployment mapping tool for in-memory FPGAs according to claim 1, characterized in that, The input of the network model consists of two parts. The first part is the name of the entire network and the dimension of the input feature map enclosed in the keyword input_shape. The second part consists of several keywords layer, which describe the information of each layer of the network model, including the network type of each layer, the data input source of each layer, the data output destination of each layer, and the network size of each layer.

3. The deployment mapping tool for in-memory FPGAs according to claim 2, characterized in that, Each instruction generated by the instruction generation module is divided into 8 parts; the number of bits and their meanings in each part are as follows: The first part, corresponding to bits 1-3, uses 3 bits to represent the network type of the network layer corresponding to the current instruction; The second part, corresponding to bits 4-16, uses a 13-bit number to represent the number of output channels; The third part, corresponding to bits 17-29, uses 13 bits to represent the number of input channels; The fourth part, corresponding to bits 30-37, uses an 8-bit number to represent the height of the input feature map; The fifth part, corresponding to bits 38-45, uses 8 bits to represent the width of the input feature map; Part 6, corresponding to bits 46-49, uses 4 bits to represent the window size; Part 7, corresponding to bits 50-52, uses 3 bits to represent the sliding window step size; Part 8, corresponding to bits 53-54, uses 2 bits to represent the padding size.

Images