A store compute FPGA architecture

Abstract

The application discloses a memory-compute FPGA architecture. Memory-compute cores, memory blocks and logic function blocks used for replacing part of the logic function blocks are arranged in a row-column array and arranged in an island type architecture. In the memory-compute cores, there are SRAMs arranged in a row-column array, the SRAMs store one-bit data, the output ends of an address decoding circuit are connected with each row of the SRAM array, the output ends of a write control circuit are connected with each column of the SRAM array, each eight columns of the SRAM array and an input register are connected with an 8-bit input port of an addition tree through a multiplexer and eight NOR gates, and the output port of the addition tree is connected with an output register through a shift and addition module and a quantization module in sequence. The application realizes in-memory computation on the FPGA, utilizes the parallelism and reconfigurability of the FPGA to support mapping of various networks, and utilizes in-memory computation technology to reduce data movement and further reduce computation power consumption.

Description

Technical Field

[0001] This invention relates to modifications of FPGA architecture and near-memory system architecture, specifically to a memory-based FPGA architecture that supports general network MAC operations. Background Technology

[0002] The development of cutting-edge technologies such as the Internet, cloud computing, artificial intelligence, and the Internet of Things has driven a dramatic increase in data volume. The development of these technologies is inseparable from deep neural networks. However, in processing data-intensive applications like deep neural networks, the frequent transfer of large amounts of data between the processor and memory causes severe performance degradation and energy consumption, which is the biggest bottleneck of the current von Neumann architecture. The traditional von Neumann architecture consists of an arithmetic logic unit (ALU), a control unit, memory, and input / output devices, with these modules interconnected via a bus. This structure includes independent computing and storage units. During the execution of various computational tasks, it requires repeated transfer of large amounts of data between the storage and computing units, leading to significant latency and energy consumption, thus limiting data processing efficiency. To break through the von Neumann bottleneck and build more efficient hardware facilities and computing architectures in the context of intelligent big data, in-memory computing technology has been proposed. This technology integrates computing and storage, supporting direct computation of data within the storage module and feeding the final results back to the processor, thereby significantly reducing the time and energy consumption of data transfer on the bus. Summary of the Invention

[0003] To address the problems existing in the background technology, this invention proposes a memory-based FPGA architecture that combines the advantages of FPGA parallelism and reconfigurability with the low power consumption of in-memory computing and supports multiple network mappings.

[0004] The technical solution adopted in this invention is:

[0005] The FPGA architecture of this invention includes input / output modules (IOB), logic function blocks (CLB), and memory blocks (BRAM). The input / output modules (IOB) are distributed on the periphery of the FPGA, the basic memory blocks (BRAM) are distributed inside the FPGA, and the logic function blocks (CLB) are distributed throughout the FPGA. It also includes a storage-in-memory core (CIM_core) used to replace some of the logic function blocks (CLB). The storage-in-memory core (CIM_core), the memory blocks (BRAM), and the logic function blocks (CLB) are arranged in a row-column array in an island-shaped architecture.

[0006] The in-memory computing core (CIM_core) mainly consists of an SRAM array, registers, an adder tree, a shift and add module (Mux and add), a quantization module, an address decoding circuit, and a write control circuit. The registers include multiple input registers and one output register.

[0007] The SRAM array contains several SRAMs arranged in rows and columns. Each SRAM is used to store one bit of data. The outputs of the address decoding circuit are connected to the SRAMs in each row of the SRAM array. The outputs of the write control circuit are connected to the SRAMs in each column of the SRAM array. Every eight columns of SRAMs and a corresponding input register are connected to an 8-bit input port of an adder tree through a multiplexer (MUX) and a NOT gate. The output port of the adder tree is connected to the output register after passing through a shift and add module (Mux and add) and a quantization module (quantization).

[0008] In this configuration, the outputs of each SRAM in each column of the SRAM array are connected to one input of a corresponding NOR gate. The other input of each NOR gate is connected to the same input register via the same multiplexer (MUX). The output of each NOR gate is connected to one input port of an adder tree.

[0009] The address decoding circuit receives address data, processes it to generate write line signals WWL, read line signals RWL, and read line inverted signals RWLB, and each write line signal WWL, read line signal RWL, and read line inverted signal RWLB is input into a row of SRAM in the SRAM array;

[0010] Each SRAM outputs a read bit line signal RBL. The write control circuit generates write bit line signals WBL and write bit line inversion signals WBLB. Each write bit line signal WBL, read bit line signal RBL, and write bit line inversion signal WBLB is input into a column of SRAMs in the SRAM array.

[0011] Each SRAM in the SRAM array is controlled by the write word line signal WWL, read word line signal RWL, read word line inverted signal RWLB, write bit line signal WBL, read bit line signal RBL, and write bit line inverted signal WBLB to output a one-bit value. Each input register stores multiple-bit values. The multiple-bit values ​​in the input register are selected by the multiplexer MUX and sent to a NOR gate. Together with a one-bit value output by one SRAM in the SRAM array, they are NORed to obtain a one-bit output value. The bit values ​​output by each SRAM in each column of the SRAM array are connected in the order of each SRAM in the column to form a multi-bit array value. The multi-bit array value is input to an input port of the adder tree.

[0012] The output port of the Adder Tree outputs all the multi-bit array values ​​of the SRAM array. After arranging them, the shift and add module Mux and add them to obtain a (inconsistent) multi-bit shifted value. Then, the multi-bit shifted value is processed by the quantization module to extract the consecutive multi-bit outputs and store them in the output register.

[0013] Each SRAM in the SRAM array outputs a single bit value under the control of the write word line signal WWL, read word line signal RWL, read word line inverted signal RWLB, write bit line signal WBL, read bit line signal RBL, and write bit line inverted signal WBLB.

[0014] The write word line signal WWL, write bit line signal WBL, and write bit line inverse signal WBLB are used to control the write operation of the SRAM array: when the write word line signal WWL of a certain row is 1, a write operation is performed on all SRAM cells in that row, and the weight of the write is determined by the write bit line signal WBL and the write bit line inverse signal WBLB of that SRAM cell.

[0015] The read word line signal RWL, the read word line inverse signal RWLB, and the read bit line signal RBL are used to control the read operation of the SRAM array: when the read word line signal RWL of a certain row is 1 and the read word line inverse signal RWLB is 0, a read operation is performed on all SRAM cells in that row, and the read weight is read out through the read bit line signal RBL of each SRAM cell and sent out of the SRAM array.

[0016] Each SRAM is mainly composed of 10 MOS transistors. The gates of MOS transistors M0, M5, and M8 are connected to the write word line signal WWL. The source of MOS transistor M0 is connected to the write bit line signal WBL. The drains of MOS transistors M0, M1, and M2, and the gates of MOS transistors M3 and M4 are connected together. The sources of MOS transistors M1, M3, and M6 are connected to voltage, and the sources of MOS transistors M2, M4, and M7 are grounded. The drains of transistors M5, M3, and M4, as well as the gates of transistors M1, M2, M6, and M7, are connected together. The source of transistor M5 is connected to the write bit line inverted signal WBLB. The drains of transistors M6 through M9 are connected together. The sources of transistors M8 and M9 are connected together and then connected to the read bit line signal RBL. The gate of transistor M8 is connected to the read word line signal RWL. The gate of transistor M9 is connected to the read word line inverted signal RWLB.

[0017] The Adder Tree is composed of multiple Ripple Carry Adders (RCAs). Each Adder Tree receives a multi-bit array value from each of its input ports, aligns and sorts these values ​​according to the input port order, and performs multiple carry additions through the RCAs to obtain the final multi-bit value.

[0018] In each carry-add process, all the current multi-bit values ​​are grouped into pairs of adjacent multi-bit values ​​and added together by a ripple carry adder (RCA) to obtain a single multi-bit value. If the number of current multi-bit values ​​is odd, the remaining single multi-bit value is not processed in the current carry-add process but is directly passed to the next carry-add process.

[0019] The addition tree consists of multiple carry-holding adders (CSA), multiple half-adders (HA), and a ripple carry adder (RCA).

[0020] The Adder Tree receives a multi-bit array value from each of its input ports. It then aligns and sorts these multi-bit array values ​​as multi-bit values ​​according to the input port order. Multiple carry-preserving adders (CSA) and half-adders (HA) are used to perform multiple carry-add operations until the number of multi-bit values ​​is reduced to only two.

[0021] In each carry-in addition process, each bit is traversed as a processing bit. First, every three consecutive values ​​under the processing bit are grouped together and added using a carry-holding adder (CSA) to obtain a value at the original processing bit and a value at the carry-over bit. If two values ​​remain in the processing bit after processing with the carry-holding adder (CSA), then these two values ​​are added using a half-adder (HA) to obtain a value at the original processing bit and a value at the carry-over bit. If only one value remains in the processing bit after processing with the carry-holding adder (CSA), then this value is not processed in the current carry-in addition process and is directly passed to the next carry-in addition process.

[0022] After the number of multi-digit values ​​is reduced to only two, a ripple carry adder (RCA) is used to add the two multi-digit values ​​to obtain the final multi-digit value.

[0023] The shift-add module (Mux and add) specifically performs shift and add operations on the multi-bit value output by the adder tree to obtain the final, unquantized result. The amount of shift depends on which bit of the input register was selected by the multiplexer (MUX). The least significant bit does not need to be shifted, the lower bit is shifted one bit to the left before being added, and so on.

[0024] The in-memory FPGA architecture described above is used for computation in the fully connected layer. It sequentially distributes the weights of each fully connected factor in the fully connected layer to the SRAMs of the same row of the in-memory core (CIM core). The weights of different fully connected factors in the fully connected layer are distributed to different rows of SRAMs. Each input register pre-stores the data of one channel of the input feature map.

[0025] The in-memory FPGA architecture used for convolutional layer operations distributes the weights of all input channels at the same position within a convolutional kernel in a convolutional layer sequentially into multiple consecutive SRAMs in the same row of a CIM core. It iterates through different positions within a convolutional kernel, and distributes the weights of all input channels at all positions within a convolutional kernel into the same row of SRAMs in the CIM core. For the input feature map, it slides through the convolutional kernel, and the activations of all input channels at the same position covered by the kernel are sequentially distributed into various input registers, with each input register storing an 8-bit activation.

[0026] The number of logical function blocks (CLBs) replaced by the storage and computation core (CIM_core) is handled according to the minimum number based on the computational allocation of the fully connected layers / convolutional layers.

[0027] This invention is based on VTR (Verilog-to-Routing) engineering. By modifying the FPGA architecture, a new in-memory compute core (CIM_core) capable of both storage and basic MAC operations is embedded. The network is fed in in a specific format, and after conversion and processing, the core information of the network is obtained. Then, based on the network size and the maximum size supported by the embedded CIM_core, the network is first segmented, and then the computation is completed using as few in-memory compute cores as possible. The newly embedded CIM_core supports both storage and computation modes. Before computation begins, the CIM_core operates in storage mode, where the network weight information is pre-stored in the CIM_core according to the timing sequence via an interface. After computation begins, the CIM_core is enabled to enter computation mode. In this mode, the data fed into the CIM_core is no longer stored but is directly placed into registers, awaiting the read weights before computation begins.

[0028] The beneficial effects of this invention are:

[0029] The in-memory computing FPGA architecture proposed in this invention enables in-memory computation to be performed on an FPGA. It leverages the parallelism and reconfigurability of FPGAs to support mappings to various networks, and utilizes in-memory computing technology to reduce data movement, thereby further reducing computational power consumption. Attached Figure Description

[0030] Figure 1 This is a schematic diagram of a custom FPGA architecture;

[0031] Figure 2 This is a schematic diagram of the internal structure of the storage kernel;

[0032] Figure 3 This is a schematic diagram of a 10T SRAM circuit structure;

[0033] Figure 4 It is an addition tree and dot matrix representation diagram constructed based on the traveling wave carry adder as the basic unit;

[0034] Figure 5 It is a dot matrix diagram that constructs an addition tree using carry-retaining adders and half-adders;

[0035] Figure 6 This is a schematic diagram of the mapping of a fully connected layer;

[0036] Figure 7 This is a schematic diagram of the mapping of convolutional layers;

[0037] Figure 8 This is a schematic diagram of the allocation of storage cores;

[0038] Figure 9 This is a schematic diagram of a 20x20 in-memory FPGA architecture;

[0039] Figure 10 This is a schematic diagram of a typical FPGA architecture, measuring 20x20 pixels.

[0040] Figure 11 This is a power report diagram using a memory-based FPGA architecture;

[0041] Figure 12 This is a power report diagram using a typical FPGA architecture. Detailed Implementation

[0042] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0043] like Figure 1 As shown, the FPGA architecture includes input / output modules (IOB), logic blocks (CLB), and memory blocks (BRAM). The input / output modules (IOB) are distributed on the periphery of the FPGA, the basic memory blocks (BRAM) are distributed inside the FPGA, and the logic blocks (CLB) are distributed throughout the FPGA. The logic blocks (CLB) contain several lookup tables, registers, and multiplexers, among other resources.

[0044] Its features include: a storage-in-memory core (CIM_core) for replacing part of the logic function block (CLB); the CIM_core, BRAM, and CLB are arranged in a row and column array, in an island-shaped architecture, and connected to each other as needed using wiring and switch boxes.

[0045] The in-memory compute core (CIM_core) replaces several logic function blocks (CLBs) to accelerate the core MAC operations of the neural network within the FPGA architecture. The core MAC operations refer to multiply-accumulate operations.

[0046] like Figure 2 As shown, the storage-in-memory core (CIM_core) mainly consists of an SRAM array, registers, an adder tree, a shift and add module (Mux and add), a quantization module, an address decoding circuit, and a write control circuit. The registers include multiple input registers and one output register.

[0047] The SRAM array contains several SRAMs arranged in rows and columns. Each SRAM is used to store one bit of data. The outputs of the address decoding circuit are connected to the SRAMs in each row of the SRAM array. The outputs of the write control circuit are connected to the SRAMs in each column of the SRAM array. Each column of SRAMs and its corresponding input register are connected to an input port of an adder tree through a multiplexer (MUX) and multiple NOR gates (the same number of SRAMs in each column). The output port of the adder tree is connected to the output register after passing through a shift and add module (Mux and add) and a quantization module (quantization).

[0048] A storage-in-memory core (CIM_core) essentially performs a matrix-vector product operation. The external vector, i.e. the activation, enters through the port and is temporarily stored in the register. In each calculation cycle, the weight of a row is read from the SRAM array and then a MAC operation is performed with the activation. In the next calculation cycle, the weight of the next row is read through the address control.

[0049] In this configuration, the outputs of each SRAM in each column of the SRAM array are connected to one input of a corresponding NOR gate. The other input of each NOR gate is connected to the same input register via the same multiplexer (MUX). The output of each NOR gate is connected to one input port of an adder tree.

[0050] The address decoding circuit receives address data, processes and generates write word line signals WWL, read word line signals RWL, and read word line inverse signals RWLB, which are the same number of rows in the SRAM array. Each write word line signal WWL, read word line signal RWL, and read word line inverse signal RWLB is input into one row of SRAM in the SRAM array.

[0051] The write control circuit generates write bit line signals WBL and write bit line inversion signals WBLB in the same number as the number of columns in the SRAM array. Each write bit line signal WBL and write bit line inversion signal WBLB is input to one column of SRAM in the SRAM array.

[0052] Each SRAM in the SRAM array outputs a single bit value to its corresponding NOR gate under the control of the write word line signal WWL, read word line signal RWL, read word line inverted signal RWLB, write bit line signal WBL, read bit line signal RBL, and write bit line inverted signal WBLB. Each input register stores multiple bits of value. The multiple bits of value in the input register are selected sequentially by the multiplexer MUX and sent to a NOR gate. Together with a single bit of value output from one SRAM in the SRAM array, they are NOR-NOT-computed to obtain a single output value. The bits of each SRAM output in each column of the SRAM array are connected in the order of the SRAMs in the column to form a multi-bit array value. The number of bits in the multi-bit array value is consistent with the number of columns in the SRAM array. The multi-bit array value is input to an input port of the adder tree.

[0053] The output port of the Adder Tree outputs all the multi-bit array values ​​of the SRAM array. After arranging them, the shift and add module Mux and add them to obtain a multi-bit shifted value with the same number of bits as the number of columns of the SRAM array. Then, the multi-bit shifted value is processed by the quantization module to extract the consecutive multi-bit outputs and store them in the output register.

[0054] Each SRAM in the SRAM array outputs a single bit value under the control of the write word line signal WWL, read word line signal RWL, read word line inverted signal RWLB, write bit line signal WBL, read bit line signal RBL, and write bit line inverted signal WBLB. Specifically:

[0055] The write word line signal WWL, write bit line signal WBL, and write bit line inverted signal WBLB are used to control the write operation of the SRAM array. When the write word line signal WWL of a certain row is 1, a write operation can be performed on all SRAM cells in that row. The weight of the write operation is determined by the write bit line signal WBL and the write bit line inverted signal WBLB of that SRAM cell.

[0056] The read word line signal RWL, read word line inverted signal RWLB, and read bit line signal RBL are used to control the read operation of the SRAM array. When the read word line signal RWL of a certain row is 1 and the read word line inverted signal RWLB is 0, a read operation can be performed on all SRAM cells in that row. The read weights are read out through the read bit line signal RBL of each SRAM cell and sent out of the SRAM array.

[0057] like Figure 3 As shown, each SRAM mainly consists of 10 MOSFETs. The gates of MOSFETs M0, M5, and M8 are connected to the write word line signal WWL. The source of MOSFET M0 is connected to the write bit line signal WBL. The drains of MOSFETs M0, M1, and M2, as well as the gates of MOSFETs M3 and M4, are connected together. The sources of MOSFETs M1, M3, and M6 are connected to voltage, while the sources of MOSFETs M2, M4, and M7 are grounded. The drains of S-channel transistors M5, M3, and M4, and the gates of MOSFETs M1, M2, M6, and M7 are connected together. The source of MOSFET M5 is connected to the write bit line inverted signal WBLB. The drains of MOSFETs M6 through M9 are connected together. The sources of MOSFETs M8 and M9 are connected together and then connected to the read bit line signal RBL. The gate of MOSFET M8 is connected to the read word line signal RWL. The gate of MOSFET M9 is connected to the read word line inverted signal RWLB.

[0058] The SRAM array consists of a basic 10T structure, where T represents a MOSFET. Compared to a traditional 6T SRAM, it adds an inverter and a transmission gate. The specific circuit structure is as follows: Figure 3 As shown.

[0059] Compared to the basic 6T SRAM structure, the 10T SRAM structure of this invention isolates the internal storage nodes and read paths through inverters. This structure allows for sufficient charging and discharging of the read bit line signal RBL, eliminating the need for an additional pre-charge circuit. Dynamic power consumption on the read bit line signal RBL only occurs when the read data changes. In other words, this structure exhibits no dynamic power loss on the read bit line signal RBL when reading data that is either constant '0' or constant '1'.

[0060] like Figure 4As shown, Scheme 1: The Adder Tree consists of multiple ripple carry adders (RCAs). The Adder Tree receives a multi-bit array value from each of its input ports, aligns and sorts the received multi-bit array values ​​as multi-bit values ​​according to the input port order, and performs multiple carry additions through the ripple carry adders (RCAs) to obtain the final multi-bit value.

[0061] In each carry-add process, all the current multi-bit values ​​are grouped into pairs of adjacent multi-bit values ​​and added together by a ripple carry adder (RCA) to obtain a single multi-bit value. If the number of current multi-bit values ​​is odd, the remaining single multi-bit value is not processed in the current carry-add process but is directly passed to the next carry-add process.

[0062] The above scheme one uses a ripple carry adder to construct an adder tree. The carry chain of each ripple carry adder used in each stage is one bit deeper than the carry chain of the previous stage. Its most obvious advantage is that the structure is relatively regular and simple.

[0063] like Figure 5 As shown, Scheme 2: The addition tree consists of multiple carry-holding adders (CSA), multiple half-adders (HA), and one ripple carry adder (RCA).

[0064] The Adder Tree receives a multi-bit array value from each of its input ports. It then aligns and sorts these multi-bit array values ​​as multi-bit values ​​according to the input port order. Multiple carry-preserving adders (CSA) and half-adders (HA) are used to perform multiple carry-add operations until the number of multi-bit values ​​is reduced to only two.

[0065] In each carry-in addition process, each bit is traversed as a processing bit. First, every three consecutive values ​​under the processing bit are grouped together and added using a carry-holding adder (CSA) to obtain a value at the original processing bit and a value at the carry-over bit. If two values ​​remain in the processing bit after processing with the carry-holding adder (CSA), then these two values ​​are added using a half-adder (HA) to obtain a value at the original processing bit and a value at the carry-over bit. If only one value remains in the processing bit after processing with the carry-holding adder (CSA), then this value is not processed in the current carry-in addition process and is directly passed to the next carry-in addition process.

[0066] After the number of multi-digit values ​​is reduced to only two, a ripple carry adder (RCA) is used to add the two multi-digit values ​​to obtain the final multi-digit value.

[0067] The second scheme described above constructs an addition tree based on the idea of ​​building multipliers using Wallace trees. It uses CSA carry-preserving adders as the basic building blocks. In fact, a CSA is essentially a full adder, except that two adjacent full adders are not cascaded during use.

[0068] The shift-add module (Mux and add) performs shift-add processing, specifically shifting and accumulating the multi-bit value output by the adder tree to obtain the final, unquantized result. The amount of shift depends on which bit of the input register was previously selected by the multiplexer (MUX). The least significant bit does not need to be shifted, the lower bit is shifted one bit to the left before accumulation, and so on.

[0069] like Figure 6 As shown, the in-memory FPGA architecture used for the computation of the fully connected layer distributes the weights of each fully connected factor in the fully connected layer sequentially to the SRAMs of the same row of the in-memory core (CIM core). The weights of different fully connected factors in the fully connected layer are distributed to different rows of SRAMs. Each input register pre-stores an 8-bit data of the input feature map.

[0070] like Figure 7 As shown, the in-memory FPGA architecture used for convolutional layer operations distributes the weights of all input channels at the same position in a convolutional kernel of a convolutional layer sequentially to multiple consecutive SRAMs in the same row of an in-memory kernel CIM_core. It iterates through different positions in a convolutional kernel, and distributes the weights of all input channels at all positions in a convolutional kernel to the same row of SRAMs in several in-memory kernels CIM_cores. For the input feature map, it slides through the convolutional kernel, and the activations of all input channels at the same position in the input feature map traversed by the convolutional kernel are sequentially distributed to each input register, with each input register storing an 8-bit activation.

[0071] like Figure 8 As shown, the output of the same row of SRAM in the in-memory core CIM_core needs to be accumulated by an accumulator and then quantized by a quantization module. In-memory cores in the same column share the value of the input vector. The accumulation by the accumulator and the quantization by the quantization module can both be implemented in the logic function block CLB, which passes the output of the same row of SRAM in the in-memory core CIM_core to the logic function block CLB for completion.

[0072] The allocation of in-memory kernels and network mapping will be explained using fully connected layers and convolutional layers as examples. The in-memory kernels mentioned above essentially perform matrix-vector product operations; the mapping process is essentially transforming a network layer into a matrix-vector product.

[0073] like Figure 6 As shown, this is a fully connected layer with 8 input channels and 4 output channels. The input activations are expanded into vectors, and the weights are expanded into weight matrices, as illustrated in the image. The chosen storage strategy is to store the values ​​of all channels for each fully connected factor in one row, with different fully connected factors stored in different rows. In fact, a fully connected layer can be considered a special convolutional layer. The input feature map is 1x1xIC, the output feature map is 1x1xOC, the convolutional kernel size is 1x1xIC, and the number of convolutional kernels is OC. Using in-memory computation to complete the mapping of a neural network is essentially a weight-stationary dataflow design. All data for the convolutional kernels is stored in in-memory kernels (CIM_cores), but the maximum number of input and output channels supported by each in-memory kernel (CIM_core) is fixed. The size of the weight matrix required for fully connected layers of different sizes and the size of each in-memory kernel (CIM_core) jointly determine the number of in-memory kernels (CIM_cores) allocated. The allocation process is as follows... Figure 8 As shown. According to Figure 8 The size of the weight matrix determines the size of W_NUM and H_NUM. W_NUM, H_NUM, and the maximum number of input channels and output channels supported by a single in-memory core (CIM_core) together determine the number of in-memory cores (CIM_cores) required for this layer. The outputs of in-memory cores in the same row need to be accumulated by an accumulator, and in-memory cores in the same column share the value of the input vector.

[0074] Compared to fully connected layers, convolutional layers use essentially the same method for mapping the weight matrix. The difference lies in that their input is a matrix, not a vector, and the matrix values ​​are obtained through the im2col transformation. Figure 7 This diagram illustrates the mapping process of a convolutional layer. It shows a 5x5x3 input feature map with 3x3 convolutional kernels, 3 kernels, and a stride of 1. After the transformation, since the in-store kernel (CIM_core) only supports matrix-vector products, the input matrix needs to be sequentially fed into the allocated CIM_core according to a specific time sequence.

[0075] Specific embodiments of the present invention are as follows:

[0076] To further illustrate the advantages of the proposed in-memory FPGA architecture, a 64x64 fully connected layer (64 input channels and 64 output channels) of the same size was mapped onto both the in-memory FPGA architecture and a general FPGA architecture. Placement and routing were then performed, and finally, power consumption was tested on both at the same clock frequency (100MHz). The in-memory core CIM_core was set to an array size of 64x256. The final mapping results are shown below. Figure 9 , Figure 10 As shown. Figure 9 and Figure 10 The input / output (IOB) modules are distributed on the periphery of the FPGA architecture. The logic function blocks (CLBs) are distributed internally within the FPGA architecture, each with a size of 1x1, and the memory blocks (BRAMs) are distributed internally within the FPGA architecture, each with a size of 1x6. Figure 10 The difference is, Figure 9 Two additional storage cores, CIM_core, are distributed, each with a size of 2x3.

[0077] Figure 11 and Figure 12 The power reports are shown for in-memory FPGA architecture and general FPGA architecture, respectively. The comparison reveals that the power consumption of the in-memory FPGA architecture is only 10.99mW, while the power consumption of the general in-memory FPGA architecture is 22.01mW. This means that for the same network size, the power consumption difference is more than double.

[0078] The comparative test of the simple network described above further validates the advantages of the proposed in-memory computing FPGA architecture. Using in-memory computing technology reduces data movement, lowers power consumption, and allows for network mapping on the FPGA, leveraging its reconfigurable capabilities to quickly support networks of various sizes.

Claims

1. A memory-based FPGA architecture, comprising input / output modules (IOBs), logic blocks (CLBs), and memory blocks (BRAMs); the IOBs are distributed on the periphery of the FPGA, the BRAMs are distributed inside the FPGA, and the CLBs are distributed throughout the FPGA, characterized in that: It also includes the storage and computing core CIM_core, which is used to replace some of the logical function blocks CLB; the storage and computing core CIM_core, storage blocks BRAM, and logical function blocks CLB are arranged in a row and column array and arranged in an island architecture. The in-memory computing core CIM_core mainly consists of an SRAM array, registers, an adder tree, a shift-add module (Muxand add), a quantization module, an address decoding circuit, and a write control circuit. The registers include multiple input registers and one output register. The SRAM array contains several SRAMs arranged in rows and columns. One SRAM is used to store one bit of data. The output terminals of the address decoding circuit are connected to the SRAMs in each row of the SRAM array. The output terminals of the write control circuit are connected to the SRAMs in each column of the SRAM array. The output port of the adder tree is connected to the output register after passing through the shift and add module Mux and add and the quantization module in sequence. In this configuration, the outputs of each SRAM in each column of the SRAM array are connected to one input of a corresponding NOR gate. The other input of each NOR gate is connected to the same input register via the same multiplexer (MUX). The output of each NOR gate is connected to one input port of an adder tree.

2. The in-memory FPGA architecture according to claim 1, characterized in that: The address decoding circuit receives address data, processes it to generate write line signals WWL, read line signals RWL, and read line inverted signals RWLB, and each write line signal WWL, read line signal RWL, and read line inverted signal RWLB is input into a row of SRAM in the SRAM array; Each SRAM outputs a read bit line signal RBL. The write control circuit generates write bit line signals WBL and write bit line inversion signals WBLB. Each write bit line signal WBL, read bit line signal RBL, and write bit line inversion signal WBLB is input into a column of SRAMs in the SRAM array. Each SRAM in the SRAM array is controlled by the write word line signal WWL, read word line signal RWL, read word line inverted signal RWLB, write bit line signal WBL, read bit line signal RBL, and write bit line inverted signal WBLB to output a one-bit value. Each input register stores multiple-bit values. The multiple-bit values ​​in the input register are selected by the multiplexer MUX and sent to a NOR gate. Together with a one-bit value output by one SRAM in the SRAM array, they are NORed to obtain a one-bit output value. The bit values ​​output by each SRAM in each column of the SRAM array are connected in the order of each SRAM in the column to form a multi-bit array value. The multi-bit array value is input to an input port of the adder tree. The output port of the Adder Tree outputs all the multi-bit array values ​​of the SRAM array. After arranging them, the shift and add module Mux and add them to obtain a (inconsistent) multi-bit shifted value. Then, the multi-bit shifted value is processed by the quantization module to extract the consecutive multi-bit outputs and store them in the output register.

3. The in-memory FPGA architecture according to claim 1, characterized in that: Each SRAM in the SRAM array outputs a single bit value under the control of the write word line signal WWL, read word line signal RWL, read word line inverted signal RWLB, write bit line signal WBL, read bit line signal RBL, and write bit line inverted signal WBLB. The write word line signal WWL, write bit line signal WBL, and write bit line inverse signal WBLB are used to control the write operation of the SRAM array: when the write word line signal WWL of a certain row is 1, a write operation is performed on all SRAM cells in that row, and the weight of the write is determined by the write bit line signal WBL and the write bit line inverse signal WBLB of that SRAM cell. The read word line signal RWL, the read word line inverse signal RWLB, and the read bit line signal RBL are used to control the read operation of the SRAM array: when the read word line signal RWL of a certain row is 1 and the read word line inverse signal RWLB is 0, a read operation is performed on all SRAM cells in that row, and the read weight is read out through the read bit line signal RBL of each SRAM cell and sent out of the SRAM array.

4. The in-memory FPGA architecture according to claim 1, characterized in that: Each SRAM is mainly composed of 10 MOS transistors. The gates of MOS transistors M0, M5, and M8 are connected to the write word line signal WWL. The source of MOS transistor M0 is connected to the write bit line signal WBL. The drains of MOS transistors M0, M1, and M2, and the gates of MOS transistors M3 and M4 are connected together. The sources of MOS transistors M1, M3, and M6 are connected to voltage, and the sources of MOS transistors M2, M4, and M7 are grounded. The drains of transistors M5, M3, and M4, as well as the gates of transistors M1, M2, M6, and M7, are connected together. The source of transistor M5 is connected to the write bit line inverted signal WBLB. The drains of transistors M6 through M9 are connected together. The sources of transistors M8 and M9 are connected together and then connected to the read bit line signal RBL. The gate of transistor M8 is connected to the read word line signal RWL. The gate of transistor M9 is connected to the read word line inverted signal RWLB.

5. The in-memory FPGA architecture according to claim 1, characterized in that: The Adder Tree is composed of multiple Ripple Carry Adders (RCAs). Each Adder Tree receives a multi-bit array value from each of its input ports, aligns and sorts these values ​​according to the input port order, and performs multiple carry additions through the RCAs to obtain the final multi-bit value. In each carry-add process, all the current multi-bit values ​​are grouped into pairs of adjacent multi-bit values ​​and added together by a ripple carry adder (RCA) to obtain a single multi-bit value. If the number of current multi-bit values ​​is odd, the remaining single multi-bit value is not processed in the current carry-add process but is directly passed to the next carry-add process.

6. The in-memory FPGA architecture according to claim 1, characterized in that: The addition tree consists of multiple carry-holding adders (CSA), multiple half-adders (HA), and a ripple carry adder (RCA). The Adder Tree receives a multi-bit array value from each of its input ports. It then aligns and sorts these multi-bit array values ​​as multi-bit values ​​according to the input port order. Multiple carry-preserving adders (CSA) and half-adders (HA) are used to perform multiple carry-add operations until the number of multi-bit values ​​is reduced to only two. In each carry-in addition process, each bit is traversed as a processing bit. First, every three consecutive values ​​under the processing bit are grouped together and added using a carry-holding adder (CSA) to obtain a value at the original processing bit and a value at the carry-over bit. If two values ​​remain in the processing bit after processing with the carry-holding adder (CSA), then these two values ​​are added using a half-adder (HA) to obtain a value at the original processing bit and a value at the carry-over bit. If only one value remains in the processing bit after processing with the carry-holding adder (CSA), then this value is not processed in the current carry-in addition process and is directly passed to the next carry-in addition process. After the number of multi-digit values ​​is reduced to only two, a ripple carry adder (RCA) is used to add the two multi-digit values ​​to obtain the final multi-digit value.

7. The in-memory FPGA architecture according to claim 1, characterized in that: The shift and add module Mux and add process specifically shifts and accumulates the multi-bit values ​​output by the adder tree to obtain the final unquantized result.

8. The in-memory FPGA architecture according to claim 1, characterized in that: The in-memory FPGA architecture described above is used for computation in the fully connected layer. It sequentially distributes the weights of each fully connected factor in the fully connected layer to the SRAMs of the same row of the in-memory core (CIM core). The weights of different fully connected factors in the fully connected layer are distributed to different rows of SRAMs. Each input register pre-stores the data of one channel of the input feature map.

9. The in-memory FPGA architecture according to claim 1, characterized in that: The in-memory FPGA architecture used for convolutional layer operations distributes the weights of all input channels at the same position within a convolutional kernel in a convolutional layer sequentially into multiple consecutive SRAMs in the same row of a CIM core. It iterates through different positions within a convolutional kernel, and distributes the weights of all input channels at all positions within a convolutional kernel into the same row of SRAMs in the CIM core. For the input feature map, it slides through the convolutional kernel, and the activations of all input channels at the same position covered by the kernel are sequentially distributed into various input registers, with each input register storing an 8-bit activation.

Images