Neural network near-memory computing system

Abstract

The present application relates to the technical field of computer, and especially relates to a neural network near-memory computing system. The system comprises at least two memory-computing modules, an image storage and broadcasting module and a controller module. The image storage and broadcasting module is used for buffering an externally input original image and broadcasting the original image to each memory-computing module. Each memory-computing module is used for distributedly storing weight data of a neural network, distributedly storing feature map data obtained by neural network computation, and performing computation in the neural network based on the weight data, the original image and the feature map data. The controller module is used for controlling the operation of each memory-computing module and the operation of the image storage and broadcasting module. The present application is used to solve the defects of low system energy efficiency and low utilization rate of computing resources in the prior art when the neural network computation is completed by applying the near-memory technology.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a neural network near-memory computing system. Background Technology

[0002] Deep neural networks currently play a crucial role in many fields, including computer vision, natural language processing, and personalized recommendation systems for websites. Depending on their structure, commonly used neural networks include convolutional neural networks, recurrent neural networks, and graph neural networks, among others.

[0003] With the significant increase in task complexity, the proliferation of application scenarios, and the improvement in inference accuracy, the computational demands of neural networks have also increased dramatically. The current need for massive storage overhead and massive computational demands has exacerbated the "memory wall" problem, posing new challenges to the computing and storage capabilities of hardware. The "memory wall" occurs when the clock frequency of a microprocessor in the von Neumann architecture grows far faster than the memory frequency. Currently, a feasible and effective solution to the "memory wall" problem is in-memory computing. In-memory computing architectures mainly include off-memory structures, near-memory structures, and on-memory structures.

[0004] When applying the more advantageous near-memory architecture to neural network computation, existing technologies employ a distributed weight storage and computation structure with centralized feature map storage. However, this centralized feature map storage structure results in high energy consumption for data movement. The input feature map data required for each tile computation needs to be read from and broadcast from the central module, and the output feature map generated by the computation also needs to be moved to the central module for storage. This leads to even greater energy consumption for data movement, resulting in low system energy efficiency and low utilization of computing resources for the entire system. Summary of the Invention

[0005] This invention provides a near-memory computing system for neural networks, which addresses the shortcomings of existing technologies that use near-memory technology to perform neural network calculations, such as low system energy efficiency and low utilization of computing resources.

[0006] This invention provides a near-in-memory computation system for neural networks, comprising at least two in-memory computation modules, an image storage and broadcasting module, and a controller module. The image storage and broadcasting module is used to cache externally input raw images and broadcast the raw images to each of the in-memory computation modules. Each of the in-memory computation modules is used to distribute and store the weight data of the neural network, and to distribute and store the feature map data calculated by the neural network, and to perform calculations in the neural network based on the weight data, the raw images, and the feature map data. The controller module is used to control the operation of each of the in-memory computation modules and to control the operation of the image storage and broadcasting module.

[0007] According to the present invention, a near-in-memory computing system for neural networks includes at least one convolutional layer, each in-memory module corresponding to at least one convolutional layer, and each convolutional layer including at least one convolutional kernel; each in-memory module includes a weight memory, a feature map memory, a computing array, a routing system, and a data communication line; for each convolutional layer corresponding to each in-memory module: the weight data of each convolutional kernel is stored on different bit lines in the weight memory, and the feature map data of each channel is stored on different bit lines in the feature map memory; the computing array is used to perform convolution and nonlinear calculations in the neural network; the routing system is used for data transmission with adjacent in-memory modules; the data communication line is used for data transmission between the weight memory and the computing array, and for data transmission between the feature map memory and the computing array.

[0008] According to a neural network near-memory computing system provided by the present invention, the controller module includes a memory controller corresponding to each of the memory modules; the computing array includes at least one computing unit; the data communication line includes a feature map transmission bus; the memory controller is configured to sequentially perform the following operations on each word line in the feature map memory: reading the feature map data on a word line based on the feature map transmission bus; broadcasting the read feature map data on a word line to each computing unit based on the routing and the feature map transmission bus, and transmitting the read feature map data on a word line to the adjacent memory module through the routing; updating the feature map memory with new feature map data output by the computing unit based on the feature map transmission bus.

[0009] According to a neural network near-memory computing system provided by the present invention, the controller module includes a near-memory controller corresponding to each of the near-memory modules; the computing array includes at least one computing unit, wherein the computing unit corresponds one-to-one with the channel in the convolutional layer of the storage module; the data communication line includes a feature map transmission line corresponding to each of the computing units; the near-memory controller sequentially performs the following operations on each word line in the feature map data: based on the feature map transmission line, read the feature map data corresponding to each channel on a word line, and transmit the feature map data to the corresponding computing unit; based on the feature map transmission line corresponding to each channel, store the new feature map data output by each computing unit into the feature map memory.

[0010] According to a neural network near-memory computing system provided by the present invention, the computing array includes at least one computing unit; each word line in the weight memory stores the weight data of at least one convolutional kernel; one convolutional kernel corresponds to E computing units, where E is a positive integer; the feature map data output by the E computing units is stored in the feature map memory based on E channels.

[0011] According to a neural network near-memory computing system provided by the present invention, a word line in the weight memory stores weight data corresponding to at least one convolutional kernel, wherein one convolutional kernel corresponds to at least one computing unit; a word line in the feature map memory stores feature map data corresponding to at least one channel, wherein the number of channels is consistent with the number of convolutional kernels; the in-memory computing controller is used to control the synchronous operation of each computing unit; the computing unit is used to generate new feature map data based on the weight data corresponding to the convolutional kernel and the feature map data.

[0012] According to a neural network near-memory computing system provided by the present invention, the feature map memory includes at least two preset latches; the latches are used to cache the feature map data read from the feature map memory; the computing controller is used to broadcast the feature map data in the latches to each of the computing units based on the feature map transmission bus.

[0013] According to a neural network near-memory computing system provided by the present invention, the computing unit includes at least one intermediate result register; the storage controller includes a computing controller; the latch is further configured to cache intermediate result data generated by convolution calculation in the computing unit; the intermediate result register is configured to cache intermediate result data generated by convolution calculation in the computing unit; the computing controller is configured to respectively count the caching duration of unupdated data in the latch and each of the intermediate result registers, and cache the intermediate result data to the latch or the intermediate result register with the longest caching duration each time the intermediate result data is generated by convolution calculation.

[0014] According to a neural network near-memory computing system provided by the present invention, the in-memory controller is configured to write back the read weight data to the weight memory when reading the weight data from the weight memory for the first time during the weight data refresh cycle.

[0015] According to a neural network near-memory computing system provided by the present invention, each of the computing modules corresponds to a weight memory, which stores the weight data of different convolutional layers in the neural network; the computing controller is used to read the block weight data corresponding to the convolutional layers in the weight memory, wherein the block weight data corresponds to the block images pre-divided in the original image; each computing module corresponds to a computing array, which performs calculations corresponding to different convolutional layers in the neural network based on the block weight data; each computing module corresponds to a feature map memory, which stores the feature map data calculated by different convolutional layers in the neural network, wherein the feature map data obtained by each computing array in one convolution calculation corresponds to the block image.

[0016] The present invention provides a neural network near-memory computing system, comprising at least two in-memory computing modules, an image storage and broadcasting module, and a controller module. The image storage and broadcasting module is used to cache the original image input from the external source and broadcast the original image to each in-memory computing module. Each in-memory computing module is used for distributed storage of the neural network's weight data and distributed storage of the feature map data calculated by the neural network, as well as for performing calculations in the neural network based on the weight data, the original image, and the feature map data. The controller module is used to control the operation of each in-memory computing module and the operation of the image storage and broadcasting module. By setting at least two in-memory computing modules and distributively storing the weight data and feature map data, multiple reads of feature map data are avoided, saving energy consumption from data movement. Simultaneously, the calculated output feature maps can also be stored separately in each in-memory computing module, further reducing energy consumption from data movement. Each in-memory computing module can synchronously complete calculations in the neural network, improving the neural network's computational efficiency. Therefore, this neural network near-memory computing system improves system energy efficiency and computational resource utilization. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of the neural network near-memory computing system provided by the present invention;

[0019] Figure 2 This is a schematic diagram of the storage and computing module structure provided by the present invention;

[0020] Figure 3This is a schematic diagram of the routing network connection relationship between the in-memory computing modules provided by the present invention;

[0021] Figure 4 This is a schematic diagram illustrating the operation principle of the storage-based computing module provided by the present invention in data broadcast mode;

[0022] Figure 5 This is a schematic diagram of the operation principle in the bitline-level read mode provided by the present invention;

[0023] Figure 6 This is a schematic diagram illustrating the convolution calculation principle under a 4x expansion of computing resources provided by the present invention;

[0024] Figure 7 This is a schematic diagram of the neural network convolutional layer computation structure provided by the present invention;

[0025] Figure 8 This is a schematic diagram of the internal structure of the computing unit provided by the present invention;

[0026] Figure 9 This is a schematic diagram of the neural network layer-by-layer convolution principle and pyramid structure provided by the present invention;

[0027] Figure 10 This is a detailed structural diagram of the neural network near-memory computing system provided by the present invention;

[0028] Figure 11 This is a schematic diagram of the chip layout provided by the present invention. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0030] First, we will further analyze the existing technologies. With the significant increase in the complexity of neural network tasks, the expansion of application scenarios, and the improvement of inference accuracy, the computational load required by neural networks has also increased dramatically. Taking the classic VGG-19 convolutional neural network (CNN) as an example, it requires 20G floating-point operations with weight parameters on the order of 150M. The current demands for large-scale storage overhead and massive computational load have exacerbated the "memory wall" problem, posing new challenges to the computing and storage capabilities of hardware.

[0031] The "memory wall" arises because, under the von Neumann architecture, the rate of increase in microprocessor clock frequency far outpaces the rate of increase in memory frequency, with Dynamic Random Access Memory (DRAM) serving as an example. Microprocessor production lines prioritize providing faster transistors for faster logic and more metal wiring layers for reduced power consumption. DRAM production lines prioritize achieving smaller DRAM cell areas to increase DRAM capacity and lower leakage current to extend DRAM refresh cycles. Therefore, processor clock frequencies and performance increase at an unimaginable pace; however, memory access speeds (primarily DRAM) grow much more slowly, widening the gap between processors and memory. The speeds of computation, storage, and input / output (I / O) become increasingly mismatched, making balanced architecture design increasingly difficult. The "memory wall," comprised of communication bandwidth and latency, becomes the biggest obstacle to improving system performance, with significant structural complexity spent on addressing data access latency. While on-chip data caching and prefetching can help reduce average memory access time, they do not fundamentally solve the problem. The development of high-performance computing requires overcoming the memory wall issue.

[0032] Currently, a feasible and effective solution to the "memory wall" problem is in-memory computing. From this perspective, we can divide neural network accelerators into two main categories: in-memory computing (including in-memory computing and near-memory computing) and traditional storage-computation separation. Traditional artificial intelligence (AI) accelerators use a traditional off-chip memory and on-chip cache and computing module structure, which we can call an off-chip structure. If the computing unit is located after the sensitive amplifier of the memory array, avoiding the power consumption and latency of memory I / O, and the data is read directly through the local bus of the memory, it is called a near-memory structure. If the computing unit is located before the sensitive amplifier of the memory array, this type of structure is called an in-memory structure.

[0033] The three structures of external storage, near storage, and internal storage are compared from multiple perspectives.

[0034] (1) From the perspective of storage density, the research on memory structure is relatively mature. Modifications to standard memory cells and memory peripherals (decoding circuits, etc.) often lead to a decrease in memory storage density and an increase in power consumption. Therefore, off-memory structures do not require modification of the memory and have an advantage in this regard; on-memory structures, modifications to memory cells and peripherals greatly reduce the storage density of the memory itself and increase the difficulty of its commercialization; while near-memory computing only modifies the peripherals of the memory, improves the bandwidth, and has a storage density between off-memory and on-memory structures.

[0035] (2) From the perspective of data read bandwidth, off-memory structures use traditional memory to store weighted data, which limits data read bandwidth. For example, DRAM storage capacity is typically 1 gigabyte (GB) to 16GB, and its DDR3 interface has a data bandwidth of 12.8 gigabits (GBps). In contrast, near-memory structures greatly improve memory data bandwidth, with 3D-Hybrid Bonding achieving a data bandwidth of 38.4 GBps / 1Gb. On-memory structures directly endow the memory with multiplication and addition operations, allowing all weighted data to participate in convolution operations simultaneously, maximizing data read bandwidth.

[0036] (3) From an energy efficiency perspective, in-memory computing has the highest energy efficiency. In terms of data transfer energy consumption between computing units and memory, in-memory architecture has the lowest data transfer energy consumption because its computing units and memory units are tightly integrated. Near-memory architecture has the second highest data transfer energy consumption. Off-memory architecture has the highest data transfer energy consumption, including I / O energy consumption and the energy consumption of the transmission lines between the two chips. In summary, in-memory architecture is currently considered the most energy-efficient neural network computing architecture.

[0037] (4) Although in-memory architecture has significant advantages in energy efficiency and bandwidth, its low computational flexibility due to the special structure designed for multiply-accumulate calculations, and the low reliability and low accuracy of analog domain calculations are problems that cannot be ignored. In addition, the in-memory architecture modifies mature memory at the storage cell level, which also increases the difficulty of its commercialization.

[0038] Therefore, taking all the above considerations into account, we believe that the near-memory architecture is a reasonable compromise between off-memory and on-memory architectures, which improves the system's energy efficiency and performance without changing the storage cell structure and ensuring high reliability.

[0039] Close-memory architectures can be divided into two categories: those based on embedded dynamic memory and those based on dynamic memory. Since the storage density of embedded DRAM is much lower than that of DRAM, the networks they are suitable for differ. Embedded DRAM close-memory architectures are mostly used for computing lightweight networks or a few layers of a large neural network, with a single chip's storage capacity typically in the 10 MB range. DRAM close-memory architectures are mostly used for large convolutional neural networks and fully connected networks, such as fully connected layers and multilayer perceptrons (MLPs), with a single chip's storage capacity typically in the 100 MB range.

[0040] To achieve high-density storage, DRAM requires specialized manufacturing processes to create the stacked capacitors within the memory cells, making it incompatible with logic processes. Integrating logic cells directly onto the DRAM chip results in significantly higher power consumption—approximately three times—compared to standard Complementary Metal Oxide Semiconductor (CMOS) processes using the same technology. Embedded DRAM (eDRAM), on the other hand, can utilize deep trench capacitors that do not affect CMOS performance, or a 2T structure that eliminates the need for additional capacitor manufacturing. Therefore, embedded DRAM is better suited for integration with logic cells onto the same chip.

[0041] DaDianNao is a machine learning supercomputer. Utilizing a near-memory architecture based on embedded dynamic memory, DaDianNao provides ample bandwidth for weight data reads. Its 64-node multi-chip interconnect system achieves a 656.63x speedup compared to a Graphics Processing Unit (GPU) and a 184x reduction in power consumption. A DaDianNao node consists of 16 tiles and a central block located in the middle of the chip. The central block includes eDRAM for storing feature map data and routing modules for broadcasting the feature map data to the 16 tiles. Each tile consists of eDRAM for weight data storage and a Neural Functional Unit (NFU) module for performing convolution calculations.

[0042] DaDianNao employs a distributed weight storage and computation architecture with centralized feature map storage, but it still has drawbacks. First, centralized feature map storage leads to high energy consumption for data movement. The input feature map data required for each tile's computation must be read from and broadcast from the central module, and the resulting output feature map also needs to be moved to the central module for storage, resulting in even greater energy consumption for data movement. Second, its feature map read / write bandwidth is small. The limited feature map bandwidth prevents this architecture from supporting convolutional operations that require significant feature map read / write bandwidth (such as depthwise separable convolutional layers) without feature map reuse. Third, DaDianNao does not consider weight reuse; its eDRAM reads weights frequently, resulting in extremely high system energy consumption and significant room for energy efficiency optimization.

[0043] Based on the above analysis, this invention provides a neural network near-memory computing system. The following is in conjunction with... Figures 1 to 11 The present invention describes a neural network near-memory computing system.

[0044] In one embodiment, such as Figure 1 As shown, the near-in-memory computing system for neural networks includes at least two in-memory computing modules, an image storage and broadcasting module, and a controller module, where P is the number of in-memory computing modules. The image storage and broadcasting module is used to cache the original image input from the external source and broadcast the original image to each in-memory computing module. Each in-memory computing module is used to distribute and store the weight data of the neural network, as well as the feature map data calculated by the neural network, and to perform calculations in the neural network based on the weight data, the original image, and the feature map data. The controller module is used to control the operation of each in-memory computing module and to control the operation of the image storage and broadcasting module.

[0045] In this embodiment, the Computation and Memory Block (CMB) is used as the basic module of the neural network near-memory computing system. At least two CMBs are set in the neural network near-memory computing system; preferably, one neural network near-memory computing system includes eight CMBs. Each CMB has the same structure and can work independently, thus forming a distributed arrangement of CMBs. The CMBs can work in parallel, thereby improving computational efficiency. The image storage and broadcasting module includes an interface for data interaction with the outside of the neural network near-memory computing system. Through this interface, the weight data of the neural network can be pre-written into the neural network near-memory computing system, and it can also receive raw images input from the outside, which are then processed by the various CMBs. The controller module is responsible for controlling the neural network near-memory computing system to operate according to preset rules.

[0046] In one embodiment, the neural network includes at least one convolutional layer, each in-memory module corresponds to at least one convolutional layer, and each convolutional layer includes at least one convolutional kernel.

[0047] Each storage module includes a weight memory, a feature map memory, a computation array, routing, and data communication lines. For each convolutional layer corresponding to a storage module: the weight data of each convolution kernel is stored on different bit lines in the weight memory, and the feature map data of each channel is stored on different bit lines in the feature map memory. The computation array is used for convolution and nonlinear calculations in the neural network. Routing is used for data transmission with adjacent storage modules. Data communication lines are used for data transmission between the weight memory and the computation array, as well as between the feature map memory and the computation array.

[0048] In this embodiment, the internal structure of the in-memory computing module is further configured to suit the structural characteristics of the neural network. Specifically, each in-memory computing module includes a weight memory, a feature map memory, a computing array, routing, and data communication lines. Preferably, the weight memory uses eDRAM, and the weight memory in the i-th in-memory computing module is denoted as eDRAMi for Weight, or eDRAMi_WT for short, where i is an integer greater than or equal to 0; the feature map storage area also uses eDRAM, and the feature map memory in the i-th in-memory computing module is denoted as eDRAMi for Feature Map, or eDRAMi_FM for short. The weight data is stored near its allocated computing resources, that is, in the weight memory near its allocated computing array (Processing Element Array, PEA), to minimize weight data movement and reduce the time and energy consumption required for data movement. Each computing array corresponds to an eDRAMi_WT that stores the weight data, and the eDRAMi_WT provides sufficient weight bandwidth for convolution calculation. Each computing array corresponds to an eDRAMi_FM that stores feature map data. Therefore, eDRAMi_FM can provide sufficient bandwidth so that the calculated feature map data can be written to eDRAMi_FM simultaneously, thereby reducing the time required for data writing and the energy consumption for moving data.

[0049] In this embodiment, to achieve a distributed storage and distributed computing structure for weight data and feature map data, the Computation and Memory Block (CMB) is defined as the basic module of this architecture. Preferably, the neural network near-memory computing system includes eight CMBs, one image storage and broadcasting module, and a controller module. Figure 2The schematic diagram of the storage and computation module shows that each module has a 4MB capacity weight memory with a read / write bandwidth of 2048 bits (256 bytes). During the initialization of the neural network near-memory computation system, the weight data corresponding to the neural network is written to the weight memory. After the neural network near-memory computation system enters the working state (inference operation), the weight data stored in the weight memory remains unchanged; no more data is written to the weight memory externally, and only the required weight data is read from the weight memory for inference. The weight memory stores the weight data of several neural network layers allocated by the CMB. The weight data of different convolutional layers is stored on different word lines. Since each weight data bit width is 8 bits, each convolutional kernel in the convolutional layer is stored on 8 word lines (k*k*N+1), where k*k*N is the weight data amount of one convolutional kernel, and 1 is the bias data amount of each convolutional kernel. During convolution operations, the weight data of 256 convolution kernels are read simultaneously from one word line of eDRAMi_WT, and the 256 convolution kernels are computed in parallel by 256 computing units.

[0050] The feature map memory has a capacity of 0.5MB and a read / write bandwidth of 2048 bits (256 bytes). The feature map memory module is not active during the initialization of the neural network near-memory computing system. After the neural network near-memory computing system enters the working state (inference operation), it is used to store the feature map data of each layer of the neural network. eDRAMi_FM stores the feature map data of several layers of the neural network required for computation. Data for one channel of each layer's feature map is stored on 8 bit lines (because the feature map data bit width is 8 bits), therefore, 2048 bit lines can store 256 channels of feature map data.

[0051] Each in-memory module contains one computation array, and each computation array includes 258 computation units (PE units). The computation array is used to perform convolution and nonlinear operations. Figure 2 In China, PE i，0 PE i，1 , ..., PE i，254 PE i，255These represent the 256 computational units in the computation array of the i-th in-memory module. The total number of PE units in the neural network near-memory computation system is 2048. PE units can perform 8-bit multiplication, 16-bit addition, max pooling, and ReLU activation operations. PE units perform output stationary computation. Output stationary means that the PE unit continuously performs multiplication and accumulation operations until the final result of this convolution (i.e., feature map data) is obtained, and the final result is written to eDRAMi_FM. The intermediate result data (patial sum, psum) of each multiplication and accumulation is stored in a 16-bit register configured inside the PE unit.

[0052] In this embodiment, the eight in-memory computing modules on the neural network near-memory computing system can work independently. The eight in-memory computing modules can work together to process a certain layer of the neural network, or they can process multiple layers of the neural network. For example, each in-memory computing module processes one layer, which makes the mapping from algorithm to hardware more flexible.

[0053] In this embodiment, each in-memory computing module is configured with a corresponding router, and feature map data is transmitted between any two in-memory computing modules via the router and data communication lines. The bit width of the routed data transmission is 64 bits. When the neural network near-memory computing system includes 8 in-memory computing modules, the routing network connection relationship between the in-memory computing modules is as follows: Figure 3 As shown, route 0 corresponds to the 0th in-memory module, route 1 corresponds to the 1st in-memory module, and so on. There is a time delay of 1 clock cycle between two adjacent in-memory modules. Therefore, in the neural network near-memory computing system, the maximum delay caused by routing between any two in-memory modules is 4 clock cycles (for example, there is a 4 clock cycle delay in data transmission between route 0 and route 7).

[0054] The feature map memory in the storage module of this invention supports two read modes: read mode one is broadcast mode, and read mode two is bitline level read.

[0055] In one embodiment, when using broadcast mode, the controller module includes a memory controller corresponding to each memory module; the computing array includes at least one computing unit; and the data communication line includes a feature map transmission bus.

[0056] The in-memory controller is used to perform the following operations sequentially on each word line in the feature map memory: read the feature map data on a word line based on the feature map transmission bus; broadcast the read feature map data on a word line to each computing unit based on routing and the feature map transmission bus, and transmit the read feature map data on a word line to the adjacent in-memory module through routing; update the feature map memory with new feature map data output by the computing unit based on the feature map transmission bus.

[0057] In this embodiment, the in-memory controller corresponding to the in-memory computing module controls the broadcast mode process. Specifically, in broadcast mode, 2048 bits of feature map data are read from a word line of the feature map memory and broadcast through the feature map transmission bus and / or routing. The feature map data can be broadcast to the local computing array of the in-memory computing module or to other in-memory computing modules. After several clock cycles, all 2048 bits of feature map data are broadcast and computed, and then the eDRAM performs the next read operation on eDRAMi_FM.

[0058] In one example, taking the 9th convolutional layer of the lightweight neural network MobileNet as an example, the computation process of the in-memory module in data broadcast mode is as follows: Figure 4 As shown:

[0059] ①: Based on Figure 4 The dashed arrows in the diagram indicate that the 9th convolutional layer of the MobileNet network has 256 convolutional kernels. The weight data of each convolutional kernel is stored in eDRAM_WT as shown in the above embodiment, with each kernel's weight data stored on 8 bit lines of the weight memory. The weight data of one convolutional kernel is 1*1*128, and there is also an 8-bit bias data, so the weights of the 9th convolutional layer are stored on 129 word lines. The 256 PE units of the in-memory computing module correspond one-to-one with the 256 convolutional kernels. Each convolutional kernel is calculated by one PE unit, so one in-memory computing module can compute 256 convolutional kernels in parallel. Since the 256 PEs perform identical processing, differing only in input data, their control signals are consistent and can be controlled by the same in-memory controller, thereby reducing the overall power consumption of the controller module.

[0060] ②: Based on Figure 4The solid lines in the diagram connect the 256 PE units each clock cycle via the feature map transmission bus and routing, sending 8 bits of feature map data. Specifically, the feature map transmission bus includes a first bus and a second bus. Feature map data is read from the feature map memory through 128 channels, then transmitted via the first bus to the local router. The router then transmits the read feature map data to the second bus, and finally, the feature map data from the 128 channels is input to each PE unit; alternatively, the router transmits the read feature map data to the routing of adjacent in-memory computing modules.

[0061] Every 128 clock cycles, the feature map transmission bus can input features. Figure 2 In a 128-channel system at a single point in 3D space, each PE (Preprocessor) unit continuously performs multiplication and accumulation operations, resulting in 1*1*128 multiplication and accumulation operations. In the 129th clock cycle, eDRAM_WT reads the bias data, which is then accumulated by the PE unit, and an activation operation is performed, outputting an 8-bit final result.

[0062] ③: Based on Figure 4 The solid arrow in the image indicates that the 8-bit final result (new feature map data) generated simultaneously by 256 PEs is written to eDRAM_FM, meaning that the output feature maps of 256 channels are stored on the same word line of eDRAM_FM.

[0063] In one embodiment, when using bitline-level read mode, the controller module includes a memory controller corresponding to each memory module; the computing array includes at least one computing unit, wherein the computing unit corresponds one-to-one with the channel in the convolutional layer of the storage module; and the data communication line includes a feature map transmission line corresponding to each computing unit.

[0064] The memory controller performs the following operations on each word line in the feature map data in sequence: based on the feature map transmission line, it reads the feature map data corresponding to each channel on a word line and transmits the feature map data to the corresponding computing unit; based on the feature map transmission line corresponding to each channel, it stores the new feature map data output by each computing unit into the feature map memory.

[0065] In this embodiment, in bitline-level read mode, after reading 2048 bits (256 bytes) of data from a word line from the feature map memory, it is directly input into 256 PE units one by one for bitline-level read without the need for bus broadcast.

[0066] In one example, taking the 10th convolutional layer of the MobileNet network as an example, the computation process of the in-memory module in bitline-level read mode is as follows: Figure 5 As shown:

[0067] ① The 10th convolutional layer (data warehouse layer, DW layer) of the MobileNet network has 256 convolutional kernels. The weight data of each convolutional kernel is stored in eDRAM_WT as shown in the above embodiment. The weight data of each convolutional kernel is stored on 8 bit lines of the weight memory. The weight data of one convolutional kernel is 3*3*1. In addition, there is 1 8-bit bias data. Therefore, the weight data of the 10th convolutional layer is stored on 10 word lines. The 256 PE units of the in-memory computing module correspond one-to-one with the 256 convolutional kernels. Each convolutional kernel is calculated by one PE unit. Therefore, one in-memory computing module can compute 256 convolutional kernels in parallel.

[0068] ②: Compared to conventional convolutional layers, the DW layer differs in that its 256 convolutional kernels convolve with each of the 256 channels of the feature map, eliminating the reuse caused by multiple kernels. That is, the feature map data in the i-th channel only needs to be convolved with the i-th convolutional kernel; the feature map data does not need to be broadcast to all 256 PE units. Feature map data read from any channel is directly input into the corresponding PE unit. Every 9 clock cycles, a multiply-accumulate result of 3*3*1 times can be calculated. In the 10th clock cycle, eDRAM_WT reads the bias data, which is then accumulated by the PE units, and an activation operation is performed, outputting an 8-bit final result.

[0069] ③: The 8-bit final result (new feature map data) generated simultaneously by 256 PE units is written to eDRAM_FM through 256 channels respectively, that is, the feature map data output by 256 channels is stored on the same word line of eDRAM_FM.

[0070] The two reading modes described above enable the in-memory computing module in this paper to support conventional convolutional neural networks in broadcast mode and to support the computation of deep separable convolutional neural networks commonly found in lightweight networks in bitline-level reading mode, thereby increasing the flexibility and adaptability of the near-memory computing system for neural networks.

[0071] In one embodiment, the computing array includes at least one computing unit; each word line in the weight memory stores weight data of at least one convolutional kernel; one convolutional kernel corresponds to E computing units, where E is a positive integer; the feature map data output by the E computing units is stored in the feature map memory based on E channels.

[0072] In this embodiment, to improve the utilization of computing resources, dynamic allocation of resources is achieved through computing resource expansion. Preferably, in the case that the neural network near-in-memory computing system includes 8 in-memory modules, for a convolutional layer with M kernels, we can allocate M, 2M, 4M or 8M PE units to its convolution process, that is, support the expansion of computing resources by 1, 2, 4 or even 8 times, thereby dynamically adjusting the amount of computing resources allocated to the layer.

[0073] The E-fold expansion of computational resources refers to the number of PEs allocated to each convolutional kernel being E. With this E-fold expansion, each convolutional kernel in the in-memory module is stored in 8E bit lines of the weight memory and allocated E PE units for convolutional computation. If the feature map data uses read mode one, i.e., broadcast mode, then E feature map data points are broadcast to the computation array of the in-memory module every clock cycle, allowing one in-memory module to perform parallel computations. Each convolutional kernel. The feature map data for each channel in the feature map memory is stored on 8E bit lines.

[0074] In this embodiment, a weighted local bus is configured in each of the E PE units for transmitting weighted data when the resources are expanded by a factor of E.

[0075] In one example, taking a fourfold increase in computing resources, the principle of convolution calculation is as follows: Figure 6 As shown, Figure 6 Taking the computational principle of a single convolutional kernel as an example, each convolutional kernel is stored in 4 bytes. Figure 6 In this architecture, the weight data of convolutional kernel Kernel 1 is stored on 4*1 byte bit lines. The weight data read from each convolutional kernel needs to be transferred to four PE units via the weight local bus for multiplication and addition operations. Specifically, Figure 6 The section consisting of multipliers, adders, a selection register (demux), two 16-bit intermediate result registers (16bREG), a multiplexer (mux), and a non-linear activation unit represents a specific implementation of the internal structure of the PE unit. The four PE units receive different feature map inputs, which are simultaneously transmitted via a feature map transfer bus. The convolution results calculated by the PE units are written bitline-level into the feature map memory. Figure 6In this diagram, OF(x,y,m), OF(x+1,y,m), OF(x+2,y,m), and OF(x+3,y,m) represent the feature map data output by the four PE units after simultaneous convolution calculations. The feature map data output by the four PE units is written to the feature map memory through their respective channels (channel 1). The feature map data storage organization principle is as follows: several channels of the feature map are stored on a single word line of the feature map memory, and each channel stores four consecutive values. This ensures that when the feature map data is read, four data points are read from each channel. To conserve energy when reading from the feature map memory, it is desirable to ensure that as much of the feature map data read from a single word line as possible can participate in the convolution calculation.

[0076] When computing resources are quadrupled, a convolutional kernel is allocated to four PE units for computation. The weight data storage organization principle is that the data of a convolutional kernel is stored on 4*8 bit lines of the weight memory. Four weight values ​​are read from one word line at a time from the weight memory and stored in the latches configured locally in the weight memory. Each clock cycle, one of the four weight values ​​is transferred to the four PE units through the weight local bus for multiplication and addition.

[0077] In this invention, since feature map data and weight data are reused multiple times in neural networks, the same data needs to be read repeatedly. Therefore, by increasing the number of times feature map data and weight data are reused, the data obtained from the feature map memory and weight memory can be reused multiple times, thereby reducing the number of read and write operations of the feature map memory and weight memory, and thus effectively reducing the energy consumption of reading and writing of the feature map memory and weight memory.

[0078] Specifically, the reuse of feature map data includes two cases: reuse caused by multiple convolutional kernels in a convolutional layer, and reuse caused by overlap.

[0079] In one embodiment, a word line in the weight memory stores weight data corresponding to at least one convolutional kernel, wherein one convolutional kernel corresponds to at least one computation unit; a word line in the feature map memory stores feature map data corresponding to at least one channel, wherein the number of channels is consistent with the number of convolutional kernels; a memory-computation controller is used to control the synchronous operation of each computation unit; and a computation unit is used to generate new feature map data based on the weight data and feature map data corresponding to the convolutional kernel.

[0080] In this embodiment, feature map reuse due to multiple convolutional kernels is achieved by computing all convolutional kernels in parallel. With a near-in-memory computing system for neural networks including 8 in-memory modules, 2048 convolutional kernels can be computed in parallel. By computing all convolutional kernels in parallel, feature map reuse due to multiple convolutional kernels can be achieved. Since the feature map memory stores multiple channel data in a single row of eDRAMi_FM (i.e., on a single word line), several rows of latches are needed to buffer the feature map data to achieve vertical reuse of the feature map. When the computing resource E expands, horizontal reuse of the feature map is achieved because multiple PE units compute several adjacent convolutions in parallel.

[0081] In this embodiment, as Figure 7 The diagram shows the computational structure of a neural network convolutional layer. A typical convolutional layer has M kernels with weights of k*k*N, where N is the number of channels, the stride is S, R and C represent the size of the input feature map, and R' and C' represent the size of the output feature map. The formula for calculating the output feature map after inputting the feature map data is as follows:

[0082]

[0083] Where x and y represent the output features Figure 2 Given a point (x, y) on a 3D plane, OF(x, y, m) represents the output feature map data corresponding to the point (x, y), F represents the input feature map data, and W... m This represents the weight value of the m-th convolutional kernel, where m ranges from 0 to M-1, z ranges from 0 to N-1, and k... x The value range of k is 0 to k-1, where k y The value range is 0 to k-1.

[0084] As shown in the above formula, the presence of M convolutional kernels allows the feature map data to be reused M times. Therefore, we achieve feature map reuse by computing M convolutional kernels in parallel. Since the 8 in-memory computing modules have a total of 2048 PE units, they can support the parallel computing of 2048 convolutional kernels, that is, the feature map is reused 2048 times.

[0085] In one embodiment, the feature map memory includes at least two preset latches; the latches are used to cache feature map data read from the feature map memory; and the memory-to-computation controller is used to broadcast the feature map data in the latches to each computing unit based on the feature map transmission bus.

[0086] In this embodiment, feature maps that need to be overlapped and retained are cached to achieve reuse caused by the overlap of adjacent feature maps.

[0087] based on Figure 7 The neural network model has the following formula:

[0088]

[0089]

[0090] Where OF(x+1,y,m) represents the output feature map data corresponding to point (x+1,y), OF(x,y+1,m) represents the output feature map data corresponding to point (x,y+1), F represents the input feature map data, and W represents the input feature map data. m This represents the weight value of the m-th convolutional kernel, where m ranges from 0 to M-1, z ranges from 0 to N-1, and k... x The value range of k is 0 to k-1, where k y The value range is 0 to k-1.

[0091] By comparing the three equations OF(x,y,m), OF(x+1,y,m), and OF(x,y+1,m), we can find that the feature map data required for these three convolutions overlaps. That is, when the convolution kernel size k is greater than the stride S, the feature maps overlap.

[0092] With an E-fold resource expansion, to achieve vertical reuse of feature maps, (kS) rows of feature map data need to be cached. During E-fold expansion, to perform E convolution calculations simultaneously, each row needs to buffer (k-S+E*S) two-dimensional feature map pixels, that is, a total of (SS)*(k-S+E*S) two-dimensional feature map points need to be buffered. Latch.

[0093] Since a 1x expansion is rare in actual computation, the focus is on meeting the feature map reuse requirements under 2x, 4x, and 8x expansion. Under 2x, 4x, and 8x resource expansion, for the most common 3x3 convolution kernel with a stride of 1, a total of 4 rows of latches are needed to buffer the feature map data. That is, in addition to the row of latches originally used for routing transmission, an extra 3 rows of latches are added to the feature map memory, for a total of 4 rows of latches.

[0094] After achieving vertical reuse of feature maps in a 3x3 convolution kernel, resource expansion enables horizontal reuse of feature maps without increasing hardware overhead. Taking a 3x3 convolution kernel with E-fold expansion as an example, the parallel computation of E horizontal convolutions allows for the reuse of feature maps that overlap horizontally between these E convolutions. Taking a 4-fold expansion as an example, four PE units compute the outputs OF(x,y,m), OF(x+1,y,m), OF(x+2,y,m), and OF(x+3,y,m) in parallel. There is significant overlap of input feature map data between these four parallel convolutions. Since feature map data can be transferred between adjacent PE units, horizontal reuse of feature maps is possible.

[0095] In this invention, the reuse of weight data is achieved by the convolution kernel moving horizontally and vertically on the feature map with a stride, resulting in multiple convolutions. Vertical reuse of weights is achieved by buffering the weight data.

[0096] In one embodiment, the computing unit includes at least one intermediate result register; the storage controller includes a computing controller; a latch is further configured to cache intermediate result data generated by convolution calculation in the computing unit; the intermediate result register is configured to cache intermediate result data generated by convolution calculation in the computing unit; and the computing controller is configured to separately count the caching duration of unupdated data in the latch and each intermediate result register, and to cache the intermediate result data to the latch or intermediate result register with the longest caching duration each time intermediate result data is generated by convolution calculation.

[0097] In this embodiment, if the weight data is reused n times vertically, each PE unit will generate n intermediate result data for convolution calculations. Typically, each PE unit uses a 16-bit intermediate result register to temporarily store the intermediate result data. However, to reuse the weight data and to integrate the neural network near-memory computing system onto a single chip, considering chip area, an additional 16-bit intermediate result register is added to each PE unit, which already has one. This allows one PE unit to temporarily store the intermediate result data obtained from two convolutions. Furthermore, the latches used for feature map data caching can be reconfigured for storing intermediate result data.

[0098] In this embodiment, as Figure 8The schematic diagram of the internal structure of the computation unit shown includes a multiplier (MUL), an adder, a selection register (demux), two 16-bit intermediate result registers (16b REG), a multiplexer (mux) for selecting intermediate result registers, a pooling layer, an activation function (ReLU), a multiplexer (mux) for selecting the function result, a multiplexer (MUX) for selecting local feature map data (FM) or other PE unit input feature map data (FMfrom other PE), and a computation controller. The weight data (WT) and the feature map data (FM) output from the multiplexer are input to the multiplier, which can select any type, such as a Booth multiplier, as needed.

[0099] In this embodiment, by buffering the weight data and configuring two intermediate result registers within the PE unit, the vertical reuse count of weights is 2. However, two reuses of the weight data are far from sufficient, as reading from the weight memory still accounts for a significant portion of system energy consumption. Therefore, to reduce overall system energy consumption, weight data reuse is prioritized over feature map data reuse. Based on this, in addition to the one row of latches supporting routing, the three additional rows of latches added for feature map data reuse (equivalent to adding 3*8-bit latches to each PE unit) can be reconfigured to store the 16-bit intermediate result data of the PE unit, thereby improving weight reuse. At this point, the weight reuse count increases from 2 to 3.

[0100] In this embodiment, the horizontal reuse of weight data occurs during computational resource expansion. Since multiple PE units input the same weight data and perform several adjacent convolutions in parallel, the weight data can be reused horizontally. Under a resource expansion of E times, the weight data is reused horizontally without increasing hardware overhead. The number of horizontal reuses of the weight data is the same as the expansion factor E, and the number of vertical reuses of the weights is the same as the number (num_psum) of 16-bit intermediate result registers within the PE unit (the latch added for feature map data reuse can be considered equivalent to one intermediate result register). Therefore, the number of weight data reuses (reuseWT) is as follows:

[0101] reuseWT = WT horizontal reuse count * WT vertical reuse count = E * num_psum

[0102] In a specific example, taking a 4x scaling, four PE units are input with the same weight data and perform four horizontal convolutions in parallel: OF(x,y,m), OF(x+1,y,m), OF(x+2,y,m), and OF(x+3,y,m). This allows the weight data to be reused four times horizontally, and since the weights can be reused three times vertically, a total of 12 reuses are achieved with a 4x scaling. Similarly, with a 1x scaling, the weights can be reused three times; with a 2x scaling, they can be reused six times; and with an 8x scaling, they can be reused 24 times.

[0103] In this invention, when eDRAM is used as the weighted memory, eDRAM utilizes the capacitors in the memory cells to store data. Therefore, due to leakage problems (such as capacitor leakage or transistor leakage), the charge in the memory cells changes over a long period of time, which in turn affects the sensitivity amplifier's judgment of the output result "0" or "1". eDRAM needs to be periodically refreshed to compensate for the leakage problem. The refresh period is determined by the effective retention time of the data on the memory node. During the refresh period, it should be ensured that the data stored in the two gain cells is not lost.

[0104] The refresh operation differs from regular read and write operations, requiring a "read and write-back" process. First, a specific row (a word line) in the storage array to be refreshed is selected, and the data is read from the storage node. However, the read data does not need to be sent to I / O (i.e., the data is not actually retrieved from outside the memory). Then, the read data is "written back" to its original location in the storage array. It's important to note that the "write-back" operation is not a regular write operation, because the data used for writing is not retrieved from outside the memory by I / O.

[0105] Refresh energy consumption is a significant issue in eDRAM architecture. The purpose of refreshing is to prevent data stored in eDRAM for extended periods from becoming corrupted. Therefore, to reduce refresh energy consumption, two approaches can be taken: one is to reduce the number of refresh cycles, thereby lowering refresh energy consumption; the other is to utilize refresh operations to reduce the additional energy overhead associated with refreshing.

[0106] When using eDRAM for feature map memory, since the feature map data is updated in real time, the refresh frequency is reduced by decreasing the data's storage lifetime. Refreshing ensures that the effective storage time of the data exceeds the refresh cycle. If the data's storage time in memory is shorter than its refresh cycle, it can be discarded within one refresh cycle, thus eliminating the need for refresh. However, in near-memory architectures, weight data is stored unchanged in eDRAM. Therefore, this method is not suitable for weight data and can only be used for feature map data processing.

[0107] In one embodiment, the memory controller is configured to write back the read priority data to the priority memory when it first reads priority data from the priority memory during the priority data refresh cycle.

[0108] In this embodiment, the additional refresh energy consumption of the weight data is reduced by using the data read from the refresh operation for convolution. In the refresh operation, the energy consumption of "reading" is the main source of energy consumption, while the energy consumption of "writing back" accounts for only a small portion. Therefore, if the weight data acquired during the refresh process can be used for convolution calculation, the additional energy consumption from the refresh operation is equivalent to only a small portion of the energy consumed by the "writing back" part. Therefore, we send the weight data acquired during the "read" operation in the refresh process to an external memory, so that it is actually read out and used by the computing unit. Furthermore, the "writing back" process remains unchanged and is the same as the writing back process in a conventional refresh operation. That is, during the weight data refresh cycle, when the weight data is read from the weight memory for the first time according to the needs of convolution calculation, this weight data is used not only for convolution calculation but also simultaneously for a "writing back" operation.

[0109] In this embodiment, the timed refresh energy consumption of eDRAM has a significant impact on system energy efficiency. To reduce the additional energy consumption caused by refresh in the near-memory computing system of neural networks, this invention utilizes the "read" process required by the refresh operation ("read" and "write back") itself, allowing the data read during refresh to participate in convolution operations, thereby reducing the additional energy consumption required for refresh.

[0110] In one embodiment, each storage and computation module has a corresponding weight memory that stores weight data for different convolutional layers in the neural network; a storage and computation controller is used to read block weight data corresponding to the convolutional layers in the weight memory, wherein the block weight data corresponds to pre-divided block images in the original image; each storage and computation module has a corresponding computation array that performs calculations for different convolutional layers in the neural network based on the block weight data; and each storage and computation module has a corresponding feature map memory that stores feature map data calculated by different convolutional layers in the neural network, wherein the feature map data obtained by each computation array in one convolutional calculation corresponds to a block image.

[0111] In this embodiment, it is assumed that the neural network has L convolutional layers. Within one refresh cycle, the weight data of the convolutional layers performing convolution calculations is used, therefore these layers do not require additional refresh energy; however, convolutional layers that are not currently performing calculations require additional refresh operations because their weight data is not needed. Therefore, after each refresh cycle, as many layers as possible should be calculated. Figure 9The left-hand diagram illustrates the layer-by-layer convolution principle of a neural network. Layer 1…Layer 1…Layer M represent M different convolutional layers. The conventional approach is to compute the entire feature map of a certain convolutional layer before proceeding to the next. Assuming that in this computational mode, M convolutional layers participate in computation within a single refresh cycle, the weight data of M convolutional layers is read at least once within that cycle. This first read of the weight data for these M layers in a refresh cycle is accompanied by a "write-back" process, thus satisfying the memory refresh requirements and eliminating the need for additional energy consumption during refresh.

[0112] In this embodiment, the neural network is further analyzed. The two-dimensional spatial correlation of natural images mainly exists locally; that is, a pixel has a high correlation with its surrounding pixels, but a low correlation with distant data. Therefore, each neuron in the neural network does not perceive the entire feature map image, but only with the feature map within its surrounding receptive field (the size of the convolutional kernel). Thus, convolutional neural networks have the characteristic of local connectivity (sparse connectivity). Based on this characteristic of local connectivity, there are connections between the feature map data of each convolutional layer. Figure 9 The data correspondence shown on the right, where Layer 1...Layer i...Layer N represent N different convolutional layers, is used to divide the original image into blocks, forming a "pyramid" structure of computational data flow. Within one refresh cycle, N convolutional layers participate in the computation within a single pyramid. The block weights corresponding to each of the N convolutional layers are read from the weight memory for calculation, thus eliminating the need for additional refreshes for N convolutional layers. After calculating in "pyramid" blocks, the size of the feature map calculated for each network layer within one refresh cycle is significantly reduced, resulting in a substantial decrease in the computation time per layer. Consequently, the number of convolutional layers N participating in the computation within a refresh cycle is greater than M. Compared to conventional methods, this effectively reduces the system's refresh energy consumption.

[0113] In an overall embodiment, such as Figure 10 The diagram shown illustrates the structural principle of a neural network near-memory computing system. The specific structure of the neural network near-memory computing system is as follows:

[0114] The neural network near-memory computing system includes eight in-memory computing modules, an image storage and broadcasting module, and a controller module.

[0115] Each in-memory computing module includes one weight memory, one feature map memory, one computing array, one routing path, and a data communication line. Both the weight memory and the feature map memory use eDRAM. Specifically, eDRAM0 for Weight, eDRAM1 for Weight, eDRAM2 for Weight, eDRAM3 for Weight, eDRAM4 for Weight, eDRAM5 for Weight, eDRAM6 for Weight, and eDRAM7 for Weight represent the weight memories corresponding to the eight in-memory computing modules. eDRAM for Feature Map represents the feature map memory corresponding to the in-memory computing module. PE array0, PE array1, PE array2, PE array3, PE array4, PE array5, PE array6, and PE array7 represent the computing arrays corresponding to the eight in-memory computing modules. Each in-memory computing module is configured with another routing path, and the communication path formed by these routes is used to transmit feature map data. Each feature map memory is configured with four rows of latches, each row being 2048 bits, meaning each feature map memory is configured with 4*2048-bit latches. In addition to supporting routing communication, this latch is also used for the reuse of feature map data or weight data.

[0116] The image storage and broadcasting module includes an interface for data interaction with the external neural network near-memory computing system. Through this interface, the weight data of the neural network can be pre-written into the neural network near-memory computing system. It can also receive raw images from external input, cache the raw images in the locally configured image memory (eDRAM_image), and then broadcast the raw images to each storage and computing module.

[0117] The controller module includes a main controller, sub-controllers corresponding to the image storage and broadcasting module, and in-memory controllers for each of the eight in-memory computing modules. Control 0, Control 1, Control 2, Control 3, Control 4, Control 5, Control 6, and Control 7 represent the in-memory controllers for each of the eight in-memory computing modules. The main controller provides overall control over the sub-controllers corresponding to the image storage and broadcasting module and the in-memory controllers for each of the individual in-memory computing modules. The sub-controllers for the image storage and broadcasting module control the image storage and broadcasting module, performing image reading, refreshing, and broadcasting. Each in-memory controller controls its corresponding in-memory computing module, performing data reading, writing, and refreshing (including weight data and feature map data). The controller module ensures that the neural network near-memory computing system can accurately perform automatic neural network calculations.

[0118] After each in-memory computation module completes its corresponding convolution calculation in parallel, the final calculation result, i.e., the classification result, is output through the routing in the in-memory computation module corresponding to the last convolutional layer.

[0119] In one embodiment, the neural network near-memory computing system is integrated into a single chip. Using a 40-nanometer (nm) process, the hardware design is synthesized and placed / routed using Synopsys Design Compiler (DC). Synopsys is a commonly used synthesis tool, and Design Compiler is the command-line interface for Synopsys synthesis tools. The chip layout is as follows: Figure 11 Wherein, Weight eDRAM represents the weight memory; FM eDRAM represents the feature map memory; PE Array represents the computing array; CTRL0, CTRL1, CTRL2, CTRL3, CTRL4, CTRL5, CTRL6, and CTRL7 represent the memory controllers corresponding to the eight memory modules respectively; Image CTRL represents the sub-controller corresponding to the image storage and broadcasting module; Image eDRAM represents the image memory locally configured in the image storage and broadcasting module; the controller module may also include memory controllers configured for each memory module according to the weight memory and feature map memory, eDRAM CTRL represents the memory controller; TOP CTRL represents the main controller; BUS represents the feature map transmission bus; and R represents routing.

[0120] The performance indicators of the chip obtained through post-simulation are shown in Table 1:

[0121] Table 1 Chip Performance Indicators

[0122]

[0123] In Table 1, GOPS (Giga Operations Per Second) refers to one billion operations per second, TOPS / W is used to measure how many trillion operations a processor can perform at 1 watt (W) of power consumption, MobileNet is a lightweight neural network, and AutoEncoder is an unsupervised learning algorithm.

[0124] The performance metrics of the proposed neural network near-memory computing system architecture are compared with other near-memory architectures, as shown in Table 2. At the same computational accuracy, the architecture proposed in this paper improves computational resource utilization and system energy efficiency compared to other near-memory architectures. However, compared to near-memory architectures based on DRAM memory, this paper uses embedded DRAM, which significantly reduces on-chip storage resources. A single chip can only support lightweight data networks with weight data volumes not exceeding 4MB. For large neural networks, multiple chips are required to implement weight data storage and near-memory computation.

[0125] Table 2 Performance Comparison

[0126]

[0127] McDRAM v2, FIMDRAM, Neurocube, and DaDianNao represent different near-memory architecture computing schemes in the prior art.

[0128] The present invention provides a neural network near-memory computing system, comprising at least two in-memory computing modules, an image storage and broadcasting module, and a controller module. The image storage and broadcasting module is used to cache the original image input from the external source and broadcast the original image to each in-memory computing module. Each in-memory computing module is used for distributed storage of the neural network's weight data and distributed storage of the feature map data calculated by the neural network, as well as for performing calculations in the neural network based on the weight data, the original image, and the feature map data. The controller module is used to control the operation of each in-memory computing module and the operation of the image storage and broadcasting module. By setting at least two in-memory computing modules and distributively storing the weight data and feature map data, multiple reads of feature map data are avoided, saving energy consumption from data movement. Simultaneously, the calculated output feature maps can also be stored separately in each in-memory computing module, further reducing energy consumption from data movement. Each in-memory computing module can synchronously complete calculations in the neural network, improving the neural network's computational efficiency. Therefore, this neural network near-memory computing system improves system energy efficiency and computational resource utilization.

[0129] Furthermore, feature map data and weight data are distributed and stored in embedded dynamic memory, utilizing the full bandwidth of the memory for data reading and writing, and performing distributed computation. The read feature map data and weight data are cached using latches to increase data reuse frequency, thereby reducing memory read frequency and energy consumption. The convolutional neural network computation adopts a "pyramid" structure to reduce unnecessary refreshes of the embedded memory. Through the above methods, the energy consumption for data movement, memory reads, and refreshes can be reduced, improving the system's energy efficiency. In addition, computational resources can be dynamically allocated for different convolutional layers of the neural network, improving computational resource utilization. Using a 40nm process, the neural network near-memory computing system provided by this invention can achieve a computational resource utilization rate of over 90% and a system energy efficiency of 3.3 TOPS / W.

[0130] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0131] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0132] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A neural network near-memory computing system, characterized in that, It includes at least two storage and computing modules, an image storage and broadcasting module, and a controller module; The image storage and broadcasting module is used to cache the original image input from the outside and to broadcast the original image to each of the storage and computing modules; Each of the aforementioned storage and computing modules is used for distributed storage of the weight data of the neural network, distributed storage of the feature map data calculated by the neural network, and performing calculations in the neural network based on the weight data, the original image, and the feature map data; The controller module is used to control the operation of each of the storage and computing modules, and to control the operation of the image storage and broadcasting module; The neural network includes at least one convolutional layer, each of the in-memory modules corresponds to at least one convolutional layer, and each convolutional layer includes at least one convolutional kernel; Each of the aforementioned memory modules includes a weight memory, a feature map memory, a computing array, routing, and data communication lines; For each convolutional layer corresponding to each of the storage modules: the weight data of each convolutional kernel is stored on different bit lines in the weight memory, and the feature map data of each channel is stored on different bit lines in the feature map memory; The computing array is used to perform convolution and nonlinear calculations in the neural network; The routing is used for data transmission between adjacent storage modules; The data communication line is used for data transmission between the weight memory and the computing array, and for data transmission between the feature map memory and the computing array; The computing array includes at least one computing unit; Each word line in the weight memory stores the weight data of at least one convolutional kernel; One convolutional kernel corresponds to E computational units, where E is a positive integer; The feature map data output by the E computing units is stored in the feature map memory based on the E channels.

2. The neural network near-memory computing system according to claim 1, characterized in that, The controller module includes a storage controller corresponding to each of the storage modules; The computing array includes at least one computing unit; The data communication line includes a feature map transmission bus; The in-memory controller is configured to sequentially perform the following operations on each word line in the feature map memory: read the feature map data on one word line based on the feature map transmission bus; broadcast the read feature map data on one word line to each computing unit based on the routing and the feature map transmission bus; and transmit the read feature map data on one word line to the adjacent in-memory module through the routing. Based on the feature map transmission bus, the new feature map data output by the computing unit is updated to the feature map memory.

3. The neural network near-memory computing system according to claim 1, characterized in that, The controller module includes a storage controller corresponding to each of the storage modules; The computing array includes at least one computing unit, wherein the computing unit corresponds one-to-one with the channel in the convolutional layer of the storage module; The data communication line includes a feature map transmission line corresponding to each of the computing units. The memory controller sequentially performs the following operations on each word line in the feature map data: based on the feature map transmission line, it reads the feature map data corresponding to each channel on a word line and transmits the feature map data to the corresponding computing unit; based on the feature map transmission line corresponding to each channel, it stores the new feature map data output by each computing unit into the feature map memory.

4. The neural network near-memory computing system according to claim 2, characterized in that, One word line in the weight memory stores the weight data corresponding to at least one of the convolution kernels, wherein one convolution kernel corresponds to at least one of the computation units; One word line in the feature map memory stores feature map data corresponding to at least one channel, wherein the number of channels is the same as the number of convolution kernels; The in-memory controller is used to control the synchronous operation of each of the computing units; The computing unit is used to generate the new feature map data based on the weight data corresponding to the convolution kernel and the feature map data.

5. The neural network near-memory computing system according to claim 2, characterized in that, The feature map memory includes at least two pre-defined latches; The latch is used to cache the feature map data read from the feature map memory; The in-memory controller is used to broadcast the feature map data in the latch to each of the computing units based on the feature map transmission bus.

6. The neural network near-memory computing system according to claim 5, characterized in that, The computing unit includes at least one intermediate result register; The in-memory controller includes a computing controller; The latch is also used to cache intermediate result data generated by convolution calculation in the computing unit; The intermediate result register is used to cache intermediate result data generated by convolution calculation in the computing unit; The computing controller is used to calculate the cache duration of the latch and each of the intermediate result registers for unupdated data, and to cache the intermediate result data to the latch or the intermediate result register with the longest cache duration each time the convolution calculation generates the intermediate result data.

7. The neural network near-memory computing system according to claim 2, characterized in that, The storage controller is configured to write back the read weight data to the weight memory when reading the weight data from the weight memory for the first time during the weight data refresh cycle.

8. The neural network near-memory computing system according to claim 2, characterized in that, Each of the storage modules corresponds to a weight memory, which stores the weight data of different convolutional layers in the neural network. The in-memory controller is used to read the block weight data corresponding to the convolutional layer in the weight memory, wherein the block weight data corresponds to the pre-divided block image in the original image; Each of the stored-computing modules corresponds to a computing array that performs calculations for different convolutional layers in the neural network based on the block weight data. Each of the aforementioned computing modules corresponds to a feature map memory, which stores the feature map data calculated by different convolutional layers in the neural network. The feature map data obtained by each computing array in one convolutional calculation corresponds to the block image.

Images