Data storage and access method and system for pulse convolutional neural network accelerator

Abstract

The invention belongs to the technical field of neural network hardware accelerator and integrated circuit design. The invention provides a pulse convolutional neural network accelerator-oriented data storage and access method and system. According to the embodiment of the invention, a data storage and access mode matched with the systolic array calculation data flow is designed according to the binary property and the multi-time-step reasoning characteristic of the pulse data in the pulse convolutional neural network. A storage sequence which is mainly based on the input pulse data row direction, is grouped according to input channels and is continuously organized in the time step dimension is adopted in the off-chip memory, so that interaction with the off-chip memory can be carried out in a continuous access mode in the data loading and calculation result write-back process. The method comprises the following steps of: aggregating pulse data of a plurality of input channels in a storage word on an on-chip memory level; and meanwhile, in combination with an input cache design of a multi-BANK structure, multi-channel input data of adjacent rows can be read in parallel in a convolution window expansion process.

Description

Technical Field

[0001] This disclosure relates to the fields of neural network hardware accelerators and integrated circuit design technology, and in particular to a data storage and access method and system for pulsed convolutional neural network accelerators. Background Technology

[0002] Spiking Neural Networks (SNNs) have garnered significant attention from academia and industry in recent years due to their potential low-power advantages stemming from their pulse sparsity during computation. Spiking Neural Networks (SCNNs) introduce convolutional structures into SNNs, enabling them to maintain low power consumption while possessing spatial feature extraction capabilities. This allows for efficient execution of perception and reasoning tasks on resource-constrained platforms such as edge devices. To meet the performance and energy efficiency requirements of SCNNs, researchers have begun exploring dedicated hardware accelerator architectures for them. Among these, systolic arrays, due to their regular data flow and good scalability, are widely used in traditional Convolutional Neural Network (CNN) accelerators and are increasingly being incorporated into hardware implementations supporting pulsed convolution operations. In systolic array-based SCNN accelerators, computation is typically clock-driven, with each processing unit performing convolution operations and neuron state updates according to a predetermined data flow pattern within discrete clock cycles. In the field of traditional CNN accelerators, a large amount of research has focused on the organization and access of data in storage systems. Typical examples include the DianNao series of accelerators (see papers such as "DianNao: A Small-Footprint High-Throughput Accelerator for Ubiquitous Machine-Learning"), which efficiently maps weights and activation data to on-chip memory to reduce the number of accesses to off-chip memory, thereby lowering energy consumption. The Eyeriss architecture (see papers such as "Eyeriss: An Energy-Efficient Reconfigurable Accelerator for DeepConvolutional Neural Networks") systematically analyzes the impact of different data flows, such as fixed weights and fixed outputs, on data reuse and storage access behavior, and optimizes the data layout for on-chip buffers and off-chip memory. Google TPU (see papers such as "In-Datacenter Performance Analysis of a TensorProcessing Unit") employs a large-scale systolic array structure and improves computational throughput and storage bandwidth utilization through reasonable data partitioning and storage scheduling. In contrast, research on hardware accelerators for SNNs, especially SCNNs, is relatively limited. Existing work focuses more on the implementation of neuron models in general spiking neural networks or neuromorphic computing platforms, neuron state storage and update mechanisms, and spiking computing unit design (see papers such as "TrueNorth: Design and Tool Flow of a 65 mW 1 Million Neuron Programmable NeurosynapticChip" and "Loihi: A Neuromorphic Manycore Processor with On-Chip Learning").In hardware implementations of SCNN, existing research has focused on accelerating spike convolution operations from the perspectives of computational architecture and data flow scheduling (see papers such as "SpinalFlow: An Architecture and Dataflow Tailored for Spiking Neural Networks"). However, in terms of storage organization and data access, some SCNN accelerators still use or simply modify the data storage methods in traditional CNN accelerators. That is, they mainly store spike data according to spatial or channel dimensions and repeatedly read the corresponding data at different time steps to complete inference computation (see papers such as "SIES: A Novel Implementation of Spiking Convolutional Neural Network Inference Engine on FPGA" and "DeepFire: Acceleration of Convolutional Spiking Neural Network on FPGAs"). Figure 1 and Figure 2 The paper demonstrates two existing pulse data storage sequences: one prioritized by spatial dimension, and the other prioritized by channel dimension. In this type of storage, the time step dimension is not explicitly arranged as a dimension in the data storage organization. The input pulse data for each time step is stored independently using the same storage layout as a single time step. During inference, the corresponding data needs to be accessed repeatedly for different time steps.

[0003] However, such storage methods typically do not fully consider the binary nature of pulse data and the additional data access requirements brought about by multi-timestep inference. In recent years, some studies have also begun to focus on the dataflow optimization problem of SCNN based on systolic arrays (see papers such as "Reconfigurable Dataflow Optimization for Spatiotemporal Spiking NeuralComputation on Systolic Array Accelerators"), which systematically analyze the impact of different placement methods of the time step dimension in pulse convolution computation on system performance from the perspective of computation scheduling and loop nesting. However, related studies mainly focus on optimization at the dataflow and computation scheduling levels, and still lack a systematic description of the physical organization of pulse data in specific storage media. In summary, compared to traditional CNNs, SCNNs exhibit significant differences in data storage and access: Firstly, impulse data is represented in binary form, and its semantics differ from continuous-value activation data. Directly using conventional numerical data storage methods may lead to low storage resource utilization. Secondly, the inference process of SCNNs needs to unfold across multiple time steps. Compared to the single forward computation of CNNs, the hardware data flow scheduling introduces an additional time dimension, making the mapping relationship between the time, spatial, and channel dimensions of impulse data more complex. In a clock-driven SCNN accelerator based on a systolic array, the placement of the time step dimension in the storage hierarchy and its correspondence with the computational data flow have a significant impact on storage access efficiency and hardware implementation complexity.

[0004] However, existing technologies still lack a data storage and access scheme that can uniformly design the data organization methods and storage access order in off-chip and on-chip memory for the binary and multi-time-step inference characteristics of pulse data, and match the data flow of systolic array computation.

[0005] Therefore, it is necessary to improve one or more of the problems existing in the above-mentioned related technical solutions.

[0006] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0007] The purpose of this disclosure is to provide a data storage and access method and system for pulsed convolutional neural network accelerators, thereby overcoming, to at least some extent, one or more problems caused by the limitations and defects of related technologies.

[0008] According to a first aspect of the present disclosure, a data storage and access system for a pulsed convolutional neural network accelerator is provided, comprising: The SCNN accelerator includes a computation module, a memory access interface module, and multiple on-chip buffers. The memory access interface module includes an input read controller, a weight read controller, and an output write-back controller. Off-chip memory is used to store input pulse data, convolution kernels, and output feature maps; An input read controller is used to read input pulse data from external memory; Weight readout controller, used to read convolution kernels from off-chip memory; Multiple on-chip buffers are used to cache convolution kernels and input pulse data; The computation module is used to perform convolution calculations on the input pulse data and combine them with convolution kernels to generate an output feature map; The output write-back controller is used to write the output feature map back to off-chip memory.

[0009] Furthermore, the input pulse data is logically divided in the off-chip memory according to the input channel group, time step, and spatial column direction to form multiple input data blocks. The specific storage order is as follows: the row direction of the input pulse data is the main storage order; within the same row, the data of different time steps are stored continuously according to the order of the input channel group, and within each input channel group, the data is further stored in the column direction of the input pulse data as the main storage order.

[0010] Furthermore, the on-chip buffer includes: Input buffer, weight buffer, and output buffer; among which, The input buffer adopts a multi-BANK structure. Each input buffer is used to store one row of input pulse data at multiple time steps. Its storage order is consistent with the order of the corresponding data in the off-chip memory. The bit width of each storage word matches the parallelism of the input channel and is used to store multiple channel input pulse data in the same column within the same input channel group. The weight cache is used to store the convolution kernels.

[0011] Furthermore, the computing module includes: The system includes a convolution window unrolling submodule, a data skew submodule, a systolic array, a pooling submodule, and a neuron state update and pulse generation submodule; among which, The convolution window unrolling submodule is used to perform convolution window unrolling operations on the input pulse data to form a matrix suitable for pulsation array calculations; The data skew submodule is used to perform timing adjustments on the convolution kernel and the unfolded matrix to meet the timing requirements of the systolic array. A systolic array is used to perform convolution calculations to obtain the convolution calculation results within the current time step; The neuron state update and pulse generation submodule is used to update the neuron membrane potential and generate output pulses based on the convolution calculation results within multiple time steps, forming a convolution result block; The pooling submodule is used to perform pooling processing on the convolution calculation results.

[0012] Furthermore, when the time step is greater than 1, the input channel data at the same spatial location at different time steps are processed sequentially to avoid frequently writing the neuron states related between time steps back to off-chip memory; Once all time steps for the current spatial location have been calculated, the next convolution result block is processed along the column direction of the input pulse data. Once all rows of input data have been computed at all time steps, the results in the output buffer are written back to the off-chip memory in a predetermined order.

[0013] Furthermore, the system also includes: The control module includes a convolution controller, a pooling controller, and a global controller; among which, The convolution controller and pooling controller are used to control the internal operation timing of the convolution calculation module and the pooling processing module, respectively. The global controller is used to uniformly schedule all functional modules and coordinate the transmission of data between modules.

[0014] Furthermore, the on-chip buffer also includes: Neuron state memory and accumulation buffer; among which, Neuron state memory, used to store neuron states; The accumulation buffer is used to accumulate the convolution output to support subsequent pooling processing.

[0015] Furthermore, the storage access interface module also includes: AXI host interface and asynchronous FIFO; among which... The AXI host interface is used to initiate data access requests to off-chip memory. Asynchronous FIFOs include read FIFOs and write FIFOs, which are used for data transmission across clock domains in the read path and write path, respectively.

[0016] According to a second aspect of the present disclosure, a data storage and access method for a pulsed convolutional neural network accelerator is provided, comprising: Read input pulse data from off-chip memory; Read the convolution kernel from off-chip memory; Cache the convolution kernels and input impulse data; The input pulse data is convolved and combined with the convolution kernel to generate the output feature map; Write the output feature map back to off-chip memory.

[0017] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects: In the embodiments of this disclosure, the above-described method and apparatus, on the one hand, address the binary nature and multi-time-step inference characteristics of pulse data in pulse convolutional neural networks, designing a data storage and access method that matches the computational data flow of the systolic array. By employing a storage order in off-chip memory that is primarily based on the input pulse data row direction, grouped by input channels, and continuously organized along the time-step dimension, continuous access to the off-chip memory is possible during data loading and write-back of computation results, thereby improving off-chip memory bandwidth utilization and reducing the number of storage access requests. On the other hand, at the on-chip memory level, by aggregating pulse data from multiple input channels within a storage word, the utilization efficiency of on-chip memory resources is improved; simultaneously, combined with a multi-BANK input cache design, multi-channel input data from adjacent rows can be read in parallel during the convolution window expansion, improving the parallelism of convolution computation and inference speed. When the convolution window slides along the input pulse data row direction, the on-chip cached input row data can be reused, requiring only the loading of new input row data from off-chip memory, thereby further reducing access to off-chip memory. Furthermore, the input and output data are organized in a consistent manner in both off-chip and on-chip memory, allowing the output feature map generated by the current network layer to be directly used as the input data for the next network layer without the need for additional data rearrangement or format conversion operations. This reduces latency and hardware implementation complexity in the multi-layer inference process at the system level.

[0018] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0020] Figure 1 This diagram illustrates pulse data storage prioritizing spatial dimensions in an exemplary embodiment of the present disclosure. Figure 2 This diagram illustrates pulse data storage prioritized by channel dimension in an exemplary embodiment of this disclosure. Figure 3 This diagram illustrates the overall architecture of a data storage and access system for a pulsed convolutional neural network accelerator, as shown in an exemplary embodiment of this disclosure. Figure 4 This diagram illustrates convolution calculation in an exemplary embodiment of the present disclosure. Figure 5This diagram illustrates the input data segmentation and calculation sequence in an exemplary embodiment of this disclosure. Figure 6 This diagram illustrates the input pulse data at T time steps in an exemplary embodiment of this disclosure. Figure 7 This diagram illustrates the storage order of input pulse data in DRAM according to an exemplary embodiment of the present disclosure. Figure 8 This diagram illustrates the input data storage, computation, and result generation during multi-timestep pulsed convolutional inference in an exemplary embodiment of this disclosure. Figure 9 This illustrates the impact of three data storage methods on DRAM access frequency under different array sizes and different network models in an exemplary embodiment of this disclosure. Figure 10 The diagram illustrates the steps of a data storage and access method for a pulsed convolutional neural network accelerator according to an exemplary embodiment of the present disclosure. Detailed Implementation

[0021] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that this disclosure will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0022] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0023] This example implementation first provides a data storage and access system for spike convolutional neural network accelerators. This system can be applied to a terminal device, such as a mobile phone, personal digital assistant, laptop, tablet, smartwatch, or other mobile terminal [adjusted flexibly according to specific circumstances, such as the terminal device being a server]. Reference Figure 3 As shown, the system may include: The SCNN accelerator includes a computation module, a memory access interface module, and multiple on-chip buffers. The memory access interface module includes an input read controller, a weight read controller, and an output write-back controller. Off-chip memory is used to store input pulse data, convolution kernels, and output feature maps; An input read controller is used to read input pulse data from external memory; Weight readout controller, used to read convolution kernels from off-chip memory; Multiple on-chip buffers are used to cache convolution kernels and input pulse data; The computation module is used to perform convolution calculations on the input pulse data and combine them with convolution kernels to generate an output feature map; The output write-back controller is used to write the output feature map back to off-chip memory.

[0024] Through the aforementioned methods and apparatus, on the one hand, a data storage and access method matching the systolic array computation data flow is designed to address the binary nature and multi-time-step inference characteristics of pulse data in systolic convolutional neural networks. By employing a storage order in off-chip memory that prioritizes the input pulse data row direction, groups input channels, and organizes them continuously along the time-step dimension, continuous access to the off-chip memory is possible during data loading and write-back of computation results, thereby improving off-chip memory bandwidth utilization and reducing the number of memory access requests. On the other hand, at the on-chip memory level, the utilization efficiency of on-chip memory resources is improved by aggregating pulse data from multiple input channels within a storage word; simultaneously, combined with a multi-bank input buffer design, multi-channel input data from adjacent rows can be read in parallel during convolution window expansion, improving the parallelism of convolution computation and inference speed. When the convolution window slides along the input pulse data row direction, the on-chip cached input row data can be reused, requiring only the loading of new input row data from off-chip memory, further reducing access to off-chip memory. Furthermore, the input and output data are organized in a consistent manner in both off-chip and on-chip memory, allowing the output feature map generated by the current network layer to be directly used as the input data for the next network layer without the need for additional data rearrangement or format conversion operations. This reduces latency and hardware implementation complexity in the multi-layer inference process at the system level.

[0025] Below, we will refer to Figures 3 to 9 The various parts of the system described in this example embodiment will be described in more detail.

[0026] In one embodiment, this application proposes a data storage and access method and system for a Spiking Convolutional Neural Network (SCNN) accelerator. The system includes a systolic array-based neural network accelerator (SCNN) and off-chip memory. The SCNN accelerator is used to complete the multi-timestep inference computation process of SCNN under clock-driven conditions. Its overall architecture is as follows: Figure 3 As shown.

[0027] The SCNN accelerator includes a computation module, a control module, a memory access interface module, and multiple on-chip buffers.

[0028] The computation module includes a convolution window unrolling module (Im2col to Column, Im2col), a data skew module, a systolic array, a neuron state update and spike generation submodule (NeuronState Update and Spike Generate, NSU&SG), and a pooling submodule. Specifically, the convolution window unrolling module performs convolution window unrolling on the input spike data to form a data format suitable for systolic array computation; the data skew module performs temporal adjustments on the unrolled data to meet the timing requirements of the systolic array; the systolic array and the neuron state update and spike generation submodule work together to accumulate, update, and generate spikes from neuronal membrane potentials over multiple time steps; and the pooling submodule performs pooling processing on the convolution computation results.

[0029] The control module includes a Convolution Controller, a Pooling Controller, and a Global Controller. The Convolution Controller and Pooling Controller are used to control the internal operation timing of the convolution calculation-related modules and the pooling processing modules, respectively; the Global Controller is used to uniformly schedule the various functional modules of the accelerator and coordinate the data transmission between the modules.

[0030] On-chip buffers are used to cache convolution kernels, input impulse data, and data generated during computation. These include a multi-bank structure: Input Buffer (IB), Weight Buffer (WB), Neuron State Buffer (NSB), Accumulation Buffer (AB), and Output Buffer (OB). The Accumulation Buffer is used to accumulate the convolution output to support subsequent pooling processing.

[0031] The memory access interface module is used to realize data interaction between the accelerator and external memory, including a memory access controller, an AXI master interface, and two asynchronous FIFOs. The memory access controller consists of three sub-controllers: an input read controller, a weight read controller, and an output write-back controller, which are used for address generation of different types of data and external memory access control, respectively. The two asynchronous FIFOs are a read FIFO and a write FIFO, used for data transfer across clock domains in the read and write paths. The input pulse data is logically divided in the off-chip memory according to the input channel group, time step, and spatial column direction to form multiple input data blocks. The specific storage order is as follows: the row direction of the input pulse data is the main storage order; within the same row, the data of different time steps are stored continuously according to the order of the input channel group, and within each input channel group, the data is further stored in the column direction of the input pulse data as the main storage order.

[0032] In the SCNN accelerator, the systolic array employs a fixed output data stream. The row direction is used for parallel processing of data from different output pixel positions, and the column direction is used for parallel processing of data from different output channels. Before inference computation, the global controller initiates a weight read request to the weight read controller in the memory access interface, loading V convolutional kernels from off-chip memory into the on-chip WB, where V is the number of columns in the systolic array, corresponding to the parallelism of the output channels. Subsequently, the global controller initiates an input pulse read request to the input read controller, loading R rows of input pulse data into the on-chip multi-bank structure IB, where R is the height of the convolutional kernel. Each row of input data is stored in one IB, after which the convolution computation process begins. The convolution computation relationship between the R rows of input pulse data and the V convolutional kernels is as follows: Figure 4As shown in the figure. Where X and Y are the height and width of the input pulse data, C is the number of input channels, S is the width of the convolution kernel, and Y_ is the width of the output feature map.

[0033] When the number of input channels C is large, the input channels need to be processed in a time-division manner. Specifically, the calculation process is uniformly managed and scheduled by the global controller. The on-chip R rows of input pulse data are divided into multiple data blocks along the input channel dimension, and each data block contains V input channels; at the same time, since the number of rows N of the systolic array is limited, the on-chip R rows of input pulse data are further divided into multiple data blocks in the column direction of the input pulse data, and each data block can be expanded into N convolution windows. The above block division method and the corresponding calculation order are as follows. Figure 5 As shown, the black solid arrows indicate division along the input channel dimension, and the red solid arrows indicate division along the column direction of the input pulse data.

[0034] During computation, the global controller controls the convolution window to read data from multiple input blocks (IBs) in parallel. The Im2col module performs a convolution window unrolling operation, expanding a data block to obtain an N-row, R×S×V-column matrix. After the convolution window unrolling is complete, a completion signal is fed back. Upon receiving the completion signal, the global controller sends an enable signal to the convolution controller, which then controls the systolic array and neuron state update and pulse generation submodules to perform convolution calculations. The above process is repeated along the input channel dimension. When all input channels at the current spatial location have been calculated, the calculation results are sent to the NSU&SG module to update the neuron membrane potential and generate output pulses, forming a convolution result block, which is then written to the OB. When the inference time step is greater than 1, the input channel data at the same spatial location at different time steps are processed sequentially to avoid frequent writing of neuron states related between time steps back to off-chip memory. When all time steps at the current spatial location have been calculated, the next convolution result block is processed along the column direction of the input pulse data. When the calculations of R rows of input data at all time steps have been completed, the results in the OB are written back to off-chip memory in a predetermined order. Because IB uses a multi-bank structure, when the convolution window slides along the row direction, the window update can be completed simply by loading new input row data from off-chip memory, and the remaining overlapping row data can continue to be reused.

[0035] The convolution result block generated by the convolution calculation is written to AB or OB depending on whether pooling is enabled, and the updated neuron state is written to NSB. After the convolution calculation is completed, the convolution controller sends a completion signal to the global controller. If pooling is enabled, the global controller sends an enable signal to the pooling controller, which controls the pooling submodule to perform pooling operations on the convolution result block. The pooling result is written to OB. Subsequently, the global controller sends a write data request to the output write-back controller, which writes the calculation result back to off-chip memory through the storage access interface module.

[0036] Understandably, the input pulse data, after being unfolded, forms a matrix; the convolution kernel itself is stored in the weight cache as an unfolded matrix (the unfolding process does not need to be performed on-chip). Data skew affects both the input pulse data and the weight data simultaneously, and these two types of data are time-adjusted synchronously to meet the timing requirements of the systolic array (pulse data is fed in from the first column PE of the systolic array, and weight data is fed in from the first row PE; both types of data need to meet the timing requirements).

[0037] Based on the aforementioned computational data flow and multi-time-step inference sequence, this application provides a unified design for the organization of pulse input data in off-chip and on-chip memory. A schematic diagram of the input pulse data for T time steps is shown below. Figure 6 As shown in (a), each row of data is grouped according to the input channel parallelism V, as follows: Figure 6 As shown in (b), the dimensions of each data set are as follows: Figure 6 As shown in (c), the input pulse data is not stored as a complete feature map in the off-chip memory. Instead, it is logically divided according to input channel grouping, time step, and spatial column direction, forming multiple input data blocks. The specific storage order is: the row direction of the input pulse data is the primary storage order, such as... Figure 7 As shown in (a); within the same row, data from different time steps are stored consecutively according to the order of the input channel groups, such as... Figure 7 As shown in (b) and 7(c), within each input channel group, the data is further stored in the column direction of the input pulse data as the primary storage order, as follows: Figure 7 (d) and Figure 7 As shown in (e). In the on-chip memory, an IB is used to store the pulse data of one row of input pulse data at T time steps. Its storage order is consistent with the order of the corresponding data in the off-chip memory. The bit width of the IB is V. The V pulse data of a certain column in the same input channel group are stored in one address of the IB.

[0038] In a specific embodiment, the data storage and access method proposed in this application is illustrated using the inference process of a single-layer spiking convolutional neural network as an example. For ease of description, the relevant parameters are first defined: (1) Let the height and width of the input pulse data be respectively and The number of input channels is The height and width of the output feature map are respectively and The number of output channels is The height and width of the convolution kernel are respectively and .

[0039] (2) Assume that the input channels are arranged according to parallelism Grouped into groups, totaling 100 groups. The current input channel group being processed is... The parallelism of the output channels is also V, and it is divided into... The current output channel group being processed is... The parallelism of the output pixels is One row of output pixels is divided into The current output pixel group being processed is .

[0040] (3) Let the total number of inference time steps of the spiking convolutional neural network be . The current processing time step is .

[0041] Under the above parameter definitions, a single line of input pulse data can be divided into multiple data blocks according to predetermined rules, as illustrated in the diagram below. Figure 8 As shown in (a). The following uses a single line of input data in a single IB as an example to illustrate its loading, on-chip storage, computation, and result generation process at multiple time steps. Before inference begins, pulse input data is stored in off-chip memory and organized in row-major, channel-grouped, and time-step-continuous order, as shown in the storage structure below. Figure 7 As shown. During data loading, the accelerator reads the data of the current processing row from off-chip memory to on-chip IBs in a continuous manner according to the storage order described above. For the first convolution calculation, multiple rows of input data covered by the convolution kernel in the row direction need to be read and written to multiple on-chip IBs respectively. In subsequent convolution window movements along the input pulse data row direction, only one row of new input data needs to be loaded from off-chip memory and written to the corresponding IB; the remaining overlapping rows of data remain on-chip, thus achieving input data reuse. The storage order of each data block in a row within a single IB is as follows: Figure 9 As shown in (b).

[0042] After completing on-chip storage of the input data, the Im2col module simultaneously reads data from multiple on-chip Interface Blocks (IBs), performs a convolution window unrolling operation, and then feeds the unrolled data into a two-dimensional systolic array for convolution calculation. The calculation order of each data block is as follows: Figure 8 As shown in (c). Reading the current data block from IB... When processing a single data entry, its logical read address can be determined based on the current channel group number, the current time step number, and the output pixel group position. In one implementation, this logical read address can be represented as:

[0043] The order in which the convolution calculation results are generated is as follows: Figure 8 As shown in (d), the resulting convolution block is temporarily stored in an on-chip OB, and the storage order of each computation result block in the OB is as follows. Figure 8 As shown in (e). In one implementation, the current computation result block contains the first... When data is written to an on-chip OB, its logical write address can be represented as:

[0044] Once the calculations for the current input row are complete across all time steps and all output pixel groups, the calculation results in the on-chip OB are written back to the off-chip memory sequentially according to their logical order in the off-chip memory. Figure 8 (b) and Figure 8 (e) It can be seen that the data storage order in the on-chip OB is consistent with the order in which it is loaded from the off-chip memory to the on-chip IB, so the write-back process does not require additional data rearrangement. The output feature map after write-back can be directly used as input data for the next network layer. As can be seen from the above implementation examples, this application coordinates the storage organization of pulse input data with the computational data flow of the pulsating array and the multi-time step inference sequence, enabling the continuous processing of data blocks in the same spatial location during multi-time step inference, reducing the number of off-chip memory accesses, and simplifying on-chip memory access and control logic, thereby supporting efficient inference computation of pulse convolutional neural networks on hardware accelerators.

[0045] Furthermore, the effectiveness of this application was evaluated using modeling and simulation methods. A cycle-level simulator for an SCNN accelerator based on a systolic array was built, and DRAMsim (see papers such as "DRAMsim3: A Cycle-Accurate, Thermal-Capable DRAM Simulator") was integrated to simulate access to off-chip memory. Under different systolic array sizes and different network models, the data storage method proposed in this application was compared with traditional methods of organizing pulse data by spatial dimension (channel priority) and by channel dimension (channel post-processing), and the number of accesses to off-chip memory during network inference was counted. Evaluation results on six convolutional neural network models (LeNet, VGG16, MobileNet, SCNN6, SCNN9, and SCNN11, with the network topologies of SCNN6, SCNN9, and SCNN11 shown in Table 1) under four systolic array sizes (16×16, 32×32, 64×64, and 128×128) show that, compared with traditional data storage methods, the data storage and access method of this application can significantly reduce the total number of access requests to off-chip memory during inference. Relevant comparative results are shown in Table 1. Figure 9 As shown, the data storage and access method proposed in this application effectively reduces off-chip storage access overhead while supporting multi-timestep pulsed convolutional inference, thereby improving the overall performance and energy efficiency of the pulsed convolutional neural network accelerator.

[0046] Table 1 Three SCNN network topologies

[0047] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0048] Furthermore, this example embodiment also provides a data storage and access method for pulsed convolutional neural network accelerators. (See reference...) Figure 10 As shown, the method may include: Step S1: Read the input pulse data from the external memory; Step S2: Read the convolution kernel from off-chip memory; Step S3: Cache the convolution kernel and input impulse data; Step S4: Perform convolution calculation on the input pulse data and combine it with the convolution kernel to generate the output feature map; Step S5: Write the output feature map back to the off-chip memory.

[0049] It should be noted that although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that these steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps. Furthermore, it is readily understood that these steps may be executed synchronously or asynchronously, for example, in multiple modules / processes / threads.

[0050] This application has the following beneficial effects: To address the binary nature of pulse data and the multi-time-step inference characteristics in pulsed convolutional neural networks, a data storage and access method matching the computational data flow of the systolic array was designed. By employing a storage order in off-chip memory that prioritizes the input pulse data row direction, groups data by input channel, and organizes them continuously along the time-step dimension, continuous access to the off-chip memory is possible during data loading and write-back of computation results. This improves off-chip memory bandwidth utilization and reduces the number of memory access requests.

[0051] At the on-chip memory level, the utilization efficiency of on-chip memory resources is improved by aggregating pulse data from multiple input channels within a storage word. Simultaneously, combined with a multi-bank input buffer design, multi-channel input data from adjacent rows can be read in parallel during the convolution window unwinding process, improving the parallelism of convolution computation and inference speed. When the convolution window slides along the input pulse data row direction, the on-chip buffered input row data can be reused, requiring only the loading of new input row data from off-chip memory, thereby further reducing access to off-chip memory.

[0052] Furthermore, this application adopts a consistent data organization method for input and output data in both off-chip and on-chip memory, enabling the output feature map generated by the current network layer to be directly used as input data for the next network layer without additional data rearrangement or format conversion operations, thereby reducing latency and hardware implementation complexity in the multi-layer inference process at the system level.

[0053] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units. Components shown as modules or units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs. Those skilled in the art can understand and implement this without any inventive effort.

[0054] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the appended claims.

Claims

1. A data storage and access system for pulsed convolutional neural network accelerators, characterized in that, include: The SCNN accelerator includes a computation module, a memory access interface module, and multiple on-chip buffers. The memory access interface module includes an input read controller, a weight read controller, and an output write-back controller. Off-chip memory is used to store input pulse data, convolution kernels, and output feature maps; The input read controller is used to read input pulse data from external memory; Weight readout controller, used to read convolution kernels from off-chip memory; Multiple on-chip buffers are used to cache convolution kernels and input pulse data; The computation module is used to perform convolution calculations on the input pulse data and combine them with convolution kernels to generate an output feature map; The output write-back controller is used to write the output feature map back to off-chip memory.

2. The data storage and access system for pulsed convolutional neural network accelerators according to claim 1, characterized in that, The input pulse data is logically divided in the off-chip memory according to the input channel group, time step, and spatial column direction to form multiple input data blocks. The specific storage order is as follows: the row direction of the input pulse data is the main storage order; within the same row, the data of different time steps are stored continuously according to the order of the input channel group, and within each input channel group, the data is further stored in the column direction of the input pulse data as the main storage order.

3. The data storage and access system for pulsed convolutional neural network accelerators according to claim 2, characterized in that, The on-chip buffer includes: Input buffer, weight buffer, and output buffer; among which, The input buffer adopts a multi-BANK structure. Each input buffer is used to store one row of input pulse data at multiple time steps. Its storage order is consistent with the order of the corresponding data in the off-chip memory. The bit width of each storage word matches the parallelism of the input channel and is used to store multiple channel input pulse data in the same column within the same input channel group. The weight cache is used to store the convolution kernels.

4. The data storage and access system for pulsed convolutional neural network accelerators according to claim 3, characterized in that, The calculation module includes: The system includes a convolution window unrolling submodule, a data skew submodule, a systolic array, a pooling submodule, and a neuron state update and pulse generation submodule; among which, The convolution window unrolling submodule is used to perform convolution window unrolling operations on the input pulse data to form a matrix suitable for pulsation array calculations; The data skew submodule is used to perform timing adjustments on the convolution kernel and the unfolded matrix to meet the timing requirements of the systolic array. A systolic array is used to perform convolution calculations to obtain the convolution calculation results within the current time step; The neuron state update and pulse generation submodule is used to update the neuron membrane potential and generate output pulses based on the convolution calculation results within multiple time steps, forming a convolution result block; The pooling submodule is used to perform pooling processing on the convolution calculation results.

5. The data storage and access system for pulsed convolutional neural network accelerators according to claim 4, characterized in that, When the time step is greater than 1, the input channel data at the same spatial location at different time steps are processed sequentially to avoid frequently writing the neuron states related between time steps back to off-chip memory. Once all time steps for the current spatial location have been calculated, the next convolution result block is processed along the column direction of the input pulse data. Once all rows of input data have been computed at all time steps, the results in the output buffer are written back to the off-chip memory in a predetermined order.

6. The data storage and access system for pulsed convolutional neural network accelerators according to claim 5, characterized in that, The system also includes: The control module includes a convolution controller, a pooling controller, and a global controller; among which, The convolution controller and pooling controller are used to control the internal operation timing of the convolution calculation module and the pooling processing module, respectively. The global controller is used to uniformly schedule all functional modules and coordinate the transmission of data between modules.

7. The data storage and access system for pulsed convolutional neural network accelerators according to claim 6, characterized in that, The on-chip buffer also includes: Neuron state memory and accumulation buffer; among which, Neuron state memory, used to store neuron states; The accumulation buffer is used to accumulate the convolution output to support subsequent pooling processing.

8. The data storage and access system for pulsed convolutional neural network accelerators according to claim 7, characterized in that, The storage access interface module also includes: AXI host interface and asynchronous FIFO; among which... The AXI host interface is used to initiate data access requests to off-chip memory. Asynchronous FIFOs include read FIFOs and write FIFOs, which are used for data transmission across clock domains in the read path and write path, respectively.

9. A data storage and access method for pulsed convolutional neural network accelerators, characterized in that, include: Read input pulse data from off-chip memory; Read the convolution kernel from off-chip memory; Cache the convolution kernels and input impulse data; The input pulse data is convolved and combined with the convolution kernel to generate the output feature map; Write the output feature map back to off-chip memory.

Images