A neuromorphic data processing system based on SPI bus

Abstract

The present invention relates to the fields of integrated circuit technology and neural network technology, and in particular to a neuromorphic data processing system based on an SPI bus. The present invention proposes a neuromorphic data processing system based on an SPI bus, which enables the coordinated operation of multiple internal architectures through an SPI protocol with adjustable data bit width. The system adopts a compact design and stores neuron parameters off-chip, effectively reducing on-chip storage overhead and power consumption. Furthermore, the system incorporates a rich set of instruction packets and highly flexible, configurable neurons, enabling easy configuration of the on-chip network. Register multiplexing technology is used to effectively reduce the large amount of register overhead in the on-chip network. Routers are used between cores to transmit information, ensuring data reliability.

Description

Technical Field

[0001] The present invention relates to the fields of integrated circuit technology and neural network technology, and in particular to a neuromorphic data processing system based on an SPI bus. Background Art

[0002] The human brain possesses an astonishing ability to recognize complex patterns, computations performed by billions of neurons and synapses within the brain. Neuromorphic computing replicates the brain's computational capabilities by simulating the connections between neurons and synapses, building upon the plasticity of these neurons and synapses to develop corresponding theories, algorithms, and system architectures.

[0003] A spiking neural network model consists of spiking neurons and synapses. Each neuron is connected to several other neurons. When a neuron reaches its excitation threshold, it transmits information through extremely short pulses. Unlike typical multilayer perceptrons, spiking neural networks exhibit spatiotemporal characteristics and sparsity, with neurons firing over multiple operating cycles. Various neuron models exist at the biomimetic level, such as the common Leaky Integrate-And-Fire (LIF) and Hodgkin–Huxley models.

[0004] Neural networks are widely used in pattern recognition, autonomous driving, and data processing. As the network size increases and the recognition tasks become more complex, hardware implementation becomes increasingly difficult. Spiking neural networks, on the other hand, encode input data as 0s or 1s through pulse coding. The encoded pulse data conforms to digital circuit design, enabling highly connected neuron arrays and exponentially reducing hardware resource overhead. Compared to software simulation, hardware-integrated spiking neural networks leverage the parallel computing power of digital circuits to accelerate network processing. Multi-core hardware architectures also enable simultaneous computation of multiple networks.

[0005] The SPI protocol is a high-speed, full-duplex, four-wire communication bus with excellent transmission reliability and scalability, making it widely used in chips such as Flash and SRAM. To meet the needs of different devices, similar buses such as Dual SPI and Qual SPI have been proposed. These increase the data transmission bandwidth based on traditional SPI and offer multiple modes, making the SPI protocol even more applicable. In the field of neural network technology, the input weight data volume is large, the activation data reliability requirements are high, while the output data volume is generally small. To address these data transmission issues, a new data transmission protocol based on traditional SPI is needed. Summary of the Invention

[0006] Because existing neuromorphic data processing systems have complex inter-core communication methods and lack a complete set of instructions and universal protocols to control the entire processing system, the present invention provides an SPI-based neuromorphic data processing system to address the problems of low router efficiency, poor configurability of neuron registers and parameters, and high power consumption in current data processing systems.

[0007] To achieve the above object, the technical solution of the present invention is:

[0008] A neuromorphic data processing system based on an SPI bus, the system comprising: an SPI host, an SPI controller, a top-level controller, a memory controller (controlling off-chip SRAM), a buffer (on-chip memory), and a core array; the SPI controller is used to bridge the SPI host and the top-level controller, establishing communication between the two in an adjustable data width manner, with data and instructions transmitted via an 8-bit MOSI (host output, slave input) channel and a 1-bit MISO (host input, slave output) channel, wherein the 8-bit data width of the MOSI channel is flexibly adjustable, and the specific data width can be set by the SPI host; the memory controller is connected to the top-level controller and the off-chip SRAM, respectively, and is used to generate an SRAM storage protocol and read and store data from the off-chip SRAM; the core array interacts with the top-level controller, and the data processing method of the core array is based on a pulse neural network algorithm, and each core in the array comprises five processing modules: a scheduler, a buffer, a neuron, a router, and a neuron controller; the scheduler is used to store The activation data is stored in a buffer, and the activation data is pulse data. The pulse data includes two parts: the first part is the original input data encoded as pulse data, and acts on the first layer of computing cores; the second part of the data comes from the pulse data output by the router after the calculation of the previous layer of computing cores is completed, and acts on other computing cores, and the other computing cores are the hidden layer and output layer in the neural network; the buffer is used to store the configuration file read from the external SRAM, and the configuration file includes parameters such as weight data, leakage voltage, router, reset mode, reset voltage, threshold voltage, initial membrane voltage, leakage reversal flag, etc.; the neuron is used to perform pulse neural network calculations, and the input data required for the calculation comes from the scheduler and the buffer, the execution command comes from the core controller, and the final calculation result is passed to the router; the router is used to send and receive pulse data, and the data sent and received comes from the cores in the five directions of east, south, west, north and local; the neuron controller is used to control the conversion of neurons, the conversion of activation data, and the recording of the number of neuron activations, and control the coordinated work of each module under each core;

[0009] The neuromorphic data processing system performs data processing in the following manner: the SPI host sends an instruction packet to the top-level controller module through the SPI controller, and the top-level controller module receives and parses the instruction packet. The instruction packet includes a control packet and a data packet. The control packet includes a global initialization signal, setting core parameters, a configuration file, a write activation data command, a start calculation signal, and a read data signal. The data packet includes core setting data, pulse data, and single neuron data. The top-level controller module controls the data flow and working state of the core array through the instructions of the control packet. The data flow includes configuration files, activation data, neuron parameters, time steps, and calculation results. The working state includes initialization, start calculation, axon switching, neuron switching, and stop calculation. The top-level controller module transfers the core setting data and pulse data in the data packet to the core array, and transfers the single neuron data in the data packet to the storage controller. The top-level controller module obtains the required configuration file data storage address from the two variables of neuron and core position, and reads data from the external memory according to the address and transfers it to the core array.

[0010] The beneficial effects of the present invention are as follows: a data processing system based on the SPI bus is proposed, which realizes the collaborative work of multiple internal architectures through the SPI protocol with adjustable data bit width, adopts a compact design, and stores neuron parameters off-chip, effectively reducing the on-chip storage overhead and power consumption, and designs a rich instruction package and highly flexible and configurable neurons for the architecture, which can easily complete the configuration of the on-chip network, and adopts register multiplexing technology to effectively reduce the large amount of register overhead in the on-chip network, and uses routers to transmit information between cores to ensure data reliability. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 Schematic diagram of the overall architecture of the neuromorphic data processing system proposed in the present invention;

[0012] Figure 2 Schematic diagram of the SPI protocol with adjustable bit width proposed by the present invention;

[0013] Figure 3 It is a schematic diagram of the internal structure of a single core proposed by the present invention;

[0014] Figure 4 Schematic diagram of the neuron register model proposed in the present invention;

[0015] Figure 5 It is a diagram of the execution steps of the pulse neural network in the prior art;

[0016] Figure 6 It is the architecture control instruction packet proposed by the present invention;

[0017] Figure 7 This is a schematic diagram of the instruction packet sending format proposed by the present invention;

[0018] Figure 8 This is a schematic diagram of the router module structure proposed by the present invention; DETAILED DESCRIPTION

[0019] To make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0020] The main approach of the present invention is to use an extensible SPI protocol to control the parameter configuration and data transfer of the neuromorphic data processing system, store neuron configuration parameters in an external SRAM, store activation data in a scheduler, and use routers to communicate between different cores.

[0021] Neuromorphic data processing systems are generally used to execute convolutional neural networks or spiking neural networks. In comparison, spiking neural networks have lower power consumption and are more in line with actual application requirements. In terms of verification, FPGA or ASIC is usually used for verification. The overall architecture of a neuromorphic data processing system proposed in this invention is as follows: Figure 1 As shown, the data processing system implementation method includes the following modules: an SPI controller module that receives, sends, and packages data for processing; a storage controller module for generating SRAM storage protocols, reading, and storing data; a core array module that performs brain-like computing, is configurable, and reconfigurable; and a top-level controller module that connects the core array, SPI controller, and storage controller to work together. The entire system can be used to execute a spiking neural network, and each core in the core array can be used as the input layer, hidden layer, and output layer of the spiking neural network, with a maximum of 16 layers of networks being able to be executed. In an embodiment, three cores can be used to complete the recognition of handwritten digit images using a three-layer spiking neural network, where the image size is 16x16=256 pixels (compressed from 28x28 pixels), and the image content contains handwritten digits from 0 to 9.

[0022] The SPI controller in the neuromorphic data processing system is used to generate and parse the SPI protocol. It is internally connected to the top-level controller and externally connected to the SPI host for data exchange between the two parts. Figure 2As shown in the figure, the SPI protocol requires only eight independent signals to complete data transmission from the host to the slave: the system clock, the chip select bit, the SPI clock (data enable signal), the data bit SPI_MOSI (master-slave), the data bit SPI_MISO (master-slave), and the read valid signal. Before data transmission, the SPI clock frequency must be less than or equal to 1 / 4 of the system clock. The SPI clock can be derived from the system clock divided down or from another clock signal. The MOSI data transmission channel offers an 8-bit data channel to meet the transmission requirements of different host computers. Within the neuromorphic data processing system, different data processing modes are selected based on the instructions from the SPI host. Regardless of the data processing mode, the input data is ultimately converted into 8-bit data and transmitted to the top-level controller. Figure 2 The data transmission of 8-bit and 1-bit is shown in the figure. The time between the host sending data and the slave receiving data is only one system clock cycle. The instruction represented by 0x02d in the figure is the core setting, and the instruction represented by 0x01 is the core selection (see Figure 7 ), 0x is the hexadecimal representation method. When the SPI clock is 1 / 4 of the system clock frequency, the data transmission of the 8-bit channel requires 4 system clock cycles, and the data transmission of the 1-bit channel requires 32 system clock cycles. In the MISO data transmission channel, since only a small part of the neural network operation process parameters and the final calculation results need to be output, a 1-bit output data channel is used in this embodiment. In order to ensure the reliability of the data, a data valid signal is introduced. During the data transmission process from the slave to the host, this signal always remains at a high level, and remains at a low level in other cases. There are only two system clock cycles between the data sent by the slave and the data received by the host. Whether it is the MOSI channel or the MISO channel, a group of data can only be transmitted when the three conditions of the chip select bit being low, the rising edge of the system clock, and the falling edge of the SPI clock are met at the same time.

[0023] The SPI controller passes the received data to the top controller for further analysis and processing. The top controller Figure 6 ) to execute different instructions. In the initial parameter configuration stage, the top controller will analyze the register configuration file (see Figure 4) is written into the external SRAM. The storage space occupied by a single core is 21x32x256bit, and the activation data (Poisson-encoded image data) is written into the scheduler under the selected first-layer core. During the neural network calculation stage, the top-level controller will cyclically read the neuron register configuration parameters from the external SRAM through the storage controller. A single cycle will read the individual neurons under all activated cores, and the sequence number of each neuron will remain consistent (such as: neuron 5 of core 0-0, neuron 5 of core 0-1...). The top-level controller will pass the neuron data of each core to the cache to prepare for the next calculation. The entire data processing system is also controlled by the top-level controller to parse external instructions to control the start and end of calculations, axon switching, conversion of neurons and output results.

[0024] like Figure 3 As shown in Figure 1, a single core in the core array consists of five parts: neuron controller, cache, scheduler, neuron and router. The core array is arranged in a 4x4 matrix structure (see Figure 1 ), where each core's serial number corresponds to its X-axis and Y-axis coordinates. All cores operate within the same time step, which can be as large as 32. The time step is a concept unique to spiking neural networks and is also known as the number of iterations.

[0025] The scheduler stores activation data (spike data). The first layer of the network receives activation data differently from the other layers. The activation data for the first (input) layer is pulse data from the input image, transmitted by the top-level controller. This pulse data is encoded from a 256-pixel image into pulses at 32 time steps, with each time step consisting of 256 1-bit data elements. The data for the hidden and output layers comes from pulses sent by the routers in the previous layer. The scheduler uses two 32x128-bit internal SRAM memory cells, with the 32 corresponding to 32 time steps and the 2x128-bit corresponding to 256-bit pulses. Unlike conventional SRAM, this SRAM can be written to simultaneously, either single or multiple bits at a time. When the previous layer is not emitting pulses, the SRAM remains idle, effectively reducing memory accesses. During computations in the data processing system, the scheduler sends a 256-bit pulse to the neuron at each time step.

[0026] The cache is used to temporarily store the register parameters of neurons, with a total of 664 bits of cache space. At the beginning of the nth calculation, the top-level controller initiates a read signal to read the configuration parameters required for the n+1th time from the external SRAM and save them in the cache. Each cache has a 256x20-bit internal storage unit, 20 bits are the membrane voltage calculated by the neuron module at each time step, and 256 represents the 256 neurons under each core. The membrane voltage parameters of the first time step are obtained from the external SRAM (contained in the neuron register parameters), and the membrane voltage parameters of the other 31 time steps are read from the internal storage unit in the cache. Before the calculation of a single neuron begins, under the action of the neuron controller, the cache passes the 664-bit parameter file to the neuron module.

[0027] Each core has 256 neurons, and each neuron has 256 synaptic units for executing spiking neural networks. A single neuron in the neuron module is configured with 15 registers, namely membrane voltage, axon type, synaptic weight, leakage inversion flag, positive threshold, negative threshold, reset voltage, reset mode, router parameters and leakage weight. The total register size is 664 bits (see Figure 4 ), register multiplexing technology is used to multiplex the registers under a single neuron to 256 neurons, and the other cores also use the same processing method. 16 cores originally require 61440 registers, but after register multiplexing, only 240 registers are actually used. In the calculation process of the neuron module, two clock cycles are required to complete the calculation of an axon. The first clock cycle is used to obtain weight data, and the second clock cycle is used to complete the summation calculation. The specific calculation process is as follows Figure 5 As shown in the figure, the cache, scheduler, neuron controller and router modules work together to execute the pulse neural network in the neuron module, including:

[0028] Step 1: After all parameters are configured, the SPI host initiates the calculation start signal, which is then passed to the neuron controller via the top controller.

[0029] Step 2: The neuron controller initiates an activation data read request and reads 256-bit input image pulse data for one time step from the scheduler.

[0030] Step 3: The neuron controller initiates the initialization of the neuron module. At this time, the buffer sends the 664-bit register parameters required by the first neuron to the neuron calculation module.

[0031] In step 4, the neuron module parses the 256*2-bit weight index, 4*9-bit weight, and other parameters from the register parameters. The 2-bit index of an axon contains exactly 4 weights. The 256-bit image pulse then determines which of the 256 axons are valid (image pulse: 0 is invalid, 1 is valid). Finally, the weights under all valid pulses are added together to obtain a calculation result.

[0032] In step 5, the calculated result is compared with the threshold voltage to determine whether to issue a pulse. If a pulse is issued, the local router transmits the pulse data based on the target core location and resets the membrane voltage to a buffer. If a pulse is not issued, the buffer stores the membrane voltage (calculation result) for the current time step. After completing the calculation of a neuron, the neuron controller activates the switching neuron and jumps to step 3. This continues until the calculation of 256 neurons is completed, completing the calculation of one time step.

[0033] Step 6. In this embodiment, when the 10th neuron in the 32nd time step is calculated, the entire calculation process ends, one image recognition is completed, the core parameters are reset, and the scheduler is waited for to be updated to start the next recognition.

[0034] like Figure 6 As shown in the figure, this is a dedicated instruction packet for the neuromorphic data processing system, and the instruction packet can be divided into a control packet and a data packet. The control packet includes: global initialization, core settings, writing configuration files, writing activation data, initiating a start signal, and starting to read data. Each process corresponds to the state machine in the top-level controller module. The data packet includes: configuration data, pulse data, and core data required for a single neuron. When the control command is 0x01 (initialization), 0x05 (start signal), or 0x06 (read data), only a single command needs to be sent; when the control command is 0x02 (core settings), a sub-command needs to be sent again before the data packet can be sent. The types of sub-commands are 0x01 (core selection), 0x02 (activation position), and 0x03 (read position); when the control command is 0x03 (write configuration file) or 0x04 (write activation data), a data packet is sent immediately after the command is sent (see Figure 7 ).

[0035] like Figure 8The figure shows the structure of a router. Each router module has five FIFOs (First Input First Output) for storing packets from the east, south, west, north, and local cores. Each packet contains four registers: the target core's horizontal position (X), vertical position (Y), axon type, and transmission time. In the transmission direction, after receiving a pulse from a neuron module, the router first transmits it horizontally (X) based on the destination coordinates. After the horizontal transmission is completed, it then transmits it vertically (Y). If a packet is sent by a router east or north of the local core, the next core receives the packet using a router west or south. After several transmissions, the local routing FIFO sends the data to the scheduler for storage. For cores located at the four boundaries, transmission to routers outside the boundary is disabled.

[0036] The neuron controller controls the start and end of neuron computations. Both signals are sent by the SPI host, controlling each neuron transition and activation data conversion, completing buffer updates. The neuron controller also stores the activation pulse counts for each neuron at 32 time steps. In this embodiment, the activation pulse counts of 10 neurons in the output layer are transmitted to the top-level controller module and then to the SPI host via the SPI protocol. The 10 neurons correspond to the handwritten digits 0 to 9. The SPI host determines the maximum activation pulse count of the 10 neurons. The sequence number corresponding to the maximum value is the result of the image digit, and the entire neuromorphic data processing system completes a recognition cycle.

Claims

1. A neuromorphic data processing system based on an SPI bus, the system comprising: SPI master, SPI controller, top-level controller, memory controller, cache and core array; The SPI controller is used to bridge the SPI host and the top-level controller, establishing communication between the two using an adjustable data width. Data and instructions are transmitted through the MOSI channel with an 8-bit data width and the MISO channel with a 1-bit data width. The 8-bit data width of the MOSI channel is flexibly adjustable, and the specific data width is set by the SPI host. The memory controller is respectively connected to the top-level controller and the off-chip SRAM. The memory controller is used to generate the SRAM storage protocol and read and store data from the off-chip SRAM. The core array exchanges data with the top-level controller. The data processing method of the core array is based on the pulse neural network algorithm. Each core in the array includes five processing modules: a scheduler, a buffer, a neuron, a router, and a neuron controller. The scheduler is used to store activation data, the type of activation data is pulse data, and the pulse data includes two parts. The first part is the original input data encoded as pulse data and acts on the first layer of computing cores; the second part of the data comes from the pulse data output by the router after the calculation of the previous layer of computing cores is completed, and acts on other computing cores. The other computing cores are the hidden layers and output layers in the neural network; the buffer is used to store the configuration file read from the external SRAM, the parameters of the configuration file include weight data, leakage voltage, router, reset mode, reset voltage, threshold voltage, initial membrane voltage, and leakage reversal flag; The neurons are used to perform spiking neural network calculations. The input data required for the calculations comes from the scheduler and the buffer. The execution commands come from the core controller, and the final calculation results are passed to the router. The router is used to send and receive pulse data, and the data sent and received comes from the cores in the five directions of east, south, west, north, and local. The neuron controller is used to control the conversion of neurons, the conversion of activation data, and the recording of the number of neuron activations, and control the coordinated operation of various modules under each core. The neuromorphic data processing system performs data processing in the following manner: the SPI host sends an instruction packet to the top-level controller module via the SPI controller, and the top-level controller module receives and parses the instruction packet. The instruction packet includes a control packet and a data packet. The control packet includes a global initialization signal, a command to set core parameters, a configuration file, a write activation data command, a start calculation signal, and a read data signal. The data packet includes core setting data, pulse data, and single neuron data. The top-level controller module controls the data flow and working state of the core array through the instructions of the control package. The data flow includes configuration files, activation data, neuron parameters, time steps and calculation results. The working state includes initialization, start calculation, axon switching, neuron switching and stop calculation. The top-level controller module passes the core setting data and pulse data in the data packet to the core array, and the single neuron data in the data packet to the storage controller. The top-level controller module obtains the required configuration file data storage address from the two variables of neuron and core position, and reads data from the external memory according to the address and passes it to the core array.

Images