A Low-Cost and Low-Power Pulse Neural Network Accelerator

Abstract

The present invention relates to a low-cost, low-power spiking neural network accelerator. While ensuring the accelerator's scalability, it further optimizes neuron pulse data packets, thereby simplifying the router design of the on-chip network, thereby reducing hardware overhead. Furthermore, by designing an efficient deployment method for mapping spiking neural networks to the accelerator, all physical neurons can operate in parallel and communicate through the on-chip network. The spiking neural network accelerator of the present invention, in a preferred embodiment that reduces hardware overhead, is more suitable for embedded devices, etc., which are sensitive to cost and power consumption.

Description

Technical Field

[0001] The present disclosure relates to electronic information technology, and in particular to a low-cost, low-power pulse neural network accelerator. Background Art

[0002] In recent years, the increasing number of network layers has posed significant challenges to the computing power and memory devices of current hardware. High power consumption is a major pain point and difficulty in applying artificial intelligence to embedded vision devices, a key factor being the low energy efficiency of traditional processors.

[0003] Compared to traditional artificial neural networks (ANNs), spiking neural networks (SNNs) process discrete pulse trains, so weighted sums are calculated using addition rather than multiplication. This significantly reduces computational power and time. Furthermore, SNs perform addition only when there are input pulses, which are actually very sparse, whereas ANNs perform multiplication on all neurons for all inputs. Therefore, SNs can simultaneously simulate the spatiotemporal characteristics of human brain neural networks, achieving higher biological realism and better performance-to-power ratios. They are expected to replace traditional processors and achieve highly energy-efficient AI.

[0004] Current pulse neural network accelerators face a contradiction between scalability and hardware resource energy consumption overhead. That is, the scalable SNN accelerator has large hardware and energy consumption overhead, while the pulse neural network accelerator with smaller overhead has difficulty in expanding the number of integrated neurons. Summary of the Invention

[0005] The main purpose of the present invention is to solve the contradiction between the scalability and hardware resource energy consumption overhead faced by current pulse neural network accelerators, that is, the hardware and energy consumption overhead of scalable pulse neural network accelerators is large, while the number of integrated neurons of SNN accelerators with smaller overhead is difficult to expand.

[0006] Traditional on-chip (NoC) architectures aim to reduce transmission delays and router congestion. However, the time-division multiplexing and asynchronous nature of neurons in spiking neural networks eliminates transmission delays between neurons as a major bottleneck. While ensuring accelerator scalability, this invention further simplifies the design of NoC routers by optimizing neuron pulse packets, thereby reducing hardware overhead. Furthermore, by designing an efficient deployment method for mapping spiking neural networks to accelerators, all physical neurons can operate in parallel and communicate via the NoC.

[0007] On the one hand, the present invention proposes a specific structure of a low-cost and low-power pulse neural network accelerator as follows: the pulse neural network accelerator includes a neural core and a router; multiple neural cores are integrated through an on-chip network, and pulse data packets are transmitted between the neural cores through the router; the neural core includes a router interface module, a pulse data input cache, a control module, a storage module, a physical neuron module, an output port and an encoding module; the router interface module is used to receive pulse data packets from the router, and decapsulate and combine the pulse data packets, or encapsulate the pulse data packets and send them to the router; the pulse data input cache is used to receive and cache pulse data packets; the control module uses time division multiplexing to update the neuron status information in the storage module in a pipeline manner according to the pulse data packet information; the neuron module updates the physical neuron status based on the neuron status information in the storage module; the output port and the encoding module generate pulse data packets using the updated neuron status information in the storage module.

[0008] Preferably, the pulse data packet only includes timestamp information and a source neuron address; the source neuron is a neuron in an upper layer of the neuron deployed on the neural nucleus that receives the pulse data packet.

[0009] Preferably, the storage module includes a neuron storage unit and a synapse storage unit; the neuron storage unit is used to store the state information of the target neuron; the target neuron is a neuron deployed on the neural nucleus that receives the pulse data packet; the synapse storage unit is used to store the weights of the target neuron and the source neuron.

[0010] Preferably, the physical neuron is implemented using a leaky integral firing model.

[0011] Furthermore, the neuron state information includes neuron membrane potential, last neuron update time, and end time of neuron refractory period.

[0012] Preferably, the deployment includes the following steps:

[0013] S100, calculating the number of neurons in each layer of the fully connected spiking neural network;

[0014] S200, determining that the number of neurons in the current layer to be deployed is N1, and the number of neurons in the previous layer is N2;

[0015] S300: If a neural core can store N3 weight values and N4 neuron state information, the maximum number of neurons that the neural core can support deployment is N. Then:

[0016]

[0017] Where, The result of dividing N3 by N2 is rounded down; min is the minimum value function;

[0018] S400, if N2 is less than or equal to N, deploy all neurons of the current layer to be deployed to one neural core; otherwise, deploy the neurons of the current layer to be deployed to multiple neural cores in sequence;

[0019] When deployed on multiple cores, the next core selected has the shortest distance to the currently deployed core in terms of routing communication.

[0020] Preferably, the router consists of an arbitrator, a distributor and a first-in-first-out data buffer.

[0021] Preferably, one router is connected to four neural cores.

[0022] Preferably, the connection relationship of the routers is determined during the deployment process of the pulse neural network.

[0023] Compared with existing technologies:

[0024] The spiking neural network accelerator of the present invention is scalable and, by optimizing pulse data packets, simplifies router design, thereby reducing hardware overhead. Therefore, the spiking neural network accelerator of the present invention facilitates the integration of large-scale neurons and, in preferred embodiments, is particularly suitable for embedded devices that are sensitive to cost and power consumption.

[0025] On the other hand, the present invention provides a method for deploying a low-cost, low-power pulse neural network accelerator, the method comprising the following steps:

[0026] S100, calculating the number of neurons in each layer of the fully connected spiking neural network;

[0027] S200, determining that the number of neurons in the current layer to be deployed is N1, and the number of neurons in the previous layer is N2;

[0028] S300: If a neural core can store N3 weight values and N4 neuron state information, the maximum number of neurons that the neural core can support deployment is N. Then:

[0029]

[0030] Where, The result of dividing N3 by N2 is rounded down; min is the minimum value;

[0031] S400, if N2 is less than or equal to N, deploy all neurons of the current layer to be deployed to one neural core; otherwise, deploy the neurons of the current layer to be deployed to multiple neural cores in sequence;

[0032] When deployed on multiple cores, the next core selected has the shortest distance to the currently deployed core in terms of routing communication.

[0033] By adopting the above-mentioned deployment method of the pulse neural network accelerator, the pulse neural network can be efficiently mapped to the accelerator, so that all neurons work and communicate through the on-chip network. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0035] Figure 1 A schematic diagram of the structure of a pulsed neural network accelerator in an embodiment of the present invention;

[0036] Figure 2 is a schematic diagram of a neural nucleus structure in an embodiment of the present invention;

[0037] Figure 3 is a schematic diagram of a fully connected spiking neural network in an embodiment of the present invention;

[0038] Figure 4 Schematic diagram of deploying a pulse neural network on an accelerator in an embodiment of the present invention. DETAILED DESCRIPTION

[0039] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0040] The terms "including" and "having" and any variations thereof in the specification and claims of this application are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus that includes a series of steps or devices is not necessarily limited to those steps or devices expressly listed but may include other steps or devices not expressly listed or inherent to such process, method, product, or apparatus.

[0041] To make the purpose, technical solutions and advantages of this application clearer, the technical solutions of the present invention are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0042] In one embodiment, a low-cost, low-power pulse neural network accelerator is used, which includes a neural core and a router; multiple neural cores are integrated through an on-chip network, and pulse data packets are transmitted between the neural cores through the router; the neural core includes a router interface module, a pulse data input buffer, a control module, a storage module, a physical neuron module, an output port and an encoding module; the router interface module is used to receive pulse data packets from the router, and decapsulate and combine the pulse data packets, or encapsulate the pulse data packets and send them to the router; the pulse data input buffer is used to receive and cache pulse data packets; the control module uses time division multiplexing to update the neuron state information in the storage module in a pipeline manner according to the pulse data packet information; the neuron module updates the physical neuron state based on the neuron state information in the storage module; the output port and the encoding module generate pulse data packets using the updated neuron state information in the storage module.

[0043] The pulse neural network accelerator in this embodiment adopts an on-chip network integrated neural core, which has the advantages of high integration, low power consumption, low cost, and small size, and is conducive to application in embedded terminals.

[0044] Since neural networks involve input neurons, intermediate neurons, output neurons, and weights between two layers of neurons, a corresponding storage module must be set up in the neural core to store information used for calculation. In a preferred embodiment, the storage module on the neural core is divided into a neuron storage unit and a synapse storage unit. When the pulse neural network is deployed on the neural core, the neuron storage unit is used to store the state information of the target neuron, and the addresses are stored sequentially starting from 0. The target neuron is the neuron deployed on the neural core that receives the pulse data packet; the synapse storage unit is used to store the weights of the target neuron and the source neuron, starting from address 0 and sequentially storing the weight values of the source neurons connected to the first target neuron. After the weight values of all source neurons connected to the first target neuron are stored, the weight values of all source neurons connected to the second target neuron are stored, and so on, until the weight values of all source neurons connected to the last target neuron are stored. When obtaining the weight value, the current target neuron address and the source neuron address in the pulse data packet are read.

[0045] One of the objectives of the present invention is to reduce hardware overhead. To achieve this goal, the present invention adopts a time-division multiplexing architecture in the internal neuron design to reduce the number of physical neurons, and at the same time adopts a pipeline processing method for updating the physical neuron state, thereby improving the accelerator's computing power. The implementation of the pipeline processing method is realized in the present invention through the control module in the neural core, that is, according to the source neuron address in the pulse data packet, the target neuron state is updated in a pipeline manner by multiplexing neurons. Specifically, the first clock cycle reads the state information of the current target neuron and the corresponding synaptic weight, which is the weight between the source neuron and the current target neuron; the second clock cycle updates the state of the current target neuron and reads the state information and weight of the next target neuron; the third clock cycle writes back the state information of the current target neuron and updates the state information of the next target neuron. At the same time, the next target neuron is used as the current target neuron and the state information and weight of the next target neuron of the current target neuron are read; and so on, until all target neural cores corresponding to this neural core are updated. Compared with the ODIN of the University of Leuven, the pipeline method adopted by the present invention to update the neuron state reduces the update time by half.

[0046] To further reduce hardware overhead, the present invention implements a leaky integral firing model for physical neurons. When a physical neuron receives an input pulse, it first determines whether it is in its refractory period. If so, the membrane potential remains at its reset value. If not, it determines whether the membrane potential is greater than a threshold. If so, the corresponding weight is accumulated and the corresponding leakage value is subtracted. If so, an output pulse is generated and the membrane potential is reset. Therefore, the neuron status information that needs to be stored in the storage module includes the neuron's membrane potential, the last neuron update time, and the end time of the neuron's refractory period.

[0047] Furthermore, because spiking neural networks are fully connected, if spike packets carry the destination neuron address, this results in higher router overhead, including energy consumption and hardware resources. Therefore, the present invention optimizes the content carried by spike packets, sending spike packets containing only timestamp information and the source neuron address along a path that covers all destination neurons, further reducing on-chip network hardware and energy consumption. The source neuron is the neuron in the layer above the neuron deployed on the neural core that receives the spike packet.

[0048] Although the pulse data packet in the present invention does not indicate "pulse", if a pulse data packet is transmitted, it means there is a pulse. The input pulse is generated by the physical neuron of the previous level / layer and transmitted to the current physical neuron.

[0049] In this invention, communication between neural cores is accomplished through routers. Routers act as communication bridges between neural cores. The number of neural cores can be increased by adding physical connections to the routers, providing scalability. The number of neural cores that a router can connect to is determined based on actual needs, taking into account transmission delays and routing congestion.

[0050] In one embodiment, Figure 1 As shown, each router is connected to four neural cores, that is, one router is responsible for the communication of four neural cores, and routers are connected through links.

[0051] The present invention simplifies router design. The router consists of an arbiter, a distributor, and a first-in, first-out data buffer. Pulse data packets sent by the neural core are encoded into communication packets based on address event representations by the router interface module and sent to the router via the arbiter. The router routes received address-representation communication packets to the destination router, which then distributes them to the connected neural core, or directly to the connected neural core. The connections between routers are configured or determined during the deployment of the spiking neural network to the neural core.

[0052] In one embodiment, the Figure 2 The neural core shown in the figure. In the figure, the neural core includes a pulse data input buffer module (AER_Input_FIFO), a neuron state storage unit (Neuron_mem), a synapse storage unit (Synapse_mem), a control module, a physical neuron module (LIF_neuron) and a router interface module. Among them, the neuron state storage unit (Neuron_mem) and the synapse storage unit (Synapse_mem) are storage modules of the accelerator. In this embodiment, the neuron state storage unit (Neuron_mem) uses a 2KB static random access memory (SRAM) to store the state information of the neuron, and the synapse storage unit (Synapse_mem) uses a 64KB static random access memory to store the weights between the source neuron and the target neuron. The spike data input buffer module uses 1KB of space to cache received spike packets. The router interface module encodes spike packets into communication packets based on timestamp information and the target neuron's emission address. It also receives communication packets distributed by the router, decodes them, and combines them into spike packets. The neural core also has an APB interface module, which is used to configure network parameters into the neuron state storage unit and synapse storage unit.

[0053] For a given neural network accelerator, it is configurable and can be configured according to different spiking neural networks. In one embodiment, the following method for deploying a spiking neural network is used. The specific steps are as follows:

[0054] S100, calculating the number of neurons in each layer of the fully connected spiking neural network;

[0055] S200, determining that the number of neurons in the current layer to be deployed is N1, and the number of neurons in the previous layer is N2;

[0056] S300: If a neural core can store N3 weight values and N4 neuron state information, the maximum number of neurons that the neural core can support deployment is N. Then:

[0057]

[0058] Where, The result of dividing N3 by N2 is rounded down; min is the minimum value function;

[0059] S400: If N2 is less than or equal to N, all neurons in the current layer to be deployed are deployed on one neural core; otherwise, the neurons in the current layer to be deployed are deployed on multiple neural cores in sequence.

[0060] When deployment on multiple neural cores is required, the next neural core selected has the shortest distance from the currently deployed neural core in terms of routing communication.

[0061] In the above-mentioned process of mapping the spiking neural network to the accelerator, a reasonable and efficient deployment strategy was implemented. That is, when the spiking neural network is deployed, if the number of neurons in a certain layer is less than or equal to the maximum number of neurons supported by the neural core, then the layer is only deployed on one neural core; otherwise, the layer is deployed in sequence on multiple neural cores from near to far based on the routing communication distance.

[0062] In one embodiment, Figure 3 The pulse neural network in the present invention is deployed on the accelerator, and the deployment diagram is shown in FIG. Figure 4 shown. Figure 3 In the fully connected spiking neural network, the input layer has 784 neurons, the two hidden layers have 500 neurons each, and the output layer has 10 neurons. Figure 4 In the accelerator structure shown, it is assumed that the neural core can store 256 neuron state information and 65536 weight information.

[0063] Now let's deploy the first middle layer. This layer has 500 neurons, and the connected input layer has 784 neurons. Considering that the synaptic memory can store a maximum of 65,536 weights, the maximum number of neurons that this neural core can support is 83, calculated as follows:

[0064]

[0065] To ensure uniform distribution, the maximum number of neurons that this neural core can support is set to 64. Therefore, the 500 neurons in this layer will be deployed on 8 neural cores, with source addresses from 0 to 783 and target addresses from 0 to 63, 64 to 127, 128 to 191, 192 to 255, 256 to 319, 320 to 383, and 384 to 499.

[0066] When the accelerator performs calculations, after the eight neural cores in the first layer have completed their calculations, the output pulse data packets are stored in the router node connected to them, and then the router node transmits them to the router node corresponding to the next layer of the network through the link. The router node then distributes them to the four neural cores connected to it, and the 500 neurons of the second intermediate layer are deployed on these four neural cores. Figure 4 shown.

[0067] Using the aforementioned spiking neural network accelerator deployment method, a spiking neural network can be efficiently mapped to the accelerator, ensuring that all neurons are active and communicating via the on-chip network. If the deployed spiking neural network changes, it can be reconfigured on the accelerator.

[0068] Any low-cost, low-power pulse neural network accelerator and its pulse neural network method for deploying neurons on the accelerator provided in the embodiments of the present application are based on the same design concept, and the technical means in any embodiment of the present application can be freely combined, and the technical means obtained by the combination are still within the scope of protection of the present application.

[0069] Those skilled in the art will appreciate that the features described in the various embodiments and / or claims of this application may be combined and / or coupled in various ways, even if such combinations and / or couplings are not explicitly described in this application. In particular, without departing from the spirit and teachings of this application, the features described in the various embodiments and / or claims of this application may be combined and / or coupled in various ways, and all such combinations and / or couplings fall within the scope disclosed in this application.

[0070] Specific embodiments are used in this application to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application and is not used to limit this application. For those skilled in the art, changes can be made in the specific implementation methods and application scope based on the ideas, spirit and principles of this application. Any modifications, equivalent replacements, improvements, etc. made by them should be included in the scope of protection of this application.

Claims

1. A low-cost, low-power pulse neural network accelerator, characterized by: The pulse neural network accelerator includes a neural core and a router; Multiple neural cores are integrated through on-chip networks, and pulse data packets are transmitted between neural cores through routers; The neural core includes a router interface module, a pulse data input buffer, a control module, a storage module, a physical neuron module, an output port and an encoding module; The router interface module is used to receive pulse data packets from the router, and decapsulate and combine the pulse data packets, or encapsulate the pulse data packets and send them to the router; The pulse data input buffer is used to receive and buffer pulse data packets; The control module updates the neuron state information in the storage module in a pipeline manner using a time division multiplexing method according to the pulse data packet information; The neuron module updates the physical neuron state based on the state information of the neurons in the storage module; The output port and encoding module store the updated neuron status information in the module to generate a pulse data packet; The pulse data packet only includes timestamp information and a source neuron address, and the source neuron is a neuron in a layer above the neuron deployed on the neural nucleus that receives the pulse data packet. The deployment includes the following steps: S100, calculating the number of neurons in each layer of the fully connected spiking neural network; S200, determining that the number of neurons in the current layer to be deployed is N1, and the number of neurons in the previous layer is N2; S300: If a neural core can store N3 weight values and N4 neuron state information, the maximum number of neurons that the neural core can support deployment is N. Then: Where, The result of dividing N3 by N2 is rounded down; min is the minimum value function; S400, if N2 is less than or equal to N, deploy all neurons of the current layer to be deployed to one neural core; otherwise, deploy the neurons of the current layer to be deployed to multiple neural cores in sequence; When deployed on multiple cores, the next core selected has the shortest distance to the currently deployed core in terms of routing communication.

2. The spiking neural network accelerator according to claim 1, wherein: The storage module includes a neuron storage unit and a synapse storage unit; The neuron storage unit is used to store the state information of the target neuron; The target neuron is a neuron deployed on a neural nucleus that receives a pulse data packet; The synaptic storage unit is used to store the weights of the target neuron and the source neuron.

3. The pulse neural network accelerator according to claim 1, characterized in that The physical neuron is implemented using a leaky integral firing model.

4. The pulse neural network accelerator according to claim 3, characterized in that The neuron status information includes neuron membrane potential, last neuron update time, and end time of neuron refractory period.

5. The pulse neural network accelerator according to claim 1, characterized in that The router consists of an arbitrator, a distributor and a first-in-first-out data buffer.

6. The pulse neural network accelerator according to claim 1, characterized in that: One router connects four neural cores.

7. The pulse neural network accelerator according to claim 1, characterized in that: The connection relationship of the routers is determined during the spiking neural network deployment process.

8. A method for deploying a low-cost, low-power pulse neural network accelerator, characterized in that: The method is applied to the low-cost, low-power pulse neural network accelerator according to claim 1, and the method comprises the following steps: S100, calculating the number of neurons in each layer of the fully connected spiking neural network; S200, determining that the number of neurons in the current layer to be deployed is N1, and the number of neurons in the previous layer is N2; S300: If a neural core can store N3 weight values and N4 neuron state information, the maximum number of neurons that the neural core can support deployment is N. Then: Where, The result of dividing N3 by N2 is rounded down; min is the minimum function; S400, if N2 is less than or equal to N, deploy all neurons of the current layer to be deployed to one neural core; Otherwise, the neurons of the current layer to be deployed are deployed to multiple neural cores in sequence; When deployed on multiple cores, the next core selected has the shortest distance to the currently deployed core in terms of routing communication.

Images