Hardware architecture and operating method for spiking neural networks

Abstract

The present invention provides a hardware architecture for a spiking neural network characterized by a combination of a fully parallel architecture and a time-division multiplexed architecture.

Description

TECHNICAL FIELD

[0001] The present invention relates to the field of computing architecture, and more particularly, to a hardware architecture for spiking neural networks and a method of operating the network. BACKGROUND

[0002] Machine learning has triggered unprecedented interest in research and industry due to recent achievements in many application environments such as image classification and object recognition. However, since the configuration of these systems requires enormous computing power, they are not suitable for embedded systems. To address such limitations, many researchers are studying brain-inspired computing, which is an alternative to conventional von Neumann architecture-based computers (CPUs / GPUs) that meet the requirements of computing performance. However, this approach has energy efficiency problems and requires the design of neuromorphic hardware circuits that are suitable for both parallel and distributed computing.

[0003] Artificial intelligence (AI) has increasingly attracted the attention of industrial institutions and research institutions over the past decade. Artificial neural networks (ANNs), which are derived from and inspired by the biological brain, have become the most well-known and most frequently used form of AI. Although ANNs have received much attention in recent years, they originated in the 1940s with the advent of the first computer. Subsequent work and progress led to the development of a variety of ANN models. However, many of these models remained theoretical and were not implemented for industrial use at the time.

[0004] Recently, these algorithms have become competitive due to two factors: first, modern computers have reached sufficient computing performance to handle ANN training and inference; second, the amount of available data has grown exponentially, meeting the need for a large amount of training data for ANNs.

[0005] However, the energy and hardware resource concentration brought about by complex forms of ANN computing does not match another current emerging technology, namely, IoT (Internet of Things) and edge computing. In order for ANNs to be executed in embedded environments, dedicated hardware architectures must be deployed for ANN acceleration. In this case, the design of neuromorphic structures is particularly interesting when combined with research into spiking neural networks.

[0006] Spiking neural networks (SNNs) for deep learning and knowledge representation are a current issue, especially with a group of researchers interested in both neuroscience and machine learning. Several specific hardware solutions have been proposed in the literature, but they are only solutions isolated from the overall space design of network topology, which is often constrained by the characteristics of circuit architecture.

[0007] The authors' article "Information encoding and hardware architecture for spiking neural networks", 2019 22nd Euromicro Conference on Digital System Design (DSD), IEEE, 28 August 2019, pages 291-298, XP033637577, proposes two different designs of hardware architectures for spiking neural networks: a time-multiplexed architecture (TMA) and a fully-parallel architecture (FPA).

[0008] These architecture solutions are classical models for hardware implementation. In the case of SNNs, these structures do not take advantage of the activity reduction throughout the depth of the network. Indeed, a more accurate analysis of the dynamics of these networks shows that most of the spikes are generated by the input layer. The first neural layer, especially in the case of convolutional layers, acts as a low-pass filter, which drastically reduces the number of output spikes. Thus, working end-to-end in a fully-parallel way does not exploit the number of HW processing elements and causes energy consumption. Moreover, the implementation of FPA does not support event-based processing and operates in a frame-based way. On the other hand, working end-to-end in a time-multiplexed way requires processing all the spikes sequentially. This leads to a time overhead in the first neural layer, which remains a large number of spikes.

[0009] The authors recommend a radically different approach, which consists in generating a structure that best supports the network topology.

[0010] There is therefore a need for a solution to the aforementioned problems. And there is a need for a neuromorphic hardware circuit that can be adapted both to parallel computing and to distributed computing. The present invention provides such a solution. SUMMARY

[0011] According to a first embodiment of the invention, there is provided a system as further described in the appended independent claim 1.

[0012] The object of the present invention is a neuromorphic hardware architecture adapted to the implementation of a spiking neural network. In particular, the present invention provides a hybrid architecture combining a fully-parallel hardware layer and a time-multiplexed hardware layer. The hybrid architecture of the present invention meets application-specific constraints.

[0013] Advantageously, a novel hybrid architecture is described that combines the advantages of time-multiplexed hardware and parallel hardware implementations.

[0014] Indeed, in this architecture, a first hidden layer, called fully-parallel hidden layer, is implemented in a fully-parallel processing module, and a plurality of deeper hidden layers, called time-multiplexed hidden layers, are implemented in a time-multiplexed processing module. This hybrid architecture is well suited to the spike- selective encoding method.

[0015] In SNNs, the hybrid architecture enables efficient processing of spikes by adapting parallelism to the activity of each layer. The hybrid architecture model breaks the uniform processing of FPA and TMA. Therefore, specific control units for processing spikes asynchronously are implemented. The hybrid architecture guarantees optimal event-based processing where units are activated only when a spike occurs. Moreover, the hybrid architecture provides optimized energy consumption by adjusting parallelism and latency.

[0016] The hybrid architecture uses a neural encoding scheme to convert input data into a sequence of spikes with an encoding paradigm that is characterized by a lower number of spikes propagating in the network.

[0017] Advantageously, the number of spike events to be processed is reduced while maintaining the same classification accuracy. By doing so, the amount of power consumed by the hardware is reduced.

[0018] The hybrid architecture has been developed in VHDL and simulated in Register Transfer Level (RTL).

[0019] Most of the spike activity in the network is located in the first layer. Therefore, the first hidden layer is the most required layer during processing. To exploit this aspect, the designed hybrid architecture mixes a Time-Multiplexed Architecture (TMA) and a Fully Parallel Architecture (FPA), where, one, the first two layers are implemented by using a neural kernel module with a structure similar to FPA; and two, the remaining layers are time-multiplexed layers where one NPU is used per layer as in TMA. In the case of large spike neural networks, the time-multiplexed part is driven by a network controller that manages the NPUs and connects the NPUs to the SDRAM holding the logical weights of the NPUs, as well as retrieves the weights from the external SDRAM memory and forwards the weights to the corresponding NPUs. This novel hybrid architecture is especially suitable for using spike selection encoding where the spike activity is concentrated in the first layer.

[0020] The hybrid architecture of the present invention exploits the sparsity of spike activity that increases as the network is entered into deeper layers. This novel hybrid architecture has fully parallel computing kernels for the most required layers and time-multiplexed computing unit architecture for deeper layers, which seems to be one of the most suitable approaches for implementing future deep SNNs in embedded systems when combined with the proposed spike selection encoding.

[0021] The hybrid architecture of the present invention is suitable for the implementation of fully connected based SNNs and the implementation of spike convolutional neural networks.

[0022] The claimed hardware architecture for spike neural networks comprises:

[0023] - A pulse generator module for receiving input pixels and generating pulse streams;

[0024] - A neural nucleus module used to receive pulse streams and filter them to generate a reduced number of pulses;

[0025] - A neural processing unit module for processing a reduced number of pulses;

[0026] - A classification module used to select the winner class for output;

[0027] The hardware architecture is characterized by a neural kernel module comprising a hidden fully parallel layer to process the received input pulses in parallel, and a neural processing unit module comprising multiple hidden time-division multiplexing layers to process a reduced number of pulses sequentially.

[0028] According to the various embodiments:

[0029] - The pulse generator module is implemented as a neural coding function, such as rate coding or pulse selection coding.

[0030] - The neural nucleus module includes an input layer that receives pulse streams and a first hidden layer implemented as a fully parallel circuit to process the pulses.

[0031] - Each of the multiple hidden layers includes a neural processing unit module, thereby simulating a time-division multiplexing layer.

[0032] - The classification module is the Termination Deltalike module.

[0033] - Pulse streams are processed in an event-based data model, in which only pulse events are processed by each layer of the architecture.

[0034] - The pulse stream is processed in a frame-based data mode, in which each "0" and "1" in the input frame is processed by each layer of the architecture.

[0035] The present invention also requests that the requested hybrid architecture be included in a field-programmable gate array (FPGA) or application-specific integrated circuit (ASIC).

[0036] The present invention also proposes a method for operating the requested hybrid architecture.

[0037] Further advantages of the invention will become apparent to those skilled in the art upon examination of the accompanying drawings and detailed description. Any additional advantages are intended to be incorporated herein. Attached Figure Description

[0038] Embodiments of the invention will now be described by way of example with reference to the accompanying drawings, in which the same reference numerals denote similar elements, and in the drawings:

[0039] Figure 1a A general block diagram of the hardware hybrid architecture of the present invention is shown;

[0040] Figure 1b One implementation of the hardware hybrid architecture in an embodiment for large SNNs and frame-based spike data is shown;

[0041] Figure 2a A detailed block diagram of the neural core module of the present invention in an embodiment is shown;

[0042] Figure 2b Another embodiment of the neural core module of the present invention is shown;

[0043] Figure 3 A detailed block diagram of the neural processing unit module of the present invention in an embodiment is shown;

[0044] Figure 4a and Figure 4b A detailed block diagram of the classification module of the present invention in two embodiments is shown;

[0045] Figure 5 A detailed block diagram of the network controller module of the present invention in an embodiment for the embodiment is shown;

[0046] Figure 6 is a flowchart of the general steps for operating the hybrid architecture of the present invention;

[0047] Figure 7 is a flowchart of the steps for operating the neural core module of the present invention in an embodiment;

[0048] Figure 8 is a flowchart of the steps for operating the neural processing unit module of the present invention in an embodiment; and

[0049] Figure 9 A block diagram of the IF neuron module in an embodiment is shown. DETAILED DESCRIPTION

[0050] Before the description of the drawings, reference is made to the article published by the inventor entitled “Exploration of Spatial Design of Hardware Spiking Neurons for Embedded Artificial Intelligence”, 2019, which is incorporated herein in its entirety.

[0051] First reference is made to a non-limiting example Figure 1a A general architecture of the hardware hybrid architecture of the present invention is described. The hybrid architecture of the present invention allows for processing event-based data, where only the spiking events (e.g., input data equal to ‘1’) are processed by each layer of the architecture, or for processing frame-based data, where each ‘0’ and ‘1’ in the input frame is processed by each layer of the architecture.

[0052] The spike event represents the address of the neuron that fired the spike in the previous layer of the architecture.

[0053] The system 100 is illustrated as having several functional block circuits, including a spike generator module 102, a fully parallel layer or a neural core (NC) module 104 implementing a hidden fully parallel layer, a time-multiplexed layer or a neural processing unit (NPU) module 106 implementing a plurality of hidden time-multiplexed layers (106-1, 106-i to 106-n), and a Termination Delta or classification module 108.

[0054] The general operation of the system 100 is to generate a stream of spikes from the input pixels of an image, which is then input to the neural core 104 where the spikes are filtered in a parallel processing. The reduced number of spikes is then processed by a plurality of neural processing units 106-i in a time-multiplexed approach; and the final operation allows the classification and selection of the winner output class.

[0055] Inspired by neuroscience, information (input pixels) is encoded into spikes by the spike generator module 102. In fact, the neuron model mimics the synaptic communication mechanism based on action potentials and biological neurons. Information is thus represented as a stream of spikes by a variety of neural coding techniques (to name a few, such as rate coding, spike selection, single burst). The skilled person will refer to the previously cited articles by the inventor for more details on these methods.

[0056] The neural core module 104 of the preferred embodiment is illustrated in Figure 2a The neural core module 104 of the preferred embodiment is illustrated in

[0057] The reduced pulse stream output from the FP layer 204 is input to the neural core control module 206. The neural core control module consists of a '1 :N' counter, a multiplexer (MUX) and an output first in first out (FiFo) buffer. It can be noted that 'N' is the number of neuron circuits of the FP layer 204. When the N neuron circuits of the FP layer 204 process the input pulse events (i.e. pulses originating from the input pulse stream) in parallel, the output pulses of the N neuron circuits are sequentially connected to the write enable of the output FiFo buffer by the multiplexer MUX. The MUX block is configured to select the output pulses one after the other by using the addresses (@Neuron) given by the 1 :N counter. These addresses are also connected to the input data of the FiFo module. In case the output pulse is high (pulse = 1), the output of the counter (i.e. the address of the neuron) is written into the output buffer (FiFo). Once the counter finishes the forwarding of all output pulses of the FP layer 204, the counter resets the count and repeats the same procedure for the next pulse.

[0058] Next, the output of the neural core 104 - the 'output events' - become the input of the time-multiplexed part of the hybrid architecture.

[0059] Figure 7 is a flow chart of the steps for operating the neural core module shown in Figure 2a .

[0060] Step 700 starts with reading the input pulse address (@In) and the'stop_network' signal provided by the termination increment module 108. In one aspect 703, the process allows to verify if the'stop_network' signal is equal to "1" to end the process 700. In another aspect 704, the input pulse address @In is sent to the neuron circuits of the FP layer to perform the integrate and fire rule. For each neuron, the input address @In is used to retrieve the weights corresponding to the cumulative internal potential of the neuron S i . As in 706, each cumulative potential is compared to the threshold "TH" in a parallel way. When the potential is higher than the threshold, a pulse is fired, i.e. pulse = 1, and the potential is updated by reducing it to the threshold "TH". These pulses are then used in the FiFo to write the output pulse address as a pulse event. A multiplexer controlled by a counter is used to forward the pulses one after the other 708 in sequence. If a pulse is fired 710, the address of the neuron that has fired the pulse (pulse event) is saved in the FiFo buffer 711. Once the counter has forwarded all the pulses, it is verified in 712 that "count = N-1" 712, the count is reset (count = 0) 713 and the process is repeated by reading new inputs (@In and stop_network) 702.

[0061] Advantageously, since the number of pulses is drastically reduced by the FPA module, such parallel computation is not required and the plurality of NPU modules 106-i are implemented in a time-multiplexed architecture to allow sequential processing of the pulses. The number of computation cycles is as many as the number of logical neurons implemented in the NPU and the number of time-multiplexed (TM) layers is predefined for a specific machine learning application.

[0062] The output of the last NPU, i.e. the output TM layer, becomes the input of a classification module (also designated as a terminal increment, maximum terminal or winner class module) 108 which determines whether to end the classification process by determining whether a sufficient number of pulses has been received to classify the input image. If not, the process is repeated or the process is stopped'stop processing'.

[0063] Figure 6 is a flowchart regarding the general steps of operating the hybrid architecture of the present application (e.g. as shown in Figure 1a the above.

[0064] The process 600 starts with a first step 602 of loading or receiving input data.

[0065] Next, the process allows the pulse generator module to generate a stream of pulses from the input data 604.

[0066] In a next step 606, the process allows the neural kernel module to process the stream of pulses by reducing the number of pulses in a fully parallel processing and allows to generate output events.

[0067] Next in steps 608-1 to 608-n, the process allows to sequentially process each output event by a plurality of neural processing units.

[0068] In a next step 610, the process allows to process the output of the last NPU by a terminal increment or classification module to determine the winner class. During the last step, if the terminal increment determines the winner class, the process allows to activate a'stop_network' signal to stop the process.

[0069] It should be understood that all steps of the process 600 work in a pipelined manner over time, optimally using the components of the architecture. For example, while step 602 loads the next input data, the pulse generator module progressively converts the pixels of the previously produced stream of pulses. At the same time, the neural kernel processes these input pulses, the plurality of NPUs processes other recent data, and the terminal increment verifies the classification of the current data received from the last NPU (608-n).

[0070] Turning to Figure 1bAnother embodiment of the hardware hybrid architecture of the present invention shows an embodiment adapted to process frame-based (non-event-based) input data, where each '0' and '1' in the input frame is processed by each layer of the architecture. The system also comprises a network controller 110 and a memory 112 to cover large spiking neural networks.

[0071] The network controller 110 allows to manage the addresses and weights used for processing. The network controller is coupled to a memory 112 capable of storing the weights. The memory usage is a general limitation of the SNN architecture, as the parameters and activity of the neurons must be stored. From this perspective, the on-chip memory of the FPGA is not enough to process deeper networks requiring a considerable memory size. Therefore, an external memory is preferably used to overcome this problem. To enhance the storage capabilities of the FPGA structure, SDRAM is used in the preferred embodiment. The network controller module connects the other modules to the external memory.

[0072] Figure 2b A variant of the neural core module 104 of the present invention adapted to process frame-based input data is shown. In frame-based or non-event-based data, the input spikes do not appear as events indicating the pixel address of the emitted spike. However, the data appears as a series of '0's and '1's. A spike equal to '0' corresponds to a pixel that has not emitted a spike and a spike equal to '1' corresponds to a pixel that has emitted a spike. To process this kind of data, a first hidden counter is used to indicate to the neuron circuit the address of the input spike equal to '1' to retrieve the appropriate weight and then perform the same processing as described in the Figure 2a

[0073] The "IF neuron module" integrates the spikes coming from the input neuron module and generates spikes according to the "integrate and fire" rule. The weights are stored in registers, so that each "IF neuron module" has its own weight stored in a dedicated register. The number of IF neuron modules is the same as the number of logical neurons in the FP layer. The output of the IF neuron module is stored in a FiFo buffer as a spike event, a counter module indicates the corresponding neuron address to be stored. The input neuron forwards the input spikes one by one, where each spike indicates the address of its origin (e.g. the address of a pixel in the image). The input spike addresses of these input pixels are passed to the hidden FP layer neurons. The FP layer neurons use these addresses to access their on-chip memory weights to retrieve their appropriate synaptic weights and then perform the "integrate and fire" rule. A counter (1:N) controls the MUX component to read the output spikes of the hidden FP layer neurons and store them in the output FiFo buffer.

[0074] Figure 3 ​A detailed block diagram of the neural processing unit module 106-i showing one hidden time-multiplexed layer of the application in an embodiment. The NPU is used to emulate the time-multiplexed layer. When there is an input event to be processed by the NPU, first, the hardware neuron 308 is enabled by the NPU controller 302 to retrieve the address of the logical neuron it represents from the counter 304 and the corresponding weights from the storage block 306. Second, the hardware neuron operates its computation and whenever it fires, the output spike is stored in the FiFo module 310 as a spike event.

[0075] The single IF neuron module operates sequentially on all neurons in the layer (implementation of the IF neuron module is shown in Figure 9 Further, the NPU comprises a FiFo memory module 310, a counter module 304 and a NPU controller 302. These modules are connected as shown in Figure 3 to form the NPU which processes spike events in a coherent manner. Except for the NPU controller, all other modules are used by the NPU to accomplish their dedicated tasks as previously described. The goal of the NPU controller is to manage the different NPU components to sequentially trigger the logical neurons, allowing the provision of valid weights and activity to the hardware neurons. Further, the NPU controllers of different NPU are connected to each other to guarantee network level synchronization. Synchronization is needed because the output classification process (termination module) depends on the order of arrival of the spikes. Thanks to the NPU controller and the counter, several logical neurons can be time-multiplexed and thus computed in a single NPU.

[0076] Figure 8 is a flowchart of the steps of operating the neural processing unit module shown in Figure 3

[0077] The process 800 starts with a first step 802 of loading the input events (empty input signal and stop processing signal). The process ends 803 whenever the termination increment module activates the stop processing signal. Otherwise, the NPU checks for the presence of an input event by verifying the status of the empty input signal "i_Empty" 804. Then, depending on the layer type (i.e. fully connected layer or convolutional layer) 806, the address of the logical neuron is forwarded to the hardware neuron to retrieve the internal activity and weights to perform the integrate and fire rule 808. If the spike is high (spike = 1), the output of this neuron is saved in the FiFo buffer 810. The process controlled by the counter 812 is repeated for all logical neurons of the layer. Once all neurons are processed, new inputs are loaded 802 to compute the next input spike.

[0078] Figure 4a and Figure 4b ​A detailed block diagram of the classification module 108 of the present application in the maximum termination and termination increment embodiments is shown. Before starting the description, a quick reminder about the class selection process is given. First, note that each output neuron corresponds to a data class. During inference, the winner class is selected as the maximum value pulse output neuron. In the termination increment ( Figure 4a ) process, the class prediction is implemented when the maximum value pulse neuron has an increment value that is greater than the second largest value pulse neuron. In addition, in the maximum termination ( Figure 4b ), the classification process is completed each time an output neuron (maximum value pulse neuron) reaches the maximum value pulse. The increment value and the maximum value are user defined parameters, usually set to 4.

[0079] For the current hybrid architecture design, to select the output winner class, the termination increment or the maximum termination is preferred because they provide the most advanced accuracy and fast class selection. Figure 4a and Figure 4b The internal structure of these modules is shown. The input of the module is a vector "activation" containing the output activity of the SNN (the number of pulses emitted so far by each output neuron).

[0080] In one aspect, in the termination increment module, two maximum sub-modules are designed to detect the maximum value of the array, which is used to determine the winner class and stop the process. The first maximum sub-module, Maxl, detects the maximum value of the output activation vector, and the second maximum sub-module, Max2, detects the second maximum value of the same vector. Then the difference between the output of the Maxl module and the output of the Max2 module is calculated. Finally, if the difference is greater than a threshold increment value, the class corresponding to the Maxl module is designated as the winner.

[0081] On the other hand, the maximum termination module incorporates only one maximum block that returns the index of the output neuron with the highest pulse activity and the activity of that output neuron. Then, the activity is compared with a user-defined threshold maximum value. If the maximum pulse activity is greater than the maximum value, the corresponding output neuron is designated as the winner class and the process ends.

[0082] Figure 5 A detailed block diagram of the network controller module 110 of the present application in the embodiments is shown. The network controller module is a combination of a FiFo module (queue) 502 and a demultiplexer (DEMUX) 504. The FiFo module accesses the SDRAM according to the NPU requests with a first come first served policy, i.e., when an NPU requests a weight, the request is put in the FiFo queue. Then, each time a weight is ready, it is transferred via the DEMUX block by selecting the corresponding NPU module.

[0083] It should be understood that, although the present application has been specifically shown and described herein with reference to the preferred embodiments, various changes in form and details can be made therein without departing from the spirit and scope of the application. The present application can be advantageously implemented on a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC).

Claims

1. A hardware architecture for a spiking neural network comprising: - a spike generator module (102) for receiving input pixels and generating a spike stream; - a neural kernel module (104) for receiving the spike stream and filtering the spike stream to generate a reduced number of spikes; - a neural processing unit module (106) for processing the reduced number of spikes; - a classification module (108) for selecting an output winner class; the hardware architecture being characterized in that the neural kernel module (104) comprises a hidden fully parallel layer to process in parallel the received input spikes and in that the neural processing unit module (106) comprises a plurality of hidden time-division multiplexing layers to process sequentially the reduced number of spikes.

2. The hardware architecture of claim 1, wherein, the spike generator module is implemented as a neural coding function.

3. The hardware architecture of claim 1 or 2, wherein, the neural kernel module (104) further comprises an input layer (202) receiving the spike stream, a fully parallel layer consisting of neurons processing input spikes in parallel, and a control module (206) sequentially reading output spikes from the fully parallel layer (204) and storing the output spikes to an output FiFo buffer.

4. The hardware architecture of claim 1 or 2, wherein, each of the plurality of hidden time-division multiplexing layers comprises a neural processing unit module to emulate a time-division multiplexing layer.

5. The hardware architecture of claim 1 or 2, wherein, the classification module is a terminal delta class module.

6. The hardware architecture of claim 1 or 2, wherein, the spike stream is event-based data, wherein only spike events are processed by each layer of the architecture.

7. The hardware architecture of claim 1 or 2, wherein, the spike stream is frame-based data, wherein each "0" and "1" in input frames are processed by each layer of the architecture.

8. The hardware architecture of claim 1 or 2, wherein, the spiking neural network is a fully connected spiking neural network or a spiking convolutional neural network.

9. The hardware architecture of claim 2, wherein, the neural coding function is a rate coding or a spike selection.

10. A field programmable gate array (FPGA) comprising the hardware architecture of any one of claims 1 to 9.

11. An application specific integrated circuit (ASIC) comprising the hardware architecture of any one of claims 1 to 9.

12. A method for processing a spiking neural network by the hardware architecture of any one of claims 1 to 9, the method comprising at least the steps of: - receiving input pixels and generating a spike stream; - filtering the spike stream to generate a reduced number of spikes, wherein spikes of the spike stream are processed in parallel; - processing sequentially the reduced number of spikes; and - selecting an output winner class.

13. The method of claim 12, wherein, the steps are performed in a pipelined manner.

Images